The Only Thing Worse Than a Fly-By…
Why “scrapers” are bad for publishing revenue.
This post is republished with permission and originally appears here.
…is a scraper! Or at least scrapers that aren’t monetized properly. Scrapers are users or automated services that systematically consume media content – especially news. Their motivations are usually commercial in nature such as a media monitoring service, advertising verification, or lead sourcing. A scraper can be identified by the fact that their volume of consumption is several standard deviations or more higher compared to the rest of the audience – even compared to fans.
It’s not uncommon for scrapers to make up less than 0.1 percent of an audience and generate 10-15 percent or more of the page views (i.e., ad inventory and revenue capacity). The graph to the right illustrates the behavioral difference between the largest segment of visitors, fly-bys, and the smallest segment, scrapers. On average, scraper behavior generates 100-150 times more page views than their percentage of the total audience represents.
So why is a scraper bad for publisher revenues? Aren’t page views good?
The first reason is that scrapers devalue a publisher’s ad inventory by lowering conversion rates for advertisers. Page views delivered to scrapers are like page views delivered to bots. An advertiser that purchases a scraper page view is wasting their money.
Therefore, mixing scraper page views with regular audience page views produces lower conversion rates. As advertisers see the lower rates, they move their advertising dollars to higher performing sites.
The second reason is that scrapers decrease sell-through rates and CPMs. Many publishers do not have 100 percent sell-through rates and as a result, it is harder to negotiate ad rates with advertisers because the publisher is trying to get all the inventory sold. In the case of selling inventory without scraper page views, the publisher creates scarcity along with higher quality (see first reason) to raise prices. Because scrapers reduce sell-through rates and CPMs, they hurt a publisher’s revenue.
The third reason is that unmonetized scrapers represent lost revenues. Remembering that scrapers have economic motivations to consume media, publisher can provide a different value proposition. Rather than monetizing their behavior like the rest of the audience, scrapers can be charged for direct licensing of the content. The beauty of direct licensing is that the publisher typically gets a multiple over the ad revenue these scrapers would have produced, and for the ones that don’t license the content, they are at least blocked from devaluing the ad inventory and lowering CPMs.
While some publishers try to identify and stop excessive consumption, the bar is often set too high.