Why We Just Blocked AI Scrapers at The Equity (And Why Your Business Should Too)

A few weeks ago, we made a small change to our servers, blocking AI scrapers from crawling the digital pages of The Equity, the Pontiac’s local newspaper of record for the past 143 years.

This wasn’t a hard decision to make.

For the better part of two decades, Big Tech built a multi-trillion-dollar empire off the backs of original content creators. Search engines and social platforms repackaged headlines and articles to keep eyeballs glued to their own apps. But what’s happening with generative AI right now is easily the biggest rip-off yet.

The AI Problem: Taking Everything, Paying Nothing

The honest reality of local journalism is that our reporters spend their days driving out to local communities, sitting through three-hour municipal council meetings, digging through budgets, and earning the trust of sources. It takes real time, real sweat, and real money.

An AI crawler doesn’t care about any of that.

In a fraction of a second, automated bots sweep in, vacuum up our articles, and bypass the paywalls that keep our reporters paid. They take that reporting and feed it directly into massive machine-learning models without asking for permission and without paying a cent.

Worse, they use that stolen work to generate quick summaries on their own platforms so people never click through to the original source. Early industry estimates suggest these AI-driven summaries could slash referral traffic to original content by anywhere from 20% to 40%.

Our reporting belongs to our community and the readers who support us—not to a Silicon Valley server farm looking for free fuel for its next software update. That’s why we pulled the plug on them.

Meta’s New Terms: A Warning Sign for Local Businesses

If you run a local service business, a shop, or a consultancy, it’s easy to look at this and think: “That’s tough for newspapers, but it doesn’t affect me. I don’t write articles for a living.”

You might want to take a closer look at Meta’s latest moves.

Meta recently updated its Commercial Terms of Service, and buried in the fine print is a major shift: if you use a Facebook or Instagram business account, you are effectively granting automated web crawlers permission to access, index, and scrape your business’s external website.

Meta claims this is standard practice to optimize advertising tools. But the language drafted gives them an incredibly wide net to analyze your copy, catalog your products, and ingest your business’s proprietary content right back into its ecosystem.

There’s an old saying about social media: “If you aren’t paying for the product, you are the product.”

For years, that meant your personal data was the currency. Today, tech giants are treating your business’s digital assets the exact same way. If they can scrape your pricing, services, and FAQs directly from your website, they can answer customer queries directly inside Facebook or Instagram—ensuring those customers never actually have to visit your site to do business.

Guarding What You Build

Don’t panic; your business isn’t going to disappear tomorrow because of a terms-of-service update. But it is a clear warning shot.

Platform algorithms change on a whim, and tech companies will always prioritize keeping users inside their own walled gardens. Giving them free rein over your hard-earned web content only helps them do that faster.

You don’t have to just roll over and accept it, either. Just like we did with The Equity, you can take simple technical steps—like updating your website’s robots.txt file—to tell unwanted corporate crawlers to back off.

More than anything, it’s a reminder to invest in assets you actually own: your website, your customer relationships, and your direct channels. The work you put into building your business should benefit your bottom line, not Meta’s.

Interested in
Learning More?

Every business is different, and no "cookie cutter" solution will fix your digital and marketing needs. Let's grab a coffee and talk about your goals and how you can achieve them.
Calumet Media's owner Jon Stewart