One Gatekeeper Is Too Many: Defending Access to Public Web Data

In a move more harmful to the web than ever, there is a new attempt to control web traffic, by blanket blocking and labeling agents, leaving one multibillion dollar company to choose who gets access to public web pages and who doesn’t. Cloudflare is trying to become the toll booth to access information in the public domain. The decision is being marketed as protecting creators, but those of us who have worked to protect the web know this is just another way to gain control of who can see and use the world’s information.

 LLMS and agents are quickly becoming the norm over search engines and human actions. If an infrastructure provider decides which page an AI model can and cannot read, it is deciding who gets visibility and who disappears. Blanket blocks do not stop data collection; they make access more expensive, shut out small businesses, and encourage unethical practices. Every major leap on the web, from price-comparison engines to large language models, was built on the ability to collect public web data, an open web is critical for advancements in technology.

What’s more is this ultimately will hurt creators and media, the very groups it claims to protect.

The move strikes at the heart of Bright Data’s core principle: public data must remain public. For a decade, we at Bright Data have fought hard to protect public web data at every turn. We’ve won legal challenges that defined public web data and its legality to access it at scale. What’s public is clear and it’s defined by law. We will not back down or give into another attempt to gate keep public information.

Bright Data is the response to the threat of an open web for everyday users and businesses. It’s the access layer for AI, Agents, and business intelligence that guarantees the data your team relies on continues to be fed into your pipeline, regardless of the attempts to block. Our products come with dedicated support and R&D team to immediately respond to website changes. 

The Web MCP is free, to everyone, and works for 96% of use cases. Add it to your current agent so you can continue to search and interact with the web without getting blocked. You can make up to 5,000 requests per month for free as part of our commitment to keep the web open. An enterprise solution is also available.

The open web is too valuable to be fenced off by any single company. If a publisher truly wants content to be private, the solution is simple: put it behind a login. Until then, public is public.

Together we can keep the internet open, fair and accessible for innovation and everyone who depends on it.

Or Lenchner
CEO, Bright Data