Why American Hospitals Are Hiding Procedure Rates, And How Data Can Bring Much-Needed Transparency To The Medical Industry

Hospitals in the US are using a designated code snippet to prevent healthcare procedure pricing pages from being indexed by search engines, undermining transparency laws, and consumer trust
data collection tools for bringing transparency to medical billings and improve hospital pricing ethics
Nadav Roiter - Bright Data content manager and writer
Nadav Roiter | Data Collection Expert
28-Mar-2021

In this article we will discuss:

Victory for healthcare pricing transparency?

At first glance, American consumers thought they had triumphed over the corporate healthcare, and insurance industries. A federal appeals court in Washington upheld the statutory intent of the Affordable Care Act (AFA) requiring hospitals to disclose secret rates they negotiate with insurance companies.

Compliance with legislation, and transparency regarding the costs of procedures, and services provided to individual patients would cost hospitals an estimated ~$39 million. Increased pricing transparency would also encourage free market competition, ultimately lowering bottom line consumer costs. But, an industry as lucrative as this, worth over 6% of annual US GDP, or roughly $1.2 trillion, was not about to give in so easily.

How hospitals are carrying out data-burying practices

So hospitals decided that they would do consumers a ‘favor’, and transparently display pricing, but not without a fight. Based on a collaborative investigation carried out by The Wall Street Journal (WSJ), and the Turquoise Health Co (THC),

The pricing pages would physically exist but medical centers came up with a plan to embed code that prevents search engines from indexing pages, rendering them virtually inaccessible to the average consumer.

In order to prevent web crawlers from crawling pricing pages, around 3,000 hospitals, and medical centers across the United States added the following meta tag to the ‘head section’ of their respective pricing pages:

HTML header meta for signaling to bots such as Google to not index a webpage so that it doesn't appear in search engine results

Image source: Bright Data

The practice itself is not novel, the context, and application, however, are.

Searchability is a key tenet of transparency, and optimizing content so that it cannot be found is a clear violation of consumer rights, and the ‘spirit of the law’. According to Google Search Central, ‘noindex’ code is typically used by developers to discourage search engines from indexing ‘alpha stage’ or ‘cached copies’ of web pages before they have been completed. Once they are finalized, this code snippet is removed to allow search engine crawlers to index the content, enabling users to find it through routine search queries.

The opaque approach that American hospitals chose to take was unveiled in the above mentioned WSJ-THC investigative report. Findings included astounding rates of medical institutions making a concerted effort to delude patients:

  • 307 hospitals directly used the above code snippet to hide pricing information pages
  • 182 hospitals removed the code after being contacted by WSJ investigative reporters (indicating intent to deceive would be patients)
  • And a total of 45% out of 2,267 investigated health institutions fell short of consumer pricing transparency, and compliance standards

Consumer-powered technologies are leveling the playing field

This is where peer-to-peer data collection networks come into play. Companies, government agencies, and researchers who are working towards building a more equitable, and transparent digital economy, are using web crawling technologies to make information accessible to the public. A good example of this is, Health Advocate, a patient advocacy organization working towards helping people become better educated health consumers. Powered by data collection technology, like hundreds of other healthcare advocacy applications, websites, and tools, Health Advocate helps consumers compare:

  • Medical procedure costs based on ZIP code/geolocation
  • Specialist visit/procedure pricing
  • Key quality/safety indicators
  • Patient reviews

The bottom line

The medical, and insurance industries benefit in many respects from a lack of consumer transparency. Data collection networks are powering technologies, applications, tools, and patient services that are helping to restore clarity so that people can get the care they need, at affordable prices.

Nadav Roiter - Bright Data content manager and writer
Nadav Roiter | Data Collection Expert

Nadav Roiter is a data collection expert at Bright Data. Formerly the Marketing Manager at Subivi eCommerce CRM and Head of Digital Content at Novarize audience intelligence, he now dedicates his time to bringing businesses closer to their goals through the collection of big data.

You might also be interested in

What is data aggregation

Data Aggregation – Definition, Use Cases, and Challenges

This blog post will teach you everything you need to know about data aggregation. Here, you will see what data aggregation is, where it is used, what benefits it can bring, and what obstacles it involves.
What is a data parser featured image

What Is Data Parsing? Definition, Benefits, and Challenges

In this article, you will learn everything you need to know about data parsing. In detail, you will learn what data parsing is, why it is so important, and what is the best way to approach it.
What is a web crawler featured image

What is a Web Crawler?

Web crawlers are a critical part of the infrastructure of the Internet. In this article, we will discuss: Web Crawler Definition A web crawler is a software robot that scans the internet and downloads the data it finds. Most web crawlers are operated by search engines like Google, Bing, Baidu, and DuckDuckGo. Search engines apply […]

A Hands-On Guide to Web Scraping in R

In this tutorial, we’ll go through all the steps involved in web scraping in R with rvest with the goal of extracting product reviews from one publicly accessible URL from Amazon’s website.

The Ultimate Web Scraping With C# Guide

In this tutorial, you will learn how to build a web scraper in C#. In detail, you will see how to perform an HTTP request to download the web page you want to scrape, select HTML elements from its DOM tree, and extract data from them.
Javascript and node.js web scraping guide image

Web Scraping With JavaScript and Node.JS

We will cover why frontend JavaScript isn’t the best option for web scraping and will teach you how to build a Node.js scraper from scratch.
Web scraping with JSoup

Web Scraping in Java With Jsoup: A Step-By-Step Guide

Learn to perform web scraping with Jsoup in Java to automatically extract all data from an entire website.
Static vs. Rotating Proxies

Static vs Rotating Proxies: Detailed Comparison

Proxies play an important role in enabling businesses to conduct critical web research.