In this article, we will discuss:
- The challenges of collecting data in the travel industry
- How companies are using web data to supercharge operations
The challenges of collecting data in the travel industry
Competitor intelligence difficulties
When attempting to collect real-time travel data (such as pricing and bundles) from target sites, the two main challenges are:
One: Competitor sites block data collection when they detect too much traffic from a single IP address (i.e., ‘rate limitations’).
Two: Gaining access to GEO-specific information tailored to target audience needs. Many sites either distort information or block requests that originate outside of their geography (i.e., a hotel in New York trying to see how prices appear on Japanese OTA’s to consumers).
API-related challenges
The main challenges when using an API for data collection in general and specifically in the context of the travel industry include:
- Stale Data – This means that the data being retrieved is not the most up-to-date version committed by the target site but rather the last cached version of the information. This can cause issues for a travel package company trying to offer the most competitive offer on the market yet basing deals on information that is a day old, for example.
- Concurrent requests – Many APIs put a limit in terms of how many active sessions a user can run simultaneously. This can be a major constraint for an OTA that needs to scan hundreds or thousands of industry sites at the same time in order to remain competitive.
- API Calls – This is another quantitative limitation as many APIs limit the number of calls that can be sent over a certain period of time. This means that instead of being able to monitor the web freely for new travel trends and consumer behavior, travel sites need to pick target data with a fine tooth comb and calculate precise batch sizes. This can severely limit the amount of data-driven opportunities that a company is able to potentially discover.
- Batch Size – Batch size limitations can also be on the target site end of things. If a target site limits batches to 100 records per request, then it will take 10 data requests in order to complete a job. What this effectively means is that the retrieval of information is slower, i.e., a company’s ability to pivot, react/act proactively in light of market shifts becomes stiffer.
How companies are using web data to supercharge operations
Company #1: A hotel chain wondering how rates are being displayed on third-party distribution channels
The company: This is a mid-sized hotel chain located in the US. They mainly cater to leisure customers and, as such, are very reliant on OTA-based web traffic where most of their customers discover, and book rooms.
The challenge: Since they are reliant on third-party sites for bookings, they cannot control how much vendors display their rooms for. OTAs notoriously cut prices without telling establishments and make up for the discrepancy by raising prices on other listings. This is a strategy in order to convert customers in certain situations, which is good for the OTA but flutes the hotel’s brand.
The hotel chain would like to call out the OTA on this and use their legal department to ask them to stop as these practices are in breach of their operating agreement. But before they can do this, they need to collect this information so that they can prove their claims. The only issue is that the hotel’s technical team is using an outdated API that has trouble sending out ‘concurrent requests’ to all third-party sites and also very often receives ‘stale data’.
The Solution: The company is now using a data collection network that provides them with access to their Rotating Residential IP network. This is a global network of IPs that belong to real individuals located in countries and cities across the globe. The hotel is now able to send an unlimited amount of concurrent requests as each request is routed through a different device.
Data is highly accurate and retrieved in real-time as this proxy network has very low latency and does not access cached information but rather obtains it directly from the live target site.
The outcome: The company is now able to ensure that all third-party vendors are maintaining prices that do not fall below pre-defined and agreed-upon rates.
Company #2: An Online Travel Agency (OTA) wondering how competitive their offers are
The company: This is an OTA that offers travel bundles including flight, rental car, and accommodations. They have many competitors across the board and operate in an ecosystem that is constantly changing and evolving.
The challenge: The company has interests in many geolocations and, as such, needs to collect real-time information on how their competition displays similar offers to different target audiences. But accomplishing this is not so easy as their head office is located in Germany. They can retrieve information from competitor sites, but very often, their IPs get blacklisted due to sending too much traffic to a target site. Other times, they cannot access a site in China, for example, using a German IP address.
The Solution: This company has decided to use a tool called Web Unlocker, which enables them to automate the process of IP rotation as well as using Machine Learning (ML) algorithms in order to solve and circumvent target site blocks and restrictions.
The outcome: This OTA is able to better compete and offer bundles that are better suited to the ecosystem within which they operate. Their systems are now updated when competitor bundle offers change, allowing them to strategically pivot where they deem necessary.
The bottom line
Many companies in the travel sector struggle at the API level with stale data and concurrent requests. Others are finding it difficult to obtain competitor information in real-time in order to grab increased market share. Using a data collection network that can help you improve your data collection capabilities and success rates can help your company shine in a very crowded market.
