Web data now answering strategic questions
Someone in a senior position gets paid to make high-quality decisions that produce beneficial business outcomes, reflecting positively on the company and the individual’s professional trajectory.
But smart decisions which usually come in the form of questions can only be made based on the most reliable, up-to-date information.
This is where web data steps in.
If we use the digital retail space as an example, some of the biggest day-to-day operational questions can be answered using web data.
Such as:
‘What are my competitor’s best selling products?’
The data points that can best help answer this question are:
- Competitor inventory monitoring, such as collecting data on which Stock Keeping Units (SKUs) have the highest Sell-Through Rates (STRs). SKUs include additional product attributes such as color, manufacturer, model, and warranty terms. This means that businesses can also look for correlations between one of these attributes, and above-average STRs.
- Positive customer ratings, and consumer sentiment on reviews indicating that the item’s quality, and delivery was satisfactory as far as consumers are concerned.
- High positioning in marketplace search results indicating that the marketplace’s algorithm identifies this product as being in high demand, and the seller as reliable.
‘What dynamic pricing strategy will help us corner our market?’
Dynamic pricing doesn’t necessarily mean lowering your pricing but rather customizing it to target consumers. Relevant data points in this respect include:
- Pricing items based on the zip code a prospect is accessing your site from. Serving higher prices to people living in more well-to-do areas, and lower prices to folks who live in less well-off places.
- Supply, and demand data indicating when competitors’ inventory levels are low, and demand is high (this is when prices should be raised) or when market inventory is abundant across-the-board and demand is low/moderate (this is when prices should be lowered).
‘How do I perform product matching for each individual product?’
Relevant data points include:
- Cross-referencing item specifics in order to confirm two item’s compatibility.
- Comparing listing titles – specifically looking at title length, keywords included, and sentence structure to determine if you are comparing oranges to oranges.
- Seeing how item visuals match up in terms of angles, styling, and quantity of images.
Competitors intentionally mesh-up these product identifiers in order to make it harder to compete, once data is used to dispel these smoke clouds, competing becomes much more straightforward.
‘Which categories, and brands are trending in specific geolocations?’
Relevant data points in this regard would be:
- Shipping data indicating what percentage of competitor sales are being sent to which regions, countries, and cities.
- Language detection including scanning the web for localization efforts by competitors, e.g. identifying an English language listing translated into French, German, and Japanese.
‘Where are there customer satisfaction gaps we can capitalize on?’
Using Natural Language Processing (NLP), text-heavy data points can be mined for consumer sentiment on specific items or broader solutions using data points such as:
- How people are reacting to an item on social media either on a video product review on YouTube or a Reddit thread discussing a product category.
- Customer reviews either on or off the marketplace where items are being sold. This is especially true of customers’ reactions to competitor products. Typically one or more specific points of contention can be picked up, and capitalized on. E.g. Slow shipping, lack of customer support, missing parts etc.
‘How do I keep track of competitor product offers that are constantly changing? ’
Relevant data points include:
- Scanning competitor stores, websites, and other Points of Sale (PoS) in real-time, discovering new products and categories that your company can start competing with/in.
- Collecting information on publicly available product catalogs so that your business can receive alerts when a new product is introduced into the inventory pool.
But obtaining web data is not so easy
The sad truth however is that actually being able to obtain customer/competitor web data in real-time, and in a structured format that can be utilized by algorithms immediately is not as simple as it sounds.
To accomplish this, professionals will need:
- Infrastructure (hardware and software) such as cloud servers, networks, Application Programming Interfaces (APIs), and regular code changes based on target site architecture changes.
- Technical teams including an extensive group of engineers as well as IT, and DevOps personnel.
- Data preparation as information is typically collected in its most basic, unstructured form. It typically needs further processing including removing duplicates, cleaning corrupted files, enriching datasets with missing information, and structuring data into a usable CSV or JSON file format.
And yet now there is hope
In recent months new autonomous data collection tools have been released. These are fully automated Web Scraper IDEs that do not require any in-house infrastructure. They build on previously developed data technology including:
- Artificial Intelligence that takes care of cleaning, matching, synthesizing, and processing data sets.
- Structuring target web data, and formatting it (JSON, CSV, HTML, or XSLS) so that it’s ready-to-use, and can be analyzed.
- Machine Learning identifies target site structure changes on an ongoing basis to help avoid blocks, and smoothly retrieve data
All of this means that executives can start answering their business questions using user-generated data collected in real-time, empowering them to make smarter decisions, while creating more meaningful corporate value.
