In this Node.js user agent setting guide, you will see:
- Why it is important to set the
User-Agentheader - What the default user agent looks like in Node.js
- How to set a user agent using the Fetch API
- How to implement user agent rotation in Node.js
Let’s dive in!
Why Setting a User Agent Is So Important
The User-Agent header is a string that identifies the client software making an HTTP request. It typically includes details about the browser or application, the operating system, and the system architecture from which the request originates. This header is usually set by web browsers, HTTP clients, or any software that performs web requests.
For example, this is the current user agent set by Chrome when requesting pages:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36Below is a breakdown of the components in this user agent string:
Mozilla/5.0: Originally used to denote compatibility with Mozilla browsers, this prefix is now added for compatibility purposes.Windows NT 10.0; Win64; x64:Indicates the operating system (Windows NT 10.0), platform (Win64), and system architecture (x64).AppleWebKit/537.36: Refers to the browser engine that Chrome uses.- (
KHTML, like Gecko): Shows compatibility with the KHTML and Gecko layout engines. Chrome/127.0.0.0: Specifies the browser name and version.Safari/537.36: Indicates compatibility with Safari.
In essence, the user agent can reveal whether the request is coming from a trusted browser or another type of software.
Web scraping bots and automation scripts often use default or non-browser user agent strings. These can expose their automated nature to anti-bot systems, which protect web data by scrutinizing incoming requests. By analyzing the User-Agent header, these solutions can discern whether the request is from a genuine user or a bot.
For more information, check out our guide on user agents for web scraping.
What Is the Node.js Default User Agent?
Starting with version 18, Node.js includes the fetch() method as a built-in implementation of the Fetch API. That is the recommended way to perform HTTP requests in Node.js because it does not require any external dependencies. Learn more about how to make HTTP requests in Node.js with Fetch API.
Like most other HTTP clients, fetch() automatically sets a default User-Agent header when making a request. For example, the same happens in the Python requests library.
In particular, the default Node.js user agent set by fetch() is:
nodeYou can verify that by making a GET request to the httpbin.io/user-agent endpoint. This endpoint returns the User-Agent header of the incoming request, helping you discover the user agent set by an HTTP client.
Create a Node.js script, define an async function, and use fetch() to make the desired HTTP request:
async function getFetchDefaultUserAgent() {
// make an HTTP request to the HTTPBin endpoint
// to get the user agent
const response = await fetch("https://httpbin.io/user-agent");
// read the default user agent from the response
// and print it
const data = await response.json();
console.log(data);
}
getFetchDefaultUserAgent();Execute the JavaScript code above, and you will receive the following string:
{ 'user-agent': 'node' }As you can see, the user agent set by fetch() in Node.js is simply node. This string is quite different from the user agents used by browsers, which can trigger anti-bot systems.
Anti-bot solutions monitor incoming requests for suspicious patterns, such as unusual user agent strings. Once detected, they mark the request as coming from a bot and block it. This is why changing the default Node.js user agent value is key to avoiding getting flagged!
How to Change the Node.js User Agent Using the Fetch API
The Fetch API specification does not provide a specific method for changing the user agent. At the same time, User-Agent is nothing more than an HTTP header. That means you can customize its value using the fetch() header options.
See how to set the User-Agent header in Node.js using fetch()!
Set a User Agent Locally
fetch() supports header customization via the headers option. Use it to set the User-Agent header when making a specific HTTP request as follows:
const response = await fetch("https://httpbin.io/user-agent", {
headers: {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
},
});Put it all together and you will get:
async function getFetchUserAgent() {
// make an HTTP request to HTTPBin
// with a custom user agent
const response = await fetch("https://httpbin.io/user-agent", {
headers: {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
},
});
// read the default user agent from the response
// and print it
const data = await response.json();
console.log(data);
}
getFetchUserAgent();Launch the above script, and this time the result will be:
{
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36'
}Fantastic! The user agent returned by the API matches the user agent configured in the code. You now know how to change the Node.js user agent.
Set a User Agent Globally
Setting the User-Agent on a per-request basis is straightforward, but it can lead to repetitive boilerplate code. What if you want to change the default Node.js user agent globally? Unfortunately, as of this writing, the fetch() API does not provide a way to override its default settings globally.
What you can do is create a wrapper function to customize the fetch() behavior with your desired configurations:
function customFetch(url, options = {}) {
// custom headers
const customHeaders = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
...options.headers, // merge with any other headers passed in the options
};
const mergedOptions = {
...options,
headers: customHeaders,
};
return fetch(url, mergedOptions);
}You can now make an HTTP request with a custom user agent by calling customFetch() instead of fetch():
const response = await customFetch("https://httpbin.io/user-agent");The complete Node.js script will be:
function customFetch(url, options = {}) {
// add a custom user agent header
const customHeaders = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
...options.headers, // merge with any other headers passed in the options
};
const mergedOptions = {
...options,
headers: customHeaders,
};
return fetch(url, mergedOptions);
}
async function getFetchUserAgent() {
// make an HTTP request to HTTPBin
// through the custom fetch wrapper
const response = await customFetch("https://httpbin.io/user-agent");
// read the default user agent from the response
// and print it
const data = await response.json();
console.log(data);
}
getFetchUserAgent();Launch the Node.js script above, and it will print:
{
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36'
}Wonderful! You just learned how to set a global user agent in Node.js using fetch().
Implement User Agent Rotation in Node.js
Overriding the default User-Agent header of an HTTP client with that of a real browser might not be enough to elude anti-bot detection. If you send too many requests from the same IP address using the same user agent, anti-scraping systems can still identify your activity as automated. The problem is that your requests will still be suggesting non-human behavior.
To reduce the risk of bot detection in Node.js, you should introduce variability into your HTTP requests. One effective technique is user agent rotation, which involves changing the User-Agent header on each request. This approach makes your automated requests appear to come from different browsers, making them less likely to be flagged by anti-bot systems.
In the following section, you will learn how to achieve user agent rotation in Node.js!
Step #1: Retrieve a List of User Agents
Visit a site like WhatIsMyBrowser.com and populate a list of some valid user agent values:
const userAgents = [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/126.0.2592.113",
// other user agents...
];Tip: The more real-world user agent strings this array contains, the better to avoid anti-bot detection.
Step #2: Randomly Pick a User Agent
Create a function that randomly selects and returns a user agent string from the list:
function getRandomUserAgent() {
const userAgents = [
// user agents omitted for brevity...
];
// return a user agent randomly
// extracted from the list
return userAgents[Math.floor(Math.random() * userAgents.length)];
}Let’s break down what happens in this function:
Math.random()generates a random number between 0 and 1- This number is then multiplied by the length of the
userAgentsarray. Math.floor()rounds down the resulting number to the largest integer less than or equal to that number.- The resulting number from the previous operations corresponds to a randomly generated index that goes from 0 to
userAgents.length - 1. - The index is then used to return a random user agent from the array of user agents.
Every time you call the getRandomUserAgent() function, you will likely get a different user agent.
Step #3: Make the HTTP Request with a Random User Agent
To implement user agent rotation in Node.js using fetch(), set the User-Agent header with the value from the getRandomUserAgent() function:
const response = await fetch("https://httpbin.io/user-agent", {
headers: {
"User-Agent": getRandomUserAgent(),
},
});The HTTP request performed via the Fetch API will now have a random user agent.
Step #4: Put It All Together
Add the snippets of the previous steps to a Node.js script, and then wrap the logic for making a fetch() request in an async function.
Here is what your final Node.js user agent rotation script should look like:
function getRandomUserAgent() {
const userAgents = [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/126.0.2592.113",
// other user agents...
];
// return a user agent randomly
// extracted from the list
return userAgents[Math.floor(Math.random() * userAgents.length)];
}
async function getFetchUserAgent() {
// make an HTTP request with a random user agent
const response = await fetch("https://httpbin.io/user-agent", {
headers: {
"User-Agent": getRandomUserAgent(),
},
});
// read the default user agent from the response
// and print it
const data = await response.json();
console.log(data);
}
getFetchUserAgent();Run the script 3 or 4 times. Statistically, you should see different user agent responses as below:
This demonstrates that the user agent rotation is working effectively.
Et voilà! You are now skilled at setting user agents in Node.js using the Fetch API.
Conclusion
In this tutorial, you saw why you should set the User-Agent header and what the default Node.js user agent looks like in fetch(). You also explored how to override this value and implement user agent rotation in Node.js to evade basic anti-scraping systems. Yet, more advanced systems can still detect and block your automated requests. To avoid IP bans, you can configure a proxy in Node.js, but even that may not always be enough!
For a more robust solution, consider Web Scraper API—a next-generation scraping service that simplifies automated web requests in Node.js or any other technology. It effectively bypasses anti-scraping measures through features like IP and user agent rotation, making web scraping easier than ever.
Sign up now and find the best product for your projects. Start your free trial today!
No credit card required
