cURL User Agent Guide: Setting and Changing

In this guide, you will learn:

• What the User-Agent header is and why it is so important
• What the default cURL user agent header is
• Two approaches to setting cURL user agent strings
• How to implement user agent rotation logic in cURL

Let’s dive in!

What Is a User Agent and Why It Is Important?

A user agent is a string that browsers, applications that make web requests, and HTTP clients set in the User-Agent HTTP header to identify the client software the request originates from. This string typically includes information such as browser/application type, operating system, and other details.

For example, this is a real-world user agent string from a Chrome request:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

The information in this header is critical in determining whether requests are coming from a known browser or another software. Scraping bots tend to use inconsistent or default user agent strings, exposing their automated nature. Thus, the user agent header helps anti-bot solutions understand whether the current user is real or a bot. 

For more information, check out our guide on user agents for web scraping. 

What Is the Default cURL User Agent?

Just like most HTTP clients, cURL sets the User-Agent header when making an HTTP request. In detail, the cURL user agent string is:

curl/X.Y.Z 

Where X.Y.Z is the version of cURL installed on your machine. 

To verify that, make a cURL request to the /user-agent endpoint of the httpbin.io project. This API returns the User-Agent header string set by the caller.

Make a GET request to /user-agent with cURL using the following instruction:

curl "https://httpbin.io/user-agent"

Note: On Windows, replace curl with curl.exe. This is because curl is an alias to Invoke-WebRequest in PowerShell, while curl.exe points to the cURL Windows executable. 

The endpoint should return something like this:

{

  "user-agent": "curl/8.4.0"

}

As you can see, the user agent set by cURL is curl/8.4.0. That is not good because it clearly identifies the request as coming from cURL. Anti-bot solutions adopted by sites to protect their pages and data could easily mark that request as not coming from a real user, blocking it accordingly. 

Here is why you need to know how to set cURL user agent header strings.

How to Set cURL User Agent Header

There are two approaches to setting a user agent in cURL. Let’s explore them both!

Set a Custom User Agent Directly

Setting user agents is so popular that cURL comes with an option exactly for that. In particular, the -A or –user-agent option allows you to specify the string to set in the User-Agent header of the HTTP request made by cURL.

This is the syntax for setting a cURL user agent header using that option:

curl -A|--user-agent "<user-agent_string>" "<url>"

Consider the following example:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "https://httpbin.io/user-agent"

The output will be:

{

  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

}

Great! The user agent detected by the httpbin.io /user-agent endpoint is the same as the one set in the request.

Keep in mind that the above request is equivalent to:

curl --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "https://httpbin.io/user-agent"

While it is not recommended, to unset the User-Agent header entirely, pass an empty string to -A. Verify that by targeting the /headers endpoint from httpbin.io, which returns the headers of the incoming request:

curl -A "" "https://httpbin.io/headers"

The result will be:

{

  "headers": {

    "Accept": [

      "*/*"

    ],

    "Host": [

      "httpbin.io"

    ]

  }

}

As expected, no User-Agent header.

If you instead want to keep the User-Agent header but give it a blank value, pass a single space string to -A: 

curl -A " " "https://httpbin.io/headers"

This time, the endpoint will return:

{

  "headers": {

    "Accept": [

      "*/*"

    ],

    "Host": [

      "httpbin.io"

    ],

    "User-Agent": [

      ""

    ]

  }

}

Note the empty User-Agent string value.

Set a Custom User Agent Header

After all, User-Agent is nothing more than an HTTP header. This means that you can set it like any other HTTP header in cURL by using the -H or –header option:

curl -H|--header "User-Agent: <user-agent_string>" "<url>"

For a complete tutorial, read our guide on how to send HTTP headers with cURL.

See the -H option in action in the following example:

curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "https://httpbin.io/user-agent"

Note that HTTP headers are designed to be case-insensitive, so User-Agent is equivalent to user-agent.

The result will be:

{

  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

}

Wonderful! The user agent value has been set as desired.

The above cURL command is equivalent to:

curl --header "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "https://httpbin.io/user-agent"

For special use cases, consider the following two cURL user agent header strings:

• “User-Agent:” to remove the User-Agent header from the request.
• “User-Agent: ” to set the User-Agent header to a blank string.

Implement User Agent Rotation in cURL

When making automatic requests at scale using cURL, setting a fixed value of User-Agent may not be sufficient. The problem is that anti-bot technologies monitor all incoming requests. If they see too many of them with the same headers and from the same IP, they are likely to take action.

The key to avoiding detection and blocking by sites when making automated requests is to randomize them. In this regard, rotating user agents helps simulate requests from different browsers. This reduces the risk of triggering temporary bans or blocking.

Achieving cURL user agent rotation with the following three steps:

1. Collect user agents:Gather a list of real-world user agent strings from various browsers, of different versions and on different devices.
2. Implement rotation logic:Select a random user agent from the list.
3. Randomize requests:Set the selected user agent string in the cURL request.

Implementing this procedure requires a few lines of code, so it changes from Unix-based systems to Windows. Find out how to implement user agent rotation in cURL in both environments!

Bash

Collect a list of valid user agents from a site like User Agent String.com and store it in an array variable:

user_agents=(

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:126.0) Gecko/20100101 Firefox/126.0"

    # ...

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"

)

Then, implement a function that randomly extracts it from the list using RANDOM:

get_random_user_agent() {

    # number of user agents in the list

    local count=${#user_agents[@]}

    # generate a RANDOM number from 0 to count

    local index=$((RANDOM % count))

    # return the randomly extracted user agent string

    echo "${user_agents[$index]}"

}

Call the function, get a rotating user agent, and set it in cURL:

# get the random user agent

user_agent=$(get_random_user_agent)

# perform a cURL request to the given URL

# using the random user agent

curl -A "$user_agent" "https://httpbin.io/user-agent"

Note: Change the target URL according to your goals.

Put it all together, and you will get the following bash file:

#!/bin/bash

# list of user agent strings

user_agents=(

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:126.0) Gecko/20100101 Firefox/126.0"

    # ...

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"

)

get_random_user_agent() {

    # number of user agents in the list

    local count=${#user_agents[@]}

    # generate a RANDOM number from 0 to count

    local index=$((RANDOM % count))

    # return the randomly extracted user agent string

    echo "${user_agents[$index]}"

}

# get the random user agent

user_agent=$(get_random_user_agent)

# perform a cURL request to the given URL

# using the random user agent

curl -A "$user_agent" "https://httpbin.io/user-agent"

Launch the above script, and every time you will see a different user agent. Mission complete!

PowerShell

Retrieve a list of real-world user agents from a site like WhatIsMyBrowser.com. Next, store that data in an array variable:

$user_agents = @(

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:126.0) Gecko/20100101 Firefox/126.0"

    # ...

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"

)

Then, create a function that randomly picks a user agent string from the list and returns it:

function Get-RandomUserAgent {

    # number of user agents in the list

    $count = $user_agents.Count

    # generate a random number from 0 to $count

    $index = Get-Random -Maximum $count

    # return the randomly extracted user agent

    return $user_agents[$index]

}

Finally, call the function, get the random user agent, and use it in cURL:

# get the random user agent

$user_agent = Get-RandomUserAgent

# make an HTTP request to the given URL 

# using the random user agent

curl.exe -A "$user_agent" "https://httpbin.io/user-agent"

Put it all together, and you will get the following code:

# list of user agents

$user_agents = @(

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:126.0) Gecko/20100101 Firefox/126.0"

    # ...

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"

)

function Get-RandomUserAgent {

    # number of user agents in the list

    $count = $user_agents.Count

    # generate a random number from 0 to $count

    $index = Get-Random -Maximum $count

    # return the randomly extracted user agent

    return $user_agents[$index]

}

# get a random user agent

$user_agent = Get-RandomUserAgent

# make an HTTP request to the given URL 

# using the random user agent

curl.exe -A "$user_agent" "https://httpbin.io/user-agent"

Store it in a .ps1 script and execute it a few times. Every time, you will get a different user agent string.

Et voilà! You are now a master in setting cURL user agent header strings.

Conclusion

In this guide, you took a look at why setting the User-Agent header in an HTTP client is so important and how to do it in cURL. This way, you can fool some anti-bot systems into thinking that your requests are coming from a relational browser. At the same time, advanced anti-bot solutions will still be able to block your requests. To get around measures like rate limiting, you could integrate a proxy into cURL. However, that may not be enough!

Avoid all this stress and try Scraper API. As is an all-in-one, next-generation, comprehensive scraping API, it provides everything you need to perform automated web requests using cURL or any other HTTP client. This full-featured solution features IP and user agent rotation, and it can bypass any anti-bot technology. Making automated HTTP requests has never been easier!

Register now for a free trial of Bright Data’s web scraping infrastructure or talk to one of our data experts about our scraping solutions.