In this post, we will cover:
- What is curl
- Installing curl.
- What you will need to start using proxies
- How to specify an HTTP/HTTPS proxy in curl.
- How to set SOCKS proxies in curl
- Tips and tricks you should know
- Which proxies are best for curl?
What Is cURL?
cURL, short for “Client URL,” is a tool to send and receive data over the Internet using URLs. As a software project, it provides both a library (libcurl) and a command-line tool (curl). Here, we will focus on curl, the command-line tool for transferring data via the URL syntax.
The project supports a wide range of protocols, including HTTP, HTTPS, FTP, FTPS, SFTP, POP3, POP3S, IMAP, IMAPS, LDAP, LDAPS, SCP, and more. This makes curl the most popular and widely used command-line HTTP client in the world.
cURL is highly versatile and represents a common solution to perform various tasks, such as making HTTP requests, downloading files, uploading data, and interacting with APIs. It offers several features, from request customization to proxy support.
Installing curl
Let’s now learn how to install curl on your machine.
macOS
You do not need to install curl on macOS. The tool is already included in the operating system, and you can use it natively in the Terminal application.
Windows
Starting with Windows 10, Windows comes with a copy of curl. At the same time, the curl command is an alias for the PowerShell Invoke-WebRequest command. This means that curl commands in the Windows terminal will invoke Invoke-Request behind the scenes. To avoid this and actually use curl from cURL, replace “curl” with “curl.exe.” This way, PowerShell will run curl and not Invoke-Request.
For example, you can verify the current version of curl installed on your Windows machine in the terminal with:
That should print something similar to:
If you are a Windows user, replace all “curl” instances in the commands of the article with the “curl.exe” string. Otherwise, set up the Windows Subsystem for Linux (WSL) and follow the instructions below.
Linux
On Linux, the procedure for installing curl changes depending on the specific distribution you are using. Popular Linux distributions, such as Ubuntu and Fedora, come with it by default. So, you can use curl directly in the terminal.
In other distributions, curl may not be included. In this case, you can add it using the distribution package manager. On a Debian-based OS, you can install curl with the following command:
What You Need to Use a Proxy in curl
A proxy acts as an intermediary between the client and the destination server. It intercepts requests from the client, forwards them to the server, receives the response from the server, and sends it back to the client. This middle-man approach increases anonymity and helps avoid network restrictions. This is because the destination server will see the requests as coming from the IP and location of the chosen proxy server and from you.
To get started using curl with proxy services, you first need access to a proxy. Specifically, here is the syntax of a proxy URL:
This consists of:
<PROTOCOL>: The protocol to use to connect to the proxy server. If no protocol is specified, curl will default to http://.<HOST>: The required IP address or URL of the proxy server’s hostname.<PORT>: The port number that the proxy server is listening to. If no port is specified, curl will use 1080 by default.<USERNAME>: The optional username to specify when authentication is required.<PASSWORD>: The optional password to specify when authentication is required.
As for proxy protocols, the most popular ones are HTTP and HTTPS, followed by SOCKS.
It is time to retrieve a valid HTTP proxy!
You can get one for free on Free Proxy List, as below:
This means the proxy URL is:
Opting for such a solution is okay for learning purposes, but you cannot rely on it in a real-world scenario. Free proxy services are unreliable, error-prone, slow, data-greedy, and short-lived. Do not use them!
The solution? Premium proxies from Bright Data, the best provider in the market. Subscribe and try our reliable proxies for free.
Keep in mind that premium solutions are protected by authentication. Suppose the protocol is HTTP, host is 71.103.203.109, and port is 9321, and the pair of credentials are admin-32458 and rTuJ3tEwf. In this case, the curl proxy URL would be:
How to Specify an HTTP/HTTPS Proxy in curl
Before getting started, launch the command below in your terminal:
As you can see, the basic curl syntax is:
A curl best practice recommends surrounding URLs with double quotes to avoid issues with special characters.
Run it and you should get something like this:
The HTTPBin project allows you to get information about your HTTP requests. In detail, the /ip endpoint returns the origin IP of the request, which is the IP address of the machine that performed the request. In other words, it returns your IP.
After setting up a proxy in curl, you should see the IP address of the proxy server and not yours.
There are several ways to set a proxy in curl. Let’s now jump into the most popular approaches to specifying an HTTP/HTTPS proxy in curl.
Using a Command Line Argument
curl offers two command line arguments to set a proxy URL:
-x--proxy
The two can be used interchangeably, as they do exactly the same thing. The latter is just an alias of the former.
So, the syntax to use curl with proxy services becomes:
Or:
Note: Command line options are case-sensitive in curl. For example, -x and -X have two different meanings.
If you now launch:
Or the equivalent:
You should get:
The origin matches the proxy server IP. This means that the target site sees the request as coming from the proxy, as desired. Great!
Do not forget that free proxies are short-lived, so the above server is unlikely to still be alive by the time you read this article. In case of an error, replace it with a fresh proxy.
Using Environment Variables
Another way to use a proxy in curl is by setting the following two environment variables:
http_proxy:The URL of the proxy server to use to access URLs involving the HTTP protocol.https_proxy: The URL of the proxy server to use to access URLs involving the HTTP protocol.
On macOS and Linux, you can set those two environments with the syntax below:
So, the two commands will become:
On Windows, you need to use this PowerShell syntax:
Then, in our example:
From now on, every curl command will go through the specified proxies without having to use the -x option:
Again, this will return:
To turn off the curl proxies, unset the environment variables with:
Or on Windows:
curl will go back to its standard behavior and https://httpbin.org/ip will now return your IP.
Using a configuration file
You can globally specify a proxy for curl by creating a .curlrc config file.
On Linux and macOS, open the terminal and navigate to your home directory:
Then, access the .curlrc with nano:
If the file already exists, it will be opened. Otherwise, it will be created first and then opened.
Add the following line to the .curlrc file:
The syntax of this instruction is:
Save the file, close the terminal and relaunch it.
On Windows, you need to create a _curlrc file inside the %APPDATA% directory with the same content. Paste %APPDATA% in the file explorer address bar and press ENTER to access the directory.
This should bring you to C:Users<YOUR_USER>AppDataRoaming.
Now, curl will automatically use the proxy specified in the configuration file:
You will get:
Fantastic! You just learned how to use curl with proxy servers in three different ways!
How To Set SOCKS Proxies in curl
If you want to use a SOCKs proxy, the command structure remains the same:
The main difference is that <PROTOCOL> will be socks4, socks4a, socks5, or socks5h, instead of http and https.
A complete example of a curl SOCKS proxy in action is:
As you can expect, the result will be:
An alternative approach involves the --socks4, --socks4a, --socks5 command-line options instead of -x. In this case, you must set the proxy URL without username and password after the option and then the credentials after --proxy-user:
For example:
This will connect to HTTPBin via the 93.16.238.41:8721 socks4 proxy, using admin334 as the username and kv4NsDgc3 as the password for authentication.
Tips and Tricks You Should Know
In this section, you will see some interesting tricks and valuable tips for dealing with a curl proxy like a pro.
Quickly Turning Proxies ON and OFF
The idea here is to define aliases to programmatically set and unset proxies via environment variables with custom commands.
On macOS and Linux, you can do that by creating a .bashrc file. This is a script file that is executed every time the user logs in.
Move to your home folder:
Then, create or open a .bashrc file:
Add the content in the format below to the file:
Thus, in our example it would be:
Save the file and restart your machine.
Now, you can activate and deactivate a curl proxy with:
Note the custom proxyon and proxyoff commands. These perform the operations specified by the aliases in the .bashrc file.
On Windows, you can achieve something similar with the PowerShell aliases.
Ignoring Proxies for a Single Request
If you want to prevent curl from using a configured proxy for a specific request, you can use the --noproxy "*” argument as below:
This instructs curl not to use proxies for all URLs.
Set up a proxy and run:
The resulting origin will contain your IP, not that of the proxy server.
Avoiding SSL Certificate Errors
When using proxies in curl, your requests may fail because of an SSL certificate errors. To avoid that, use the -k option as in the syntax below:
-k helps you avoid certificate errors by allowing insecure server connections over SSL.
For example, you can contact HTTPBin bypassing SSL issues with:
Getting More Information About the Request
When a request in curl fails, it is not easy to understand what happened. To diagnose the error and see all the operations performed by curl, set the -v option:
This will output the details about the connections established and the headers used by the tool:
Which Proxies Are Best for curl?
The answer to this question depends on your use case and the nature of the curl requests you have to make. To find the right solution for your needs, take a look at the different types of proxies available:
- Datacenter proxies: They are fast but may be detected and blocked easily by sites due to their identifiable IP ranges.
- Residential proxies: They guarantee a high level of anonymity because they offer IP addresses coming from real devices in specific locations. Residential proxies are perfect for accessing sites relying on geo-restriction blocks or anti-bot measures, and to perform web scraping with curl.
- ISP proxies: They are fast, secure, and highly reliable as they provide IPs from devices registered to ISPs. ISP proxies represent a perfect solution for SEO monitoring, web browsing, and market research.
- Mobile proxies: They offer IPs from real mobile devices for a high level of anonymity. They are useful for accessing mobile-specific applications, sites, or content.
This was just a brief recap, but you can find out more in our guide to proxy IP types.
Conclusion
In this curl proxy guide, you learned what cURL is, what it has to offer, and how to set up an HTTP/HTTPS/SOCKS proxy. As it turns out, you can’t rely on curl with proxy services from free providers. Thus, the main technical decision to make is which proxy provider to adopt. Save time and energy and go for the best on the market, Bright Data.
Bright Data controls the best proxy servers in the world, serving Fortune 500 companies and over 20,000 customers. Its worldwide proxy network involves:
- Datacenter proxies – Over 770,000 datacenter IPs.
- Residential proxies – Over 72M residential IPs in more than 195 countries.
- ISP proxies – Over 700,000 ISP IPs.
- Mobile proxies – Over 7M mobile IPs.
Overall, that is one of the largest and most reliable scraping-oriented proxy networks on the market. Join the largest proxy network and get a free trial.
No credit card required
