curl, short for “client URL,” is a command line tool used to transfer data to a specific server through its URL. curl is highly accessible: it requires only a command line interface (CLI) and an internet connection, which makes it widely used in routers, printers, cars, and medical devices. It’s even used on Mars.
For most developers, curl is an excellent tool to quickly test a backend service directly from any operating system.
In this article, you’ll learn more about curl and how to use it.
History of curl
Originally created and distributed by the Brazilian developer Rafael Sagula in 1996, Daniel Stenberg took over the project soon after its first release, implementing some adjustments that allowed Stenberg to automatically fetch currency exchange rates from the web.
Given its function, curl was initially named httpget. When the number of supported protocols increased, the name changed to urlget. By 1998, when the project had grown enough to support uploading features, it was finally named curl.
Its growth did not stop there. Over the years, curl grew to support over twenty-five protocols and several functionalities, and it handles hundreds of millions of requests monthly. Today, curl is an open source project maintained by a community of voluntary members globally.
What curl Does
curl’s command line tool is known among developers for its ability to perform quick HTTP requests to make your testing and debugging tasks easier. By running simple commands using your computer’s terminal, curl can perform GET, POST, or other HTTP requests supported by the specified API.
While users interact with curl’s command line tool the most, the hardest part of the work is handled by its development library called libcurl, an immense library that supports the majority of codebases. This core part of the curl project works behind the scenes, handling requests and responses.
How to Use curl
To use curl command lines, you need a CLI to execute them—Mac Terminal for macOS, Command Prompt for Windows, or Bash for Linux—and an internet connection so that your machine can complete them, show the result, and read the response, where present.
Each curl command follows a straightforward syntax:
curl [OPTIONS] [URL]
In this scheme, [URL] is the endpoint to which you are sending the request, whereas [OPTIONS] is the command that determines the type of HTTP request you want to make. The default format is a dash followed by a letter (eg -d) or, in a more verbose form, a double dash followed by a word (eg --data). Additionally, you can specify HTTP methods using the -X [METHOD] syntax (eg -X POST).
Here are a few curl commands every developer should know.
Sending GET Requests
The most common form of HTTP request is a GET request. It’s most often used to retrieve data—whether as text, image, or file—from the specified address. The default syntax for a curl GET request command is curl [URL]. No option is specified here because GET is the implied default behavior defined for a curl command.
Using the sample URL https://dummyjson.com/products, a GET request performed to this endpoint with curl looks like this:
curl https://dummyjson.com/products
The result for this command looks something like this:
However, not all GET requests are this easy to perform, though. Some require parameters, and while they can be specified in the URL, it’s easier to use curl’s appropriate syntax. Add -G to specify a GET request followed by -d "parameter1=value1" for as many parameters as required. The syntax should look like this: curl -G -d "parameter1=value1" -d "parameter2=value2" https://yoururl.com/yourendpoint.
If you send a GET request to that same URL as before but set the limit number to only 1, it should look like this:
curl -G -d "limit=1" https://dummyjson.com/products
curl uses this command to form the complete URL https://dummyjson.com/products?limit=1 using the base URL and the parameter you’ve set to send a GET request.
This would be the output:
While curl is powerful, it doesn’t format the output it receives from the request. On Mac, you can prettify the response by adding | json_pp to your command line:
Windows doesn’t come with such a pre-installed tool, so you need to install a tool like jq and use its shortcut | jq to get pretty formatted JSONs.
Also, note that you can handle curl requests using your built-in programming language. For example, you can read more about managing GET requests using PHP on this blog.
Sending POST Requests
Another frequently used HTTP is POST, which sends and updates data to a server.
The base syntax for a POST request with curl is curl -X POST [URL]. This type of request sends a POST request without the body of the data, which may be useful when the request is being made solely to update the status of an item in the database. For example, using the sample URL https://httpbin.org/anything, you can perform a simple POST request with the following command:
curl -X POST https://httpbin.org/anything
On most occasions, you need to upload data. To do so, start by telling curl how to read the body of the data you’re sending by specifying its format. Add a header with the command -H 'HEADER: VALUE' and then insert the body of your request using the data command -d 'VALUE'. The complete syntax is curl -X POST -H 'HEADER: VALUE' -d 'VALUE' [URL].
If you want to send a JSON body containing a first and last name, you can use Content-Type to define your content type to JSON format and define a simple JSON as your data:
curl -X POST -H "Content-Type: application/json" -d "{
"FirstName": "Joe",
"LastName": "Soap"
}" https://httpbin.org/anything
Here’s the output for the preceding command:
As you can see, the data field contains the body of the data you sent in your request.
From version 7.82, curl also introduced the --json shortcut, which simplifies your POST requests with a JSON body to curl --json '[JSON BODY]' [URL]. The command to perform the same task as before using the --json shortcut would be the following:
curl --json '{"FirstName": "Joe", "LastName": "Soap"}' https://httpbin.org/anything
Or if you have a long JSON body that you’d prefer to save as a local file, you can choose to upload that instead using the curl --json @[FILENAME].txt [URL] syntax:
curl --json @file-name.txt https://httpbin.org/anything
You have several options for uploading data to your server. For example, you may need a different format, or you may want to upload files to the web. You can read more about POST requests with curl here.
Connecting through Proxies
Many developers use proxy servers for improved security. Just like a firewall, a reliable proxy server safeguards its users and their network from potential threats on the internet. It also offers enhanced privacy.
To use a proxy server correctly, you need the following information about your proxy:
- The protocol, such as
http:// - The host, usually four numbers separated by dots, such as
71.00.00.00 - The port, normally a four-digit number, such as
0000 - Credentials (optional) in the form of a username and a password
The syntax of your proxy URL follows the scheme [PROTOCOL]://[[USERNAME]:[PASSWORD]][HOST]:[PORT] or, if the proxy is not protected by credentials, [PROTOCOL]://[HOST]:[PORT].
Once you have the proxy URL, you can set it up on curl using the command -x. Note that this is different from -X, which refers to a custom HTTP request method. (Remember, curl is case-sensitive.) You can also use the more verbose option --proxy if you find it easier to remember. Either command should be followed by the proxy URL and the final URL.
Considering the mentioned points, your curl command to set up a proxy URL should look similar to this:
curl --proxy "http://71.00.00.00:0000" https://httpbin.org/ip
Your response should contain an origin object, and its value should correspond to your proxy’s host IP address:
{
"origin": "71.00.00.00"
}
This means that the website thinks that the request comes from your proxy’s IP address rather than your computer’s.
For more about how to use curl with proxies, check out this blog.
Using Headers
As you might have noticed, some HTTP requests require additional information to complete a task correctly. For example, you used the Content-Type header in your POST request earlier to let curl know how to read the body of the data you uploaded.
Headers, which are always preceded by the command -H, can also be used on several other occasions.
One common header type is Authorization: Bearer [TOKEN], which specifies a bearer token to authenticate and access a protected URL. Another commonly used header is Accept: application/json, which specifies a JSON response preference.
If you’d like to receive a list of the response headers for a URL, you can use the -I curl command in your terminal, like so:
curl -I https://dummyjson.com/products
You can also use -i to show the response to the request along with the headers.
You can read more about headers in curl on this blog.
Saving Outputs
Sometimes, reading data is not enough, and you need to save those outputs somewhere on your computer. As previously mentioned, curl is capable of handling data that comes in forms other than text. For example, it can download or upload entire files.
By using the -o command, you can specify the name and location of the file you want saved. For instance, if you’d like to save the content of this page in TXT format to your desktop, you could use this command:
curl -o /Users/User/Desktop/file.txt https://brightdata.com/blog
Run this command and open your desktop to see your newly saved file.
Uploading Files
Just as curl is capable of downloading files from the web, it also allows you to upload them. To let curl know you’re not uploading data in textual form but rather as an actual file, you have to place the symbol @ before the file name.
Many developers make the mistake of using the -X POST flag alone, thinking it is the right command for uploading files to a server using curl. However, file uploads typically require the multipart/form-data content type as the files are attached as part of a form. So you need to set the correct header type along with the -X POST flag. The -F flag sets the correct content type header, and the -X POST flag automatically—which results in a shorter, simple command for you.
With the -F flag, you can submit a form by specifying the values of the form fields. To attach a file as a form field, use the @ symbol before the file name.
The correct syntax is curl -F "file=@[FILE PATH]/[FILE NAME].txt" [URL], where file is the name of the file field in the form. For example, assuming that you would like to upload the file you downloaded earlier to a dummy URL like https://example.com/upload, your command should look like this:
curl -F "file=@/Users/User/Desktop/file.txt" https://example.com/upload
If you would like to upload more than one file, you can repeat the -F command syntax as many times as you’d like before the definition of the URL.
curl Use Cases
Because of its high compatibility and versatility, curl can be used in a wide range of different scenarios.
The most common use of curl is as an HTTP client for doing API testing, interacting with web services, and performing different HTTP operations such as GET or POST. Numerous programming languages also use libcurl to perform HTTP requests internally.
While curl is primarily centered on performing HTTP requests and retrieving their responses, it can also be combined with other tools or scripts for parsing and processing received content. For example, it can retrieve the HTML content of a web page as part of web scraping.
Advantages of curl
curl offers many advantages. Most notable are its high compatibility rate, which makes it usable by almost any device, and its syntax, which is simple to pick up and master.
curl is also feature-rich, versatile, fast, and particularly lightweight, which allows it to perform tasks without overloading your CPU.
Conclusion
In this article, you learned more about curl’s history, the way it works, its use cases, and its advantages. You also saw how to perform some of the most basic commands with curl.
And if you’re interested in quicker ways for web scraping, be sure to check out the Bright Data Web Scraper API. Sign up now for a free trial!
No credit card required
