Application programming interfaces (APIs) define the standards and protocols that allow different software components to communicate with each other. They let applications request data and actions from independent systems.
APIs are everywhere. They underpin virtually all interactions with your devices and software. For instance, the apps on your phone use APIs to fetch data from their servers and then rely on separate APIs provided by iOS and Android to put the data on the screen, send it to you through push notifications, or share it with a contact.
Because there are so many forms of APIs, it can be confusing to understand how they fit together. What types of APIs are there? When does something qualify as an API? How can you create your own? In this comprehensive guide, you’ll learn the answers to all these questions.
What Are APIs?
The term application programming interface can seem obtuse, but it refers to a specific concept. At its simplest, an API is something that allows developers to program code that works with an application. It sets out an interface that facilitates interoperability or the rules, procedures, expectations, and standards that two or more systems must adhere to.
Let’s explore this idea with an example: consider an identity platform that exposes an API for registering new users. External applications can use the API to create users on demand, but for this to work, data needs to be in a format that the platform expects.
The API sets out these requirements: it could state that the client application must make an HTTP POST request to example.com/users; that Name, Email, and Password data fields need to be included; and that a JSON body will be issued in response, containing the new user’s ID. With this information, developers can use the API to successfully register new users.
Essentially, an API is a combination of the platform code that developers can call and the documentation that explains how to use it.
Why APIs Are Important?
APIs allow data to flow between systems. This allows the software to build upon other applications, creating more powerful solutions.
APIs are also essential to automation. By combining the capabilities of different APIs into one application, you can move data between systems as actions and events occur. Developers can write a few lines of code to implement complex processes that would otherwise require painstaking manual programming.
For example, web scraping is a complex task. To implement an effective web scraper, developers would need to create sophisticated logic for controlling a web browser instance, setting up geolocation proxies, and bypassing CAPTCHAs. By selecting an API, you can access all this functionality with a few network requests. Then you can use other APIs to manipulate the scraped data, analyze it, and send the results to a team member in your chat platform.
In addition, APIs are a business asset. Facilitating easy integration with other tools makes your platforms more appealing to customers. External developers are free to build their own solutions that are greater than the sum of their parts.
APIs are vitally important to today’s hyperconnected processes. Many technologies that are taken for granted are powered by an intricate web of APIs. For instance, online shopping usually involves payment collection, shipping requests, and email delivery APIs hosted by several independent vendors.
Types of APIs
Individual APIs offer different functions to match the service they’re a part of. For example, an identity management solution will offer very different API capabilities to those of a search engine scraping provider.
However, the technical characteristics of seemingly unrelated APIs are often very similar. A handful of different standards are used by most popular APIs, representing techniques that the software industry has found effective for integrating systems.
Let’s take a look at the different types of APIs:
#REST
Representational state transfer (REST) was first theorized by Roy Fielding in 2000 and is now used by most web services.
REST represents a system’s data as stateless resources that are mapped to HTTP URLs. HTTP methods (GET, POST, PUT, and DELETE) are used to retrieve resources and perform actions on them.
For example, a GET request to example.com/users/100 should return the following information about the user with ID 100:
{
"Id": 100,
"Name": "Example User",
"Email": "[email protected]"
}If you then issue a DELETE request to the same URL, the service should destroy the object.
REST is popular because it’s easy to implement, builds upon HTTP, and effectively models how many real-world applications handle their data. Interactions with many systems are often a combination of a verb (DELETE) and a noun (user), and this architecture can be directly expressed by REST.
#SOAP
Unlike REST, Simple Object Access Protocol (SOAP) is a formal specification for data sharing. All SOAP exchanges use the XML data format, whereas REST APIs may offer JSON, XML, CSV, or a platform-specific alternative. A simple SOAP API call could produce a response like this:
xml
<?xml version="1.0" ?>
<soap:Envelope
xmlns:soap="https://www.w3.org/2003/05/soap-envelope/"
soap:encodingStyle="https://www.w3.org/2003/05/soap-encoding">
<soap:Body>
<m:GetUserResponse>
<m:Id>100</m:Id>
<m:Name>Example User</m:Name>
<m:Email>[email protected]</m:Email>
</m:GetUserResponse>
</soap:Body>
</soap:Envelope>
The use of XML and the inclusion of protocol-specific attributes make SOAP more verbose than typical REST APIs. However, SOAP’s standardization benefits mean it’s favored by many large enterprises and commercial systems. The operations available in the API are explicitly defined by XML schemas. These describe the structure and data types of each request and response, reducing the risk of client-server mismatches from occurring.
#GraphQL
GraphQL is a relatively young technology for building manipulable APIs. It was developed at Facebook in 2012 and published openly in 2015.
GraphQL is designed to solve several of the challenges with REST and SOAP APIs. It simplifies complex queries by providing an expressive language that the client can use to extract data from the API. GraphQL lets you retrieve just the specific data fields you require instead of always providing entire objects. This prevents wasteful transfers of redundant data.
Here’s a simple GraphQL query to get a user’s email address:
{
user {
Email
}
}This query would produce the following response:
json
{
"user": {
"Email": "[email protected]"
}
}GraphQL is gaining popularity because it’s a more versatile option that better suits today’s highly connected applications, where small portions of data are independently fetched by several different components. However, implementing GraphQL can be relatively complex and is best handled using language-specific programming tools
#RPC
Remote procedure calls (RPCs) are a simple form of API. The technique refers to calling a remote function as if it were available locally. A basic network request causes the API server to perform the task and provide the result. The client isn’t exposed to the details of the network communication.
RPC APIs look similar to functional programming interfaces. Both the verb and noun will be included in the URL you call, such as example.com/deleteUser?User=100. This contrasts with REST, where you apply a verb to a specific noun (DELETE example.com/users/100). RPC more directly maps to code, whereas REST attempts to model your data structure.
RPC is easy for both the client and server to use. It’s a collection of URLs that accept various request parameters and send data in response. There is no standardization, however, and these APIs tend to be difficult for developers to explore. Whereas the endpoints of most REST APIs can be anticipated once you know the names of the service’s resources, and the URLs used for RPC will be unique to each platform.
Improvements in RPC are being made by modern projects such as gRPC. This is a framework that works with multiple programming languages and can be used to quickly define RPC API services that communicate with clients using protocol buffers, Google’s performant approach to serializing structured data.
System APIs
REST, SOAP, GraphQL, and RPC APIs are used in the context of network communications between systems. Other APIs exist for different kinds of integration, such as the system interfaces that let applications access your device’s functions.
These APIs are provided by operating systems such as Windows, Android, and iOS. They’re exposed by programming frameworks and SDKs that developers can call from their code. System APIs allow convenient access to features such as notifications, launcher icons, media playback, and device sensor access without needing programmers to write low-level code.
Programming Language APIs
Similarly, programming languages and dependencies have their own APIs. The modules included in a language’s standard library represent an API. Third-party packages you install in your project are also APIs, and the components you write are connected together using defined interfaces.
The API is the distinction between the internal workings of a system, which could change at any time, and the stable outside interface that integrators rely upon. Methods and functions that are marked as public in your codebase create an API that other code can consume.
Synchronous and Asynchronous APIs
APIs can be either synchronous or asynchronous. A synchronous API will return the result of the requested operation immediately, while an asynchronous API may continue execution after the data exchange is complete.
For a data collection API, requesting the currently collected data would be a synchronous task that always returns the data retrieved to date. Requesting a new data collection scrape could be asynchronous because the process is likely to take a long time to complete. It’s more efficient for the API to terminate the communication immediately after informing the client that the collection has been scheduled.
A Detailed Look: How APIs Work
Each of the common API types has its own grammar. For instance, REST works with objects and verbs, while GraphQL offers a more versatile solution that’s client-centric instead of server-centric. Let’s take a more detailed look at these two options:
# REST
REST uses HTTP method verbs to perform actions on resources. The most common methods are GET, POST, PUT, and DELETE:
– GET /users/100 returns the ID with user 100.
– POST /users creates a new user.
– PUT /users/100 updates a user by ID.
– DELETE /users/100 deletes a user by ID.
This illustrates the basic REST syntax. The URL provides the ID of the object you’re interacting with and the plural noun it’s an instance of. The HTTP method used by the client determines the action that will be performed.
When an action requires additional data to complete, it’s supplied as the payload of the HTTP request. For example, when creating a user with POST /users, the body will contain the username and password to assign.
The API responds to each request with an HTTP status code that describes its outcome. For instance, a 404 Not Found response to GET /users/100 indicates that user ID 100 doesn’t exist, while 202 Accepted for DELETE /users/100 means the user has been successfully deleted.
#GraphQL
In comparison, GraphQL is a different approach to APIs. It’s marketed as [“a query language for your API,”](https://graphql.org) hinting at the more advanced functionality it supports. Whereas REST often wastes bandwidth by including unwanted object properties, GraphQL lets you ask for the exact data you want.
APIs using GraphQL are authored as services, which define the endpoints clients can call. A service is a typed schema for an entity. Each field in the schema has a specific data type assigned:
type Team {
id: ID
name: String
}
type User {
id: ID
name: String
email: String
team: Team
}
You can fetch data from schemas using queries:
{
user(id: 100) {
email,
team {
name
}
}
}
This example query could return the following data:
{
"email": "[email protected]",
"team": "Example Team"
}
The fields in the query are backed by resolvers. When the query is executed, the resolvers are responsible for producing a value for each field. In the previous example, the team resolver would extract the name property from the team assigned to the requested user instead of returning the entire object.
GraphQL also provides a consistent way to update data using mutations. Mutations are similar to queries, but they cause a state change on the server. Mutations are implemented by defining a function for each field in your services. This function is responsible for saving a new value to the field.
GraphQL APIs are usually created by adding a GraphQL client library to your project. These tools allow you to conveniently generate GraphQL schemas from your existing ORM models and other code.
How to Integrate APIs
An API integration describes the process of adopting an API within a software system. Although the API offers ready-to-use functions, you still have to write some custom code to use them within your project.
A typical API integration involves the following steps:
1. Evaluate the options available: To begin, you need to assess the different APIs that solve your use case and identify the best one for your product. This includes looking at how good the quality of the documentation is, whether there’s an active community using the API, and how quickly the maintainers respond to support requests, security issues, and bug-fix reports.
2. Sign up for the service and request an API key: Some APIs are publicly exposed and require no authentication, but most APIs will require you to sign up and obtain an API key once you exceed some basic usage thresholds. API keys should be treated as sensitive values and stored securely—do not hard-code them into your project’s code. The key authenticates you to the service and identifies your application for rate limiting and usage tracking purposes.
3. Look for an API client library for your programming language: You can integrate APIs by making direct network requests using your programming language’s HTTP library. However, many API vendors also offer client libraries and SDKs that wrap around the API to provide a more convenient programming language. Choosing to use a client library when one’s available will further simplify your implementation and shield you from any breaking changes to the underlying API.
4. Write your code: After identifying your library, it’s time to write the code that interacts with the API. You’ll need to configure the library you’re using with the API key you’ve obtained. Your code may also have to set any config parameters the service requires, such as the data center region and preferred response format.
5. Test your API integration: Finally, test your integration to make sure it works as you expected. Your testing should include checks of error handling routines, such as what happens if the API’s unavailable. This will help ensure your application is resilient when the service is offline.
It’s also important to consider the security implications of integrating an API. Although third-party APIs can simplify key development tasks, you should act cautiously when sending user data to an external service. Will the platform be able to meet the same security standards as your own? If you can easily replicate an API’s functionality, it can be safer to build your own implementation within your application.
Real-Life API Example
Ready to use an API? To quickly experiment with web APIs, you can use HTTP tools you’ve already got on your machine, including curl and wget. If you prefer graphical interfaces, Postman is a good alternative.
Getting Fake Data with Faker
The Faker API project is a popular collection of APIs that return randomly generated data on a variety of subjects. The Faker API is often used during product development to populate interfaces before the real backend is available.
The Faker API uses REST principles, with the noun at the end of the URL defining the type of data to generate:
$ curl https://fakerapi.it/api/v1/books?_quantity=1
{
"status": "OK",
"code": 200,
"total": 1,
"data": [
{
"id": 1,
"title": "Duck and a pair of.",
"author": "Jessyca McKenzie",
"genre": "Sit",
"description": "ALL RETURNED FROM HIM TO YOU,\"' said Alice. 'I wonder how many miles I've fallen by this time, as it can be,' said the Cat. '--so long as I used--and I don't take this child away with me,' thought.",
"isbn": "9796054956226",
"image": "http://placeimg.com/480/640/any",
"published": "2010-09-14",
"publisher": "Quod Enim"
}
]
}
Scraping Search Engine Listings with Bright Data
Bright Data provides a comprehensive suite of SERP APIs and proxy APIs, and it’s a commercial platform that you need to register for:
To start using it, sign up for a free trial, then follow the documentation to add the SERP API to your account. Then you need to enable asynchronous mode in the API’s Advanced Options:
After activating the API, you can submit a POST request to gather search engine results:
$ curl -i "https://brightdata.com/api/serp/req?customer={CUSTOMER_ID}&zone={CUSTOMER_ZONE}" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer {API_TOKEN}" \
-d '{"country":"us","query":{"q":"apis"}}'
...
x-response-id: s3wt2t...Generate an API token in your account settings, then substitute it into the command instead of `{API_TOKEN}`:
You can find the respective values of the other placeholders for your account, {CUSTOMER_ID} and {CUSTOMER_ZONE}, in the SERP API playground.
This example query uses the API to schedule a US Google search for apis. Copy the value of the x-response-id response header displayed in the command’s output. You can use this value to retrieve the SERP result after it’s been generated. Wait a minute, then issue the following request:
$ curl "https://brightdata.com/api/serp/get_result?customer={CUSTOMER_ID}&zone={CUSTOMER_ZONE}&output=json&response_id={RESPONSE_ID}"Replace RESPONSE_ID with the value you copied previously. The data generated by your search will be displayed in your console.
These endpoints are an example of an RPC API. If the API is RESTful, the URLs will look like this:
POST /api/serp/request and GET /api/serp/results/{RESPONSE_ID}.
Summary
APIs are clearly defined interfaces that can be used to reliably connect different software components together. However, what constitutes an API can be confusing because there are so many different forms, variants, and use cases.
In general, an API is a mechanism that code can and should use to access functions implemented by other code. The API is supported by the remote system’s developer and documented with instructions on how to use it. This creates a contract between the service and the client applications that use the API. If the client sends data in the expected format, it’s guaranteed to get a response with a predictable structure.
APIs simplify the implementation of specialized functionality within your systems. You can let sector experts do the hard work for you and then plug their platforms into your code. Try using Bright Data’s suite of SERP APIs andproxy APIs to perform your web scraping tasks.
