Introduction to APIs

I’ve written this post to help some of my colleagues, who traditionally don’t work with APIs to understand some foundations. They are designers, product managers etc. If you’ve used APIs before, this post is probably not for you ;-).

An “Application Programming Interface” (API) allows two software systems to communicate in an agreed manner, set out by open specifications.

We are going to focus on basic concepts to allow us to be able to communicate effectively about APIs, in an engineering product team. There will be a plethora of links to other sites with better content than this if you want to learn more.

Types of API

There are many types of APIs on the internet. For example

• RESTful (Representational State Transfer)
• Soap (Simple Object Access Protocol)
• RPC (Remote procedure call)
• GraphQL

Each type of API has pros and cons, and also rules you need to follow. This post will not get into all that, but some of the links above do. If you’re interested, checkout the links and go down the rabbit hole. The rest of this post will focus on RESTful APIs, since that is currently what my team is working with. We are going to use examples from the Prolific API.

Note, Prolific is an online platform that allows you to conduct research with 130,000+ vetted participants. This gives you insights you can rely on.

Endpoints

We often talk about endpoints when it comes to APIs. An endpoint is a location that allows you to send or retrieve data. Using the API mentioned above, we could send a POST request to the https://api.prolific.com/api/v1/studies/ endpoint to create a Study resource on the Prolific Platform. It’s like the address of the API on the internet.

Each endpoint allows you to do certain things with it.

Methods

There is a language to talk to APIs. It’s worth understanding these in the context of what you’re building, as it facilitates knowledge transfer.

Let’s agree on some terminology here.

• A document or resource is a bunch of information related to a single “thing”. For example, a Study document would have a title, description, who created it, and a bunch of other details.
• A collection stores many documents/resources. A collection keeps a related set of documents together. So you would have a study collection, a user collection, etc.

For those two categories, we can do the following things with them.

• GET - Fetches a collection of documents, or retrieves a single document.
• POST - Creates a document in a collection.
• PUT - Replaces an existing document in a collection.
• PATCH - Partially updates an existing document.
• DELETE - Deletes an existing document in a collection.
• OPTIONS - Fetches the available operations you can do on an endpoint.

These are called HTTP Verbs, and we are going to see this in action, next.

Requests

At this point, we now actually want to communicate with the API. This is done via an HTTP Request. When you search google, that is generating an HTTP Request. When you log into Twitter (if you can), that is generating an HTTP Request, and so on. In fact, everything you are doing on the internet is generating requests in some way.

In its basic form, a request takes an endpoint, a method, and a payload of some kind and sends that to the API.

Rather than using a browser, we will use a command line tool to see more of what is happening under the hood for this example.

This HTTP Request is asking the API to respond with information about my user account.

curl -X GET \
  'https://api.prolific.com/api/v1/users/me/' \
  --header 'Accept: */*' \
  --header 'User-Agent: curl' \
  --header 'Authorization: Token <redacted>'

With this command you can see a few things happening:

• curl is simply a command line browser, akin to Chrome/Safari etc.
• -X GET is the HTTP verb/method being used. As mentioned above, that means we want to retrieve this resource from the API.
• --header, think of this as extra information on an envelope you send in the mail. It helps the payload (the letter you’re sending), get to the destination. These are called Headers, and you can define a lot more than described in the example above.
  • In this case, we define the following.
  • Accept defines what we will accept as content. This could be text/html, application/json, etc. The server will respond in turn, with the content type used. You will see this in a moment.
  • User-Agent allows us to tell the server how we are communicating with it. This helps identify libraries being used, applications etc. It’s basically a string of text.
  • Lastly, in this example, we define the Authorization header, which allows us to gain access to information via the API. If we did not define this header, we would be denied access to the data.

At this point, we are now sending a semantically correct request to the Prolific Platform API.

Responses

Once a request is received by the API, the API needs to respond. It’s only polite to do so!

Based on the example HTTP Request above, this is the response we have received from the API.

< HTTP/2 200
< date: Fri, 18 Nov 2022 13:51:04 GMT
< content-type: application/json
< content-length: 1802
< allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
< x-frame-options: DENY
< vary: Origin, Cookie
< x-content-type-options: nosniff
< referrer-policy: same-origin
< cross-origin-opener-policy: unsafe-none
< via: 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
* Connection #0 to host api.prolific.com left intact
{"id":"<redacted>","email":"[email protected]","date_joined":"2022-04-21T10:29:49.967000Z","first_name":"Ben","last_name":"Selby","name":"Ben Selby","username":"[email protected]","user_type":"Researcher","currency_code":"GBP","balance":0,"available_balance":0,"is_email_verified":true,"billing_address":null,"has_password":true,"datetime_created":"2022-04-21T10:29:49","last_login":"2022-11-18T09:34:34","address":null,"fees_percentage":0.0,"service_margin_percentage":0.0,"fees_per_submission":0.0,"vat_percentage":0.2,"country":"GB","referral_url":"https://app.prolific.com/register/researcher?ref=redacted","vat_number":null,"email_preferences":{"bonus_payments":true,"just_published":true,"referrals":true,"marketing":true,"pre_publish":true},"terms_and_conditions":true,"beta_tester":false,"privacy_policy":true,"experimental_group":10,"representative_sample_credits":0,"redeemable_referral_coupon":null,"minimum_reward_per_hour":60000,"status":"OK","on_hold":false,"can_topup_3d":true,"has_answered_vat_number":true,"balance_breakdown":{"project_funds":0,"service_fees":0},"can_oidc_login":false,"topups_over_referral_threshold":false,"is_staff":true,"referral_incentive":{"minimum_topup":0.0,"recipient_credit":0.0,"recipient_rep_sample_credit":0.0,"referrer_credit":0.0,"referrer_rep_sample_credit":0},"needs_to_confirm_US_state":false,"mua_beta_user":true,"current_project_id":"redacted","has_accepted_survey_builder_terms":true,"can_cashout_enabled":true,"can_contact_support_enabled":true,"can_instant_cashout_enabled":false,"invoice_usage_enabled":true,"can_oidc_login_enabled":false,"can_run_pilot_study_enabled":false,"_links":{"self":{"href":"https://api.prolific.com/api/v1/users/redacted/","title":"Current"}}}%

This response can be broken down into the HTTP Response Headers, and the HTTP Body. Let’s start to understand some of the headers.

• HTTP/2 200 This gives us the HTTP Response Code of 200, which means “Successful/OK”. It’s worth understanding the Response codes and agreeing on what codes should be used for your API. This provides context to your API consumers on what is happening/happened with their request. The consumers of your API can understand what you’re saying just by the response code. For example, if you requested a document that didn’t exist, you would get an HTTP Response code of 404. I’m sure you’ve all seen a 404 page before.
• content-type: application/json. We talked about this above, this is the API telling us it’s responding to our request with JSON. It could have been any other format too.

After the headers, we can then see the JSON data we wanted. This could be HTML, XML, Text, JSON, or a raft of other things, such as image data etc.

We should all strive to make the API responses consistent across our API endpoints. This makes it easier for systems to talk to us in a consistent way.

Specifications

So, in the last section, we talked about consistency. Well, there are specification frameworks out there that help us do just that.

• JSON API
• OData
• Potentially an in-house specification

I would highly recommend following an existing specification, as there will be libraries out there to help both the providers of the API and the consumers of the API. Does it add value to your customers if you write an in-house specification?

Lastly, you can document your APIs, using something called an Open API Specification. This is a specification or blueprint that the API authors write, and publish to consumers. At the beginning of this post, I mentioned the Prolific API Open API Specification. This is defined in YAML (other formats are available), and we use a tool called Redoc to build a website we host showcasing how to use our API.

Connecting to APIs

So, we have gone into some key areas of building and talking to an API. More than likely, you’re wanting to integrate an API into your product somehow. There are several ways you can do this.

• Roll your own code to talk to the API.
• SDKs, such as the google/jsonapi Go library et al.
• Generic clients that build out code from your Open API Specification.
  • Within my team, as a proof of concept we built a generic client of our API using openapi-typescript-codegen.
  • This takes the Open API Specification mentioned above and then generates classes/functions to drop into your project to call the API.
  • In this case, it produced a TypeScript client we could start using straight away. All the boilerplate code was already built for us.

There are, as with everything, pros and cons for each approach. For example, writing your own gives you full control. By generating code from the specification you get to focus on your product and the value that brings, rather than writing standard code to hit an API. It’s a team decision on where the value is. Different teams will decide on different outcomes based on their experience and desired outcomes.

Summary

There is a lot more to APIs than what is mentioned above. For example, you could dedicate an entire post to Authentication, and the different types out there nowadays. However, the above should get you going at least.

I would like to say one last thing though. Nothing is perfect, and something is better than nothing.