Blog

- March 16, 2015

A Web API corresponds to an application programming interface that can leverage the Web technologies to execute processing and manipulate data. Modern Web APIs follow the REST principles to be as easy to use and lightweight as possible. Typically it provides a set of resources that exchange representations.

Far from being a high level guide, we want to provide here a practical and technical guide describing how to design Web APIs and pointing out how to avoid pitfalls along the way. It’s based on our experience and feedback when implementing Web APIs.

Before diving into the way to design a Web API, let’s recall some common concepts and principles of REST, on which Web APIs are based.

Concepts

REST (REpresentational State Transfer) is a style of architecture targeted at distributed systems in the context of Web technologies. Whereas it doesn’t depend on the HTTP protocol, the latter is particularly suitable to implement these concepts. We will focus here only on the concepts that will be useful to design a Web API.

The central element of REST is the resource. We can see a resource as a programming element that manages a kind of data and a state, and provides some processing on this kind of data. Several resources can manage the same data kind (for example, one for handling a list of elements, and one for a single element).

A resource is accessible on the Web from an URL and we can interact with it using a predefined set of methods to get its state, update it or execute processing. When used with HTTP, we can use all the methods (also called verbs) the protocol provided. We will describe later how to use them correctly following the REST approach.

The formats of exchanged data correspond to representations. The latter provide for the managed data kind. We can notice that a representation defines a structure of data and isn’t linked to a particular format such as JSON, XML or YAML. This concept is called variant.

The following figure describes these concepts and the relationships between them:

REST concepts

Defining resources and methods

Now we have seen all these high-level concepts. Great, but you probably wonder how to use them to actually build your Web API. You’re right! It’s not so simple and you need more details. We will focus on this all along this section.

When designing an API, we first need to define all the entities we want to implement, then the resources to manage them. We used to define several resources to handle each entity kind:

  • A resource to get the list of elements of this kind and to add new ones,
  • Another to manage a particular element of this kind,
  • And resources that manage no state and provide specific processing logic.

A resource is attached to a URL in order to make it accessible from the Web. We commonly call this URL a resource path. Like all URLs, they are hierarchical, and this aspect can help us define relationships between managed entities.

There are several ways to send parameters to resources:

  • Within the resource path itself:
    Such parameters are called path variables. They correspond to parameters that identify the resource. We commonly define them when attaching a resource to a path. For example, the identifier of an element: /myentity/{entityid}. Most REST frameworks know how to extract the parameter based on previous patterns.
  • Within the query string (what is specified after the ? within a URL):
    Such parameters are called query parameters and provide additional parameters. Be sure not to use such parameters for hints that should take place in the path variables.

Let’s take a sample with an entity Product. We define the following resources to manage entities of this kind.

A resource ProductListResource attached to a path /products or /products/. It provides two methods:

  • Method GET to return a list of products, either the complete list or a filtered list based on criteria provided as query parameters,
  • Method POST to add a new product.

A resource ProductResource attached to a path /products/{productId}. It provides three or four methods:

  • Method GET to return a particular product,
  • Method PUT to update the full content of a particular product,
  • Method PATCH to update a sub content (set of fields) of a particular product,
  • Method DELETE to delete a particular product

We can see that the resource ProductResource is under the resource ProductListResource if we see at corresponding path level. This means that processing of the first resource tackles the list of products and the other one manages a particular product within the list.

The following figure summarizes these resources and what they do:
rest-concepts-entity-management

Regarding the identifiers present within resource paths, we need to be careful to distinguish them with the internal identifiers. As a matter of fact, if we leverage the resource path hierarchies, identifiers allow to identify elements in the parent elements. For the sample of products, the value of the path variable productId correspond to the internal product identifier itself since we are located at the root of the path. Let’s take the sample of a two level hierarchy for a versioned element: /elements/{elementId}/versions/{versionId}. In this case, the value of the path variable versionId identifies the version for a value elementId.

When implementing Web APIs with HTTP, it is very important to take advantage of status codes. They allow to specify if processing works well or if something wrong occurs. Always returning a 200 status code and using the payload for the error details is considered an anti-pattern. However, in the case of an error and in addition to an error status code, the payload can be used to provide more details and eventually a structured error message.

The (well-known) status codes for successful requests are:

  • 200 (OK): everything works well, and the response contains a payload with the result,
  • 201 (Created): everything works well, and an element was created,
  • 204 (No content): everything works well, but the payload of the response is empty.

We will describe in more details the status codes in the case of errors later in the section .

Let’s focus now on the exchanged data structures.

Defining exchanged data structures

Within REST, data structures are called representations. They correspond to the structure of exchanged data independently of the format used. The latter is called variant and is closely linked to a media type. The following figure describes the relationships between these three concepts.

REST concepts / representations

We can distinguish two kinds of representations:

  • Structured representations. This corresponds to content that is structured to be understood by both client and server applications. On the Web, they classically use the following formats: JSON, XML and YAML.
  • Raw representations. This corresponds to both text or binary content that aren’t structured for application communication. They can be zip content, images, and so on.

In the following, we use the JSON format for our samples since it’s lightweight, self descriptive and easy to read. Moreover we will base our samples on the GitHub API (see: http://developer.github.com).

Regarding representations, there are some aspects we need to take care of:

  • how to define and choose the data contained in representations,
  • how to handle collections,
  • how to handle graphs of data,
  • how to handle relationships between data within representations.

Let’s start with the first item. Representations correspond to data we exchange with the API consumers. That’s why we’re free to choose the data they contain for each method of our resources. Be careful not to directly expose your underlying business objects though. A good approach consists in defining different entities and populate them according to your needs.

Moreover, input and ouput representations don’t need to be necessarily the same. It seems obvious but it’s not. Take the sample of a resource that manages a particularly element, for example an issue element from the GitHub API. Do we need to use the same representations for the GET method (retrieve issue hints) and the POST method (add issue hints)?. The answer is no. Hints like a field lastUpdated can be deduced on the server-side and not directly updatable by the end user. Following content describes the input representation:

{
   "title": "Found a bug",
   "body": "I'm having a problem with this.",
   "assignee": "octocat",
   "milestone": 1,
   "labels": [
      "Label1",
      "Label2"
    ]
 }

and this one the output representation:

{
   "id": 1,
   "url": "https://api.github.com/(...)/issues/1347",
   "html_url": "https://github.com/(...)/issues/1347",
   "number": 1347,
   "state": "open",
   "title": "Found a bug",
   "body": "I'm having a problem with this.",
   "user": {
       "login": "octocat",
       "id": 1,
       (...)
   },
   "assignee": {
       "login": "octocat",
       "id": 1,
       (...)
   },
   "milestone": {
       "url": "https://api.github.com/(...)/milestones/1",
       "number": 1,
       "state": "open",
       (...)
   },
   "created_at": "2011-04-22T13:33:48Z",
   "updated_at": "2011-04-22T13:33:48Z",
   (...)
 }

The last bullet is about the way to handle collections. You should simply say: use arrays. Yes, you’re of course right, but we could also have a need to have additional metadata about the collections: number of elements, current page of data, number of elements within the current page, and so on. As you can see, it’s tied to pagination and its’ up to you to choose the best approach for your use case. The following sample describes a representation containing a collection of issues using the array approach:

[
  {
  "id": 1,
  "url": "https://api.github.com/(...)/issues/1347",
  "html_url": "https://github.com/(...)/issues/1347",
  "number": 1347,
  "state": "open",
  "title": "Found a bug",
  "body": "I'm having a problem with this.",
  "user": {
     "login": "octocat",
     "id": 1,
     (...)
  },
  (...)
  },
  (...)
  {
     "id": 5,
     "url": "https://api.github.com/(...)/issues/1349",
     "html_url": "https://github.com/(...)/issues/1349",
     "number": 1349,
     "state": "open",
     "title": "Found another bug",
     "body": "I'm having another problem.",
     "user": {
        "login": "octocat",
        "id": 1,
        (...)
     },
     (...)
   }
]

We could adapt this representation to contain the pagination hints, as described below:

{
  "number": 23,
  "page_number": 2
  "number_per_page": 10,
  "list": [
     {
        "id": 1,
        "url": "https://api.github.com/(...)/issues/1347",
        "html_url": "https://github.com/(...)/issues/1347",
        "number": 1347,
        "state": "open",
        "title": "Found a bug",
        "body": "I'm having a problem with this.",
        "user": {
            "login": "octocat",
            "id": 1,
            (...)
        },
        (...)
    },
    (...)
    {
        "id": 5,
        "url": "https://api.github.com/(...)/issues/1349",
        "html_url": "https://github.com/(...)/issues/1349",
        "number": 1349,
        "state": "open",
        "title": "Found another bug",
        "body": "I'm having another problem.",
        "user": {
           "login": "octocat",
           "id": 1,
           (...)
        },
        (...)
     }
   ]
}

As you can see in the samples above, the data can contain several kinds of data and really correspond to data graphs with specific depths. The following lists these different kinds:

  • Fields with primitive types. The value of the field corresponds to a primitive type.
  • Fields with composite types. This corresponds to fields with inner fields
  • References to other elements. This corresponds to relationships between elements. They can be displayed as a reference or embedded to simulate a composite type, but the element has its own lifecycle.

For each kind, we can have single or multiple cardinality with null values allowed or not. Such rules are defined by the back-end itself but need to be checked when adding or updating data.

Following code describes this aspect:

{
   "id": 1,
   "number": 1347,
   "state": "open",
   "title": "Found a bug",
   "body": "I'm having a problem with this.",
   "user": {
      "login": "octocat",
      "id": 1,
      (...)
   },
   "assignee": {
      "login": "octocat",
      "id": 1,
      (...)
   },
   "labels-ref": [
      "https://api.github.com/(...)/labels/bug",
      "https://api.github.com/(...)/labels/enhancement"
   ],
   (or)
   "labels": [
      {
         "url": "https://api.github.com/(...)/labels/bug",
         "name": "bug",
         "color": "f29513"
      }
   ],
   "milestone-ref": "url": "https://api.github.com/(...)/milestones/1",
   (or)
   "milestone-ref": {
      "url": "https://api.github.com/(...)/milestones/1",
      "number": 1,
      "state": "open",
      (...)
   }
}

Most of time, the structure of the returned data is hard-coded for Web APIs. Moreover, some Web APIs allow to specify which data we want in the returned data graph. We can distinguish two approaches:

  • Specify the fields we want to get in the request response. This can be done using a query parameter containing the list of field names. The OData specification uses the query parameter $select for this.
  • Specify the sub element to load and integrate in the request response. This can be done either using headers (like with Parse) or query parameters. The OData specification uses the query parameter $expand for this.

The following URL describes how to use this feature with OData:

http://services.odata.org/V4/Northwind/Northwind.svc/Customers?$select=CustomerID,Orders&expand=Orders

To end this section, we will give some best practices and errors to avoid regarding representations:

  • Be careful not to duplicate data. As a matter of fact, some data can be deduced from the path variables at the resource paths or from the security context. In general, we dont need to put it again within the representations. For example, it’s not necessary to send the identifier of an element if it’s already present as path variable.
  • It’s not also necessary to put the parent element (in the case of parent child relationships) if we use resource paths providing hierarchical levels. For example, for a resource path like this: /elements/{elementid}/subelements/{subelementid}.
  • We should always refer to hints defined in the resource paths. It mainly applies to element identifiers to prevent from identifier usurpation and also to parent element references to prevent from re-attaching element to another one if not allowed

The last two points can correspond to potential data security issues of your API and should be handled with care.

Filtering data

Returning all data of a collection in a one shot is not a good idea. It can correspond to a lot of data and it’s not necessary (exploitable) by the end user. Paging is a good approach in such use cases. Paging can be controlled by a set of query parameters:

  • The size of elements in a page. The API should return the first page when not specified.
  • The number of the current page. The API should support a default value when not specified.

The following request describes the way to provide these parameters to the API using URL:

https://api.github.com/(...)/issues?page=2&page_size=10

Regarding pagination, Github provides an interesting approach based on the header Link that provides the previous and next pages. The representations are in this case simple arrays of data. Following code describes the content of this header:

(...)
Link: <https://api.github.com/(...)/issues?page=2>; rel="next",
 <https://api.github.com/(...)/issues?page=5>; rel="last"
(...)

Querying (also called filtering) is an important part of Web APIs, since we can be interested only in a subset of data. The below approaches can be implemented for this aspect:

  • Provide query parameters to filter the list of data to be returned. This typically applies to an HTTP method GET,
  • Provide a query parameter to identify the subset of data to be returned. This typically applies to an HTTP method GET,
  • Provide a query as request payload to identify the subset of data to be returned. This typically applies to an HTTP method POST,
  • Leverage the resource paths,
  • Leverage the navigation links within data.

The first one simply gives parameters to filter the returned data, as described below:

http://(...)/issues?state=open&label=bug

The second one simply uses a query parameter that contains an expression corresponding to the query string itself. The expression syntax depends on the Web API, but commonly applies to the element targeted by the resource path. For example, we can call it query and use the following expression:

http://(...)/issues?query=name%20eq%20'test'

The third differs from the first one in the way that the payload of the request is used to specify the query content. The REST API of ElasticSearch works like this, as described below:

{
   "match" : {
      "message" : {
         "query" : "this is a test",
         "operator" : "and"
      }
   }
}
 

The two last ones are more limited but can be very powerful to provide a simple criteria to apply directly within the resource paths. The Github API uses this approach to get list of issues:

  • Resource path /issues lists all issues across all the authenticated user’s visible repositories including owned repositories, member repositories, and organization repositories,
  • Resource path /user/issues lists all issues across owned and member repositories for the authenticated user,
  • Resource path /orgs/:org/issues lists all issues for a given organization for the authenticated user,
  • Resource path /repos/:owner/:repo/issues lists all issues for a repository.

We can use the resource paths to navigate through the returned data graph. This corresponds to implicit queries, since the inclusion within a parent element automatically handles the filtering. With the issue sample, we could have something like that:

http://(...)/user/issues/1/user
http://(...)/user/issues/1/labels
http://(...)/user/issues/1/labels
http://(...)/user/issues/1/milestone/creator

For a reminder, here is the content of a Github issue:

{
   "id": 1,
   "url": "https://api.github.com/(...)/issues/1347",
   "html_url": "https://github.com/(...)/issues/1347",
   "number": 1347,
   "state": "open",
   "title": "Found a bug",
   "body": "I'm having a problem with this.",
   "user": {
      "login": "octocat",
      "id": 1,
      (...)
   },
   "labels": [
      {
         "url": "https://api.github.com/(...)/labels/bug",
         "name": "bug",
         "color": "f29513"
      }
   ],
   "assignee": {
      "login": "octocat",
      "id": 1,
      (...)
   },
   "milestone": {
      "url": "https://api.github.com/(...)/milestones/1",
      "number": 1,
      "state": "open",
      "title": "v1.0",
      "description": "",
      "creator": {
         "login": "octocat",
         "id": 1,
         (...)
      },
      "open_issues": 4,
      (...)
   },
   "comments": 0,
   "pull_request": {
      (...)
   },
   "closed_at": null,
   "created_at": "2011-04-22T13:33:48Z",
   "updated_at": "2011-04-22T13:33:48Z",
   "closed_by": {
      "login": "octocat",
      "id": 1,
      (...)
   }
}

We notice that we can combine approaches based on the resource paths with the one using a query parameter. Here is a sample from the GitHub API:

http://(...)/user/issues?state=open&label=bug

As we saw, some APIs always return the same data structure (it’s the case of the Github one) but other allow to select the exact data to get. This can be efficient when the amount of received data can be huge (list of elements with a lot of fields). Such Web APIs leverage headers or query parameters to select the fields to be present in the response representations.

Handling lists

We need to take care of the way to handle list efficiently. As a matter of fact, we probably don’t want to send all the list contents to update them. We should create a resource with a set of methods to manage the list itself:

  • An HTTP method POST to add an element to the list
  • An HTTP method DELETE to remove an element from the list

The following figure summarizes the aspect:

Resources for entity sublist management

Another aspect of how to handle sublists is: should we embed the complete element content in the sublist or only the references to them? In fact, it depends on what we want to do but it’s clear that only the identifiers / references of sublist elements are required when managing them using the approach described above.

This resource has a path under the element resource. For example: /products/{productId}/categories or /products/{productId}/categories/.

Handling errors

The status code is the way to specify to the end user if the processing succeeds or not. As we mentioned earlier, always returning a status code 200 whenever anything happens, and describing the error in the response content corresponds to an anti-pattern.

We must leverage HTTP status codes to notify end users of errors when calling a resource. The response content can provide, in addition, more details of the error(s). This content can be text or structured (like JSON).

Never forget to add this, since this is a key part that will make your APIs more robust. Feel free to leverage support provided by the technologies you use to implement server-side processing!

You can find out below the main status codes that are commonly used to notify errors.

The first family (status code 4xx) corresponds to errors that can occur if the end user provides wrong data or executes methods in a wrong context:

  • 400: the most generic one telling that the request sent isn’t correct. An error message should necessarily be provided to give more details.
  • 401: this occurs when we aren’t authenticated unauthorized.
  • 403: this occurs when we haven’t enough to access a resource.
  • 404: probably one of the most well-known error. It occurs when trying to use data that doesn’t exist. It can be used for different HTTP methods, not only GET. For example when trying to update or delete data that doesn’t exist.
  • 405: this occurs when trying to use an HTTP method that isn’t supported for a resource.
  • 406: the server resource can’t return the expect content specified in the header Accept.
  • 409: conflict for optimistic locking.
  • 412: this occurs when we aren’t in the right context to execute processing.
  • 415: this occurs when the structure of the values sent isn’t correct (for example a JSON content that isn’t well-formed).
  • 422: some REST APIs don’t like to use this code since it doesn’t come from the HTTP specification itself, but from WebDAV. It tells that the validation of the data provided by the end user aren’t correct and can’t be processed. It doesn’t correspond to format errors but to value ones. However it’s widely used in Web APIs and describes well the problem.

The second family (status code 5xx) corresponds to errors that can occur on the server side. We expect them not to occur, but we need to take them into account within the clients that use the Web API. Here is a list of the most common ones:

  • 500: something wrong within the server side processing when trying to execute the request.
  • 501: the corresponding server side processing isn’t implemented for the request sent.
  • 503: the Web API isn’t currently available at the moment.

Summary

In this article, we described the foundations for designing a Web API. We will see in a further articles some more advanced aspects. So please stay tuned!

CTA_free trial_3