COMP9321:
Data services engineering
Week 5: Designing RESTful API and RESTful API Client
Term1, 2021
By Mortada Al-Banna, CSE UNSW
Designing an API – who should you target?
Apps
API Developer
API
End Users
Your Application (with useful functions and data)
Apps
“App” Developer
2
Your Service Value Chain
Designing RESTful APIs
A well-designed API should make it easy for the clients to understand your service without having to “study” the API documents in-depth.
self-describing, self-documenting as much as possible
the clients are developers like yourself, so probably they would like to have an API that is easy to pick up and go
The RESTful service principles actually give us a straightforward guideline for designing the Web API …
“Clean, Clear, Consistent” are the key
3
URI design (= API endpoints)
Avoid using ‘www’, instead:
• https://api.twitter.com
• https://api.walmartlabs.com
Identify and “name” the resources. We want to move away from the RPC-style interface design where lots of ‘operation names’ are used in the URL
e.g., /getCoffeeOrders, /createOrder, /getOrder?id=123
Instead:
• Use nouns (preferably plurals) (e.g., orders)
• Walmart • LInkedIn • BBC
4
/items /items/{id} /people /people/{id} /programmes /programmes/{id}
5
URI Design …
Use of Query Strings. Use it when appropriate …
Search or Select
• /orders?date=2015-04-15
• /customers?state=NSW&status=gold • Twitter /friendships/lookup
• Walmart /search?query
Expression of the relationships between resources
• Facebook /me/photos
• Walmart /items/{id}/reviews
• /customers/123/orders vs. /orders?customer=123
Think: what is the expected resource as return in this URI?
URI design …
On the resources’ URIs … we add ‘actions/verbs’ Completes the endpoints of your APIs
Should it?
Note: PUT could also return 201 if the request resulted in a new resource 6
Decide How to Use the Status Codes
Using proper status codes, and using them consistently in your responses will help the client understand the interactions better.
The HTTP specification has a guideline for the codes …, but at minimum:
And utilise more of these, but restrict the number of codes used by your API (clean/clear)
RFC2616 (HTTP status codes) https://www.w3c.org/Protocols/rfc2616/rfc2616.html
7
Decide your response format
Should support multiple formats and allow the client content negotiation (JSON only?) Use simple objects.
A single result should return a single object.
Multiple results should return a collection – wrapped in a container.
8
To summarise so far:
GET /coffeeOrders/123
9
Success
Code = 200 OK
Send back the response (in the format requested by the client)
4xx Bad request e.g., 404 not found
5xx Processing error e.g., 503 database unreachable
Note: response body could contain detailed error messages
Failure
To summarise so far:
POST /coffeeOrders
May return:
• Location: …/coffeeOrders/123, or
• the updated object in the body, or
• both.
Success
Code = 201 Created
4xx Bad request e.g., 400 missing required field 5xx Processing error e.g., 503 database unreachable
Note: response body could contain detailed error messages
10
Failure
To summarise so far:
PUT /coffeeOrders/123
PATCH /coffeeOrders/123
May return (optionally)
• Location: …/coffeeOrders/123, or
• the new object in the body, or
• both.
Success
Code = 200, Code=204, Code =201
4xx Bad request e.g., 400 missing required field 5xx Processing error e.g., 503 database unreachable
Note: response body could contain detailed error messages
11
Failure
To summarise so far:
DELETE /coffeeOrders/123
May return (optionally)
Deleted resource in the body, or nothing …
Success
Code = 200, Code = 204
4xx Bad request e.g., 404 not found
5xx Processing error e.g., 503 database unreachable
Note: response body could contain detailed error messages
12
Failure
13
Taking your API to the Next Level
HATEOAS = Hypermedia As The Engine Of Application State
From Wikipedia: The principle is that a client interacts with a network application entirely through hypermedia provided dynamically by application servers. A REST client needs no prior knowledge about how to interact with any particular application or server beyond a generic understanding of hypermedia.
Think how people interact with a Web site. No one needs to look up a manual to know how to use a Web site … Hypermedia (i.e., documents with links to other links) itself serves as a self-explanatory guide for the users.
The HATEOAS principle aims to realise this in API design.
Not Using HATEOAS
Not implementing the links in REST API would look like this:
You assume that the client knows how to construct the next request path (i.e., combine /coffeeOrders and id:100) – maybe by reading your API document?
14
Using HATEOAS
15
16
Using HATEOAS
Use links to:
• help the clients use the API (self-describing as possible)
• navigate paging (prev, next)
• help create new/related items
• allow retrieving associations (i.e., relationships)
• hint at possible actions (update, delete)
• evolve your workflow (e.g., adding extra step in a workflow = adding a new link)
Standard link relations: http://www.iana.org/assignments/link-relations/link-relations.xhtml
Although the principle is well-understood, how HATEOAS links are implemented (i.e., how the links appear in the responses) is different from one implementation to another …
17
API Versioning
• When your API is being consumed by the world, upgrading the APIs with some breaking changes would also lead to breaking the existing products or services using your API.
• Try to include the version of your API in the path to minimize confusion of what features in each version.
Example: http://api.yourservice.com/v1/stuff/34/things
Standarised API specification
OpenAPI Specification (Swagger, swagger.io)
• a standard, language-agnostic interface definition to RESTful APIs
• both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection.
• “Auto-documentation” (from the written specification)
• “Auto inspection/testing interface”
• “Auto client code generation”
An idea that is increasingly becoming attractive … and this helps RESTful client application development (every aspect of the standard is to help API consumer understand and consume API quicker/less labour intensive way)
cf. SOAP/WSDL – the calculator service (http://www.dneonline.com/calculator.asmx)
18
Standarised API specification
Basic structure (what is in it …)
• A quick run through: https://swagger.io/docs/specification/basic-structure/
• Petstore.yaml (https://editor.swagger.io/)
19
20
Consuming a REST API
RESTful Service Client
•
The client in this situation is pre-written software program
Application
Web server
HTTP/HTML
REST requests/responses When necessary
HTTP/JSON (XML)
API
Full UI experience … Conventional
Web application
REST client
(API experience)
21
Application
(with useful functions and data)
Web and RESTful client to Services
So … an application may have ‘RESTful client’ embedded so that it can incorporate external APIs
• Understanding, getting to know an API quickly is also a “skill”
• Something like OpenAPI Specification could help here …
• Pure HTTP request/response (stateless) interactions with the API expected
Two types:
Simple programmatic interactions with the service
• Using so-called HTTP Client libraries
• Maybe less ”human-based interaction” with the response content
Increasing interactivity/responsiveness for the UI layer
• Increased interactivity with the ‘User’, not with the ‘API’
• Not drawing the whole page again
• Numerous JavaScript based frameworks (tool)
• Single Page Applications (SPA) 22
How to write Web REST API Client
What are the steps involved in writing client code for a RESTful Web API? (e.g., JSON) First, read and understand the “contract” – API documentation.
e.g., Twitter API Doc.
23
Restful Web Clients by Mike Amundsen
How to Write Web REST API Client
• •
The purpose of understanding the contract is for you to understand the following basic tasks that are common in all Web API client
Common Tasks:
• Recognising the Objects in HTTP responses
• Constructing Addresses (URLs) for interacting with the service
• Handling Actions such as filtering, editing or deleting data
Take the example from the book:
• http://rwcbook03.herokuapp.com/task/
• http://rwcbook03.herokuapp.com/files/json-client.html • https://rwcbook.github.io/json-crud-docs/
•
24
Restful Web Clients by Mike Amundsen
25
Restful Web Clients by Mike Amundsen
Use a client-API interaction pattern
Very basic “Single Page Application” structure …
A REST Client App could be designed to act in a simple, repeating loop that roughly goes like this:
1. Execute an HTTP request
2. Store the response (JSON) in memory;
3. Inspect the response for the current context;
4. Walk through the response and render the context-related information on the screen
JSON-client code (from the Amundsen example)
• A processing pattern for a typical listing item handling:
• A processing pattern for adding appropriate “actions” to each list item:
(All done via the basic interaction pattern mentioned above)
Simple programmatic interactions In Python: What can Requests do?
Requests will allow you to send HTTP/1.1 requests using Python. With it, you can add content like headers, form data, multipart files, and parameters via simple Python libraries. It also allows you to access the response data of Python in the same way.
This type interaction covers wider range of application scenarios where the client application does not necessary have human users (i.e., a complete automation)
Response content types of the API may matter (more choice favourable?)
26
The API by Restful Web Clients by Mike Amundsen
27
https://realpython.com/api-integration-in-python/
Simple programmatic interactions In Python: What can Requests do?
Requests will allow you to send HTTP/1.1 requests using Python. With it, you can add content like headers, form data, multipart files, and parameters via simple Python libraries. It also allows you to access the response data of Python in the same way.
This type interaction covers wider range of application scenarios where the client application does not necessary have human users (i.e., a complete automation)
Response content types of the API may matter (more choice favourable?)
Simple programmatic interactions
the json argument to post. If we use that, requests will do the following for us:
• Convert that into a JSON representation string, json.dumps()
• Set the requests’ content type to “application/json” (by adding an HTTP
header).
28
https://realpython.com/api-integration-in-python/
29
The API by Restful Web Clients by Mike Amundsen
Requests (check for status code)
• It is not always the case that things will go smoothly “Status code check”
Example:
resp = requests.get(‘https://api.example.com/stuff/’) if resp.status_code != 200:
# This means something went wrong.
raise ApiError(‘GET /stuff/ {}’.format(resp.status_code))
• Pay attention that the status code changes according to operation
thing = {“summary”: “someThing”, “description”: “” }
resp = requests.post(‘https://api.example.com/stuff/’, json=thing) if resp.status_code != 201:
raise ApiError(‘POST /stuff/ {}’.format(resp.status_code))
30
https://requests-cache.readthedocs.io/en/latest/index.html
Caching API Requests
• To implement caching, we can use a simple package called Requests-cache, which is a “transparent persistent cache for requests”.
• How does it work(Simple Approach):
import requests_cache
requests_cache.install_cache(cache_name=‘mystuff_cache’, backend=’sqlite’, expire_after=180)
• By default the cache is saved in a sqlite database. We could also use a python dict, redis, and mongodb
31
https://requests-cache.readthedocs.io/en/latest/index.html
Build your own API Library
• If you are developing a sophisticated application you need to move away from simple calls into constructing your own API library.
• This Also applies if you are the owner of the API so having an API library can make your API more usable.
32
Build your own API Library (Example)
import requests def get_stuff():
return requests.get(_url(‘/stuff/’)) def describe_stuff(stuff_id):
return requests.get(_url(‘/stuff/{:d}/’.format(stuff_id))) def add_stuff(summary, description=””)
return requests.post(_url(‘/stuff/’), json={ ‘summary’: summary,
‘description’: description, })
33
Restful Web Clients by Mike Amundsen
Coping with changes …
Software systems evolve over time … The APIs you rely on will too.
There is a principle that has been used by HTTP itself to evolve its versions without causing failure …
Must Ignore
HTTP directive “MUST IGNORE” = any element of a response that the receiver do not understand must be ignored without halting the further processing of the response.
HTML evolved over time using the similar approach … (“forgiving” browsers) Take V4. of the Amundsen’s API.
MUST IGNORE …
V1.
{
“links”: [ …], “data”: [ …]. “actions”: […] }
V2.
{
“links”: [ …], “data”: [ …]. “actions”: […], “extensions”: […] }
WITH response Do
PROCESS response.links PROCESS response.data PROCESS response.actions
END
(proceeds without breaking/haltering)
If Client wants to take advantage of the new feature, the client code will be updated, of course.
34
Restful Web Clients by Mike Amundsen
Useful Resources
• https://stackoverflow.blog/2020/03/02/best-practices-for-rest-api-design/
• https://realpython.com/api-integration-in-python/
• Book: RESTful Web Clients: Enabling Reuse Through Hypermedia, By Mike Amundsen
35
Restful Web Clients by Mike Amundsen
36
Q&A