API pagination

A common pattern when interacting with APIs is dealing with paginated results, although there are various approaches to pagination, many common pagination designs can be handled by the api command.

The three main flags to be aware of are --pagination.next, --pagination.url, --pagination.max.

NOTE: These flags are only available for the api command.

--pagination.next

The --pagination.next flag represents the key in the API's response that contains a pointer to the next page of results, this can be a url, a hash, id, offset, or another value.

See further examples below.

--pagination.url

The --pagination.url flag is an optional flag that can be used to create a tokenize-able URL to get the next page of results. This is almost always used in conjunction with the --pagination.next flag, which represents a hash, id, or other value that can be used to get the next page of results.

See further examples below.

--pagination.max

The --pagination.max flag is an integer value that sets the maximum number of paginations that a stage will make. This flag's default value is 5. Set this value to 0 for no limit on the number of allowed paginations.

Examples

Example 1

As a simple example, let's assume the --pagination.next key contains a complete URL to the next page of results:

{ "results": [{ "key1": "value1", "key2": "value2"}, ...], "nextPage": "https://{api}/ASDFGHJKL123456789" }

In tabular form:

results.key1	results.key2	nextPage
value1	value2	https://{api}/ASDFGHJKL123456789
...	...	...

If we set the --pagination.next flag like this, --pagination.next "nextPage", then the pagination will sequentially request the url contained in the nextPage key until the nextPage key is either empty, missing, or the maximum number of pages is hit (see the --pagination.max flag).

Example 1 runnable query

api get https://pokeapi.co/api/v2/pokemon
  --pagination.next "next"

This query will retrieve the first 5 pages (because the default --pagination.max is 5) of results from the https://pokeapi.co/api/v2/pokemon endpoint.

Another example query using the --pagination.max flag would be:

api get https://pokeapi.co/api/v2/pokemon
  --pagination.next "next"
  --pagination.previousCommandIndex 3

This query will retrieve the first 3 pages (because --pagination.max is set to 3) of results from the https://pokeapi.co/api/v2/pokemon endpoint.

Example 2

Often an API response will include a pagination token or offset value that can be used to construct a URL for the next set of results.

Here are some results containing a nextPage key with a hash value that can be used to get the next page of results.

{ "results": [{ "key1": "value1", "key2": "value2"}, ...], "nextPage": "ASDFGHJKL123456789" }

In tabular form:

results.key1	results.key2	nextPage
value1	value2	ASDFGHJKL123456789
...	...	...

For this example we will need to set both the --pagination.next flag so that we know which key contains the nextPage hash, and the --pagination.url flag to construct a URL containing the token.

We can set --pagination.next "nextPage" and --pagination.url "https://{api}?nextPage=$pagination.next$"

The command will now sequentially request the URL contained in the --pagination.url flag after replacing the $pagination.next$ token in the URL with the value retrieved using the key from --pagination.next.

In this example, the URL after token replacement would become https://{api}?nextPage=ASDFGHJKL123456789.

The command will sequentially paginate until the --pagination.next value is either empty, missing, or the maximum number of pages is hit (see the --pagination.max flag).

Example 2 runnable query

NOTE: This query will require you to first configure a Twitter API credential.

api get https://api.twitter.com/2/tweets/search/recent?query=NASA&max_results=100
  --bearer "$CREDENTIALS.twitter_bearer$"
  --pagination.next "meta.next_token"
  --pagination.url "https://api.twitter.com/2/tweets/search/recent?query=NASA&max_results=100&next_token=$pagination.next$"

This query will retrieve the first 5 pages (because the default --pagination.max is 5) of results from the https://api.twitter.com/2/tweets/search/recent?query=NASA&max_results=100 endpoint.