Skip to main content

Expanding links from an API (Hacker News)

In this example, we will get back the last 10 items from the Hacker News API. Comments, stories, etc. are all considered items with a unique id and metadata defining if it is a comment, etc. We will first get the largest item (most recent) and use it to construct a range of items to query. We will then use this range to make an expanding stage that makes an api request for each item.

You may find that this query runs slowly as you increase the number of items. This due to the ~1 request per second default domain policy. The Hacker News API does not have a request limit, so we could update the domain policy for the API endpoint to allow more requests per second. See the domain policies documentation for more information.

Example: Hacker News Comments​

Full Query​

api get https://hacker-news.firebaseio.com/v0/maxitem.json?print=pretty
|| math "min = data - 10"
|| range $min$ $data$ item
|| api get https://hacker-news.firebaseio.com/v0/item/$item$.json
|| filter "(type == 'comment')"

Stage 1: Retrieving the current max item​

api get https://hacker-news.firebaseio.com/v0/maxitem.json?print=pretty

The first stage will make a get request to the maxitem endpoint, which we will use in later stages to construct a range.

Stage 2-3: Building a range of items​

...
|| math "min = data - 10"
|| range $min$ $data$ item

We want the most recent 10 items, so we will use the math command to subtract 10 from the max item value found by the previous stage (contained in data).

We will then construct a range from the calculated min value to the max value contained in data. This range will exist in a new column named item.

...
|| api get https://hacker-news.firebaseio.com/v0/item/$item$.json
|| filter "(type == 'comment')"

With our range of items, we will use the api command with an $item$ token used to pluck the item value from each row and use it to construct an HTTP endpoint. Each request will be made asynchronously (throttled/limited by domain policy and available browser workers) and we will then filter the results to only include items representing a comment.

We now have a data set of the latest comments on Hacker News.