Skip to main content

Queries

The main way to use crul is through authoring and running queries. These queries are composed of one or more stages, "piped" together using the double pipe || syntax.

Let's look at an example to better understand how queries are constructed.

Query Grammar​

Queries can be broken up into 3 distinct parts, the query itself, the individual stages, and the command run in a stage. A stage is defined by one of the commands in the crul query language.

Each stage runs sequentially, and operates on the results of the previous stage.

The command is structured as follows, the command name, ordered arguments, and flags.

  • Command name: The command name identifies which command to run, for example the api command or the find command.
  • Ordered arguments: Many commands take one or more ordered arguments. These arguments generally consist of a target column or destination to interact with, filter, or modify.
  • Flags: Flags are constructed using the --flag value syntax. Flags can be used to give fine grained control over how a command executes.

Example query​

api get https://... --useragent "custom" /* stage 1 */
|| find "keyword" /* stage 2 */

Stage 1​

In the above query, stage 1 is described by a command, and any arguments and flags that determine how to execute the stage. In this example, the command is api, the first argument is get (a REST method), the second argument is https://... (the endpoint to request), and the --useragent is a flag that in this case will set the useragent of the API request.

Stage 2​

Once stage 1 has completed, the results are "piped" (||) to stage 2. Stage 2 is described by a command (find), and a single argument ("keyword"). This stage will search for matches to the provided key word.

Additional stages​

We can add additional stages to this query by adding || [command] [args] [flags]. See the commands reference for a full list of available commands. For example, from here, we could sort the results, find the min or max, add a timestamp, freeze our results to write them to a 3rd party destination, and much more!

Expanding stages​

Many commands are simple "reducing" type commands, for example, sort or find will take the data that is piped to it from a previous stage and transform it in some way. However other commands can be used in expanding stages, which take the data piped to it from a previous stage and operate on each row in an expanding fashion. Let's look at a more concrete example.

api get https://... /* stage 1 */
|| api get $url$ /* stage 2 */

Stage 1​

In the above query, stage 1 will make a GET request to the provided address (https://).

Let's say the result set after stage 1 completes looks like this:

{
  "account": "0987654321",
  "url": "https://.../asdf-1234-qwer"
},
{
  "account": "1234567890",
  "url": "https://.../zxcv-5678-tyui"
}

Stage 2​

Stage 2 is an example of an expanding stage. The $url$ represents a token that will be replaced based off of the matching key value(s) from the results "piped" in from stage 1. The query processor will recognize that stage 2 includes a token, and will run through each result in the "piped" in results to replace $url$ with the url value from the previous results.

Stage 2 will go from:

api get $url$

to

Stage 2 Command 1:

api get https://.../asdf-1234-qwer /* see url value in the first result from stage 1 */

Stage 2 Command 2:

api get https://.../zxcv-5678-tyui /* see url value in the first result from stage 1 */

These commands will run in parallel, and when both complete, the results of each command will be merged and stage 2 will be considered complete.