Tokenization

Tokenization is an important concept in the crul query language. Tokenization allows for usage of previous results in the execution of a stage. Tokens in the crul query language are represented by a leading $ and a closing $.

Examples

Flag Example

A token can be used in place of a flag value in order to dynamically set it based on previous results.

devices || head 5 || open https://www.example.com --device $name$

This query will first get a list of available devices. The resulting data set will include a name column with the device name. This device name can be passed into a flag for the open command (among others). We can dynamically set the device name in our preceding stages using the name column value.

name	userAgent	...
iPhone	...	...
iPad	...	...
Blackberry	...	...
...	...	...

The next stage will use the head command to reduce the number of devices in our data set to 5.

The next stage (open) will demonstrate the important concept of expansion with tokenization. the stage will go through each row of the previous devices stage, and replace the $name$ flag value, with the name value in the previous results row.

The end result of this query is opening the same webpage (https://www.example.com) once for every available device. This query could be made immediately useful to check that a webpage contains the same content across different devices.

Argument Example

A token can be used in place of a flag value in order to dynamically set it based on previous results.

seed '[{"url": "https://www.example.com"}]' || open $url$

This query will first create an artificial data set using the seed command that contains a url key and value.

The next stage (open) will go through each row of the previous results, and replace the $url$ flag, with the url value in the previous results row.

Advanced Example

A token can be used as a part of a value (argument or flag), and just the token value will be substituted. This is powerful for constructing URLs or other values based off of a static value and previous results.

range 1 5 page || open https://www.example.com?page=$page$

This query will first create a data set of 4 rows each containing a single column page with incrementing values.

page
1
2
3
4

The next stage (open) will go through each row of the previous range stage, and replace the $page$ token, with the page value in the previous results row.

After the $page$ tokens have been replaced with the page value, the open command will open the following URLs:

https://www.example.com?page=1
https://www.example.com?page=2
https://www.example.com?page=3
https://www.example.com?page=4