Exporting results

Exporting results to 3rd party data lakes or other destinations is straightforward with the help of the freeze command. With the freeze command, we can easily push the results of our query directly to a preconfigured 3rd party store, or save the results locally to a file.

Results that are saved locally or in read/write stores (ex. Amazon S3, Google Cloud Storage, Azure Blob Storage) can be retrieved and pushed into a query pipeline using the thaw command.

Example: Freezing to a local file

devices
|| freeze mylocalfile

This query will save results from the previous stage to a local file set by the first argument. If results have already been written to a local file with the same name, the results will be overwritten, otherwise a new file will be created. You can use tokens in constructing names for uniqueness or filtering.

After freezing, the stored results can accessed using the thaw command.

thaw mylocalfile

All locally frozen results can be found in the cellar.

Example: Freezing to a Kafka topic

devices
|| freeze --store "kafka-prod" --kafka.topic "mytopic"

This query shows how to export results to a (preconfigured) 3rd party store. Unlike in the local file query, there is no argument provided to the freeze command, instead a --store flag is set to a preconfigured store.

In this query the --store was configured with the name kafka-prod and is a Kafka type store. The freeze command also sets the --kafka.topic flag which is a store specific flag (kafka.*, splunk.*, etc.) to control destination settings, the --kafka.topic flag will override the default configured Kafka topic.