Query

We call SQL queries that actually process data simply queries. There are two main types that we highlight below: Streaming or Continuous Query and Query on Materialized Views. In the UI Query tab or when listing queries with the CLI, only long-running streaming queries will appear because queries on Materialized Views are snapshot queries and are not long-lived.

Streaming or Continuous Query

A streaming or continuous query in DeltaStream reads from one or more Streams and/or Changelogs, processes the data according to the query logic, and generates one or more Streams or Changelogs. Streaming queries are continuous, meaning that once they are started, they keep running until explicitly terminated. There are two types of streaming queries in DeltaStream:

• Persistence streaming (continuous) queries: These queries persist the result of the query in the streaming storage by creating new Streams or Changelogs.

• Interactive streaming (continuous) queries: Once started, these queries continuously run and stream the results of the query to the user.

Streaming queries start in the NEW state, and once they are fully created and operating, they transition to the RUNNING state at which they process records from the source Relation(s) and write into the sink Relation. A streaming query that fails to continue to operate ends up in the ERRORED state.

Queries may automatically get restarted if an underlying problem has been detected by the system, e.g. network issues. In failure scenarios, queries are restarted from the last known state and continue to operate normally. Unrecoverable failures are reported back through the DESCRIBE QUERY command or by clicking on the state on the Web App.

Queries that are terminated by the user are considered finished and are fully cleaned up from the system. See Query for a list of statements that can start a query for processing records within a Store.

Query on Materialized Views

Once you build a Materialized View in DeltaStream, you can query the content of the view using standard SQL queries. Unlike continuous queries, queries on Materialized Views compute the answer based on the content of the view at the time of the query issuance and return the results. These queries behave like the queries on tables in traditional relational databases.