FlareDB: An Apache Beam native Streaming Database
Introduction
FlareDB is built on a unified worldview where stream processing and database storage are not two separate systems, but two sides of the same coin. Traditional data architectures separate stream processing engines from databases. Streaming systems such as Apache Flink and Google Cloud Dataflow excel at stream processing, while databases provide durable storage and query capabilities. This separation requires pipelines to move data from one system into another before it can be queried, analyzed and served. FlareDB eliminates this divide by building its execution engine directly around the Stream-Table Duality.
Streams and tables are fundamentally the same data expressed in two different relative states:
Streams are data in motion: They represent a continuous, append-only history of transformations. A stream captures the evolution of data over time the individual delta changes, transactions, and events flowing through the system. Transformations on data produce streams.
Tables are data at rest: They act as the conceptual resting place where those changes accumulate. A table allows you to observe the dataset as a whole at a specific point in time. It represents the current materialized state computed from the stream’s history.
This is a fundamental idea in databases. Every change to a table is first recorded as an append-only log (or write-ahead log), while the table itself represents the accumulated state of those changes. Streams and tables are simply two different views of the same data.
FlareDB applies these principles to Apache Beam.
Repository : https://github.com/flare-db/flare-db
If you find the project interesting, consider giving it a star 🌟
How does a pipeline run in FlareDB
FlareDB uses a graph based execution model, where the pipeline is a DAG of worker nodes and runner nodes connected by PCollection edge. The Executable graph is formed after greedy fusion of raw pipeline proto received from the runner sdk. The goal of an executable graph is to divide responsibilities between worker SDK harness and runner and FlareDB orchestrate the execution of nodes, handle execution ordering and PCollection materialization between stages.
Runner nodes are the ones that need to be executed by the runner and worker nodes, ones that need to be executed by sdk harness. The runner and worker nodes in the graph are connected by PCollection edge which holds the metadata about producer and consumer nodes, its PCollection information, coders and sub component coders. For example, the executable graph of a WordCount Pipeline[2] looks like this [3]. FlareDB has utils to visualize the every executable graph of every pipeline it receives.
1. Node Scheduling
To execute the DAG, FlareDB uses a scheduling approach using topological sorting to identify the right execution order of nodes in the DAG based on its PCollection dependencies. The executor identifies the root node of the graph with an in-degree of zero (no incoming data dependencies, Typically it’s an impulse transform) and starts the execution queue.
When a node completes executing and produces Pcollections, the runner identifies all downstream consumer nodes waiting on that data and adds them to the queue for the next level of execution. This step-by-step progression continues until all nodes in the graph have been traversed and executed.
Since every node executes only after its input PCollections become available, scheduling naturally follows the PCollection dependency rather than an explicit execution plan.
2. The Element Store
To let different nodes exchange data without tight coupling, FlareDB manages a centralized Element Store. Every PCollection produced by node execution is materialized into the Element Store using its unique PCollection ID. Instead of passing records directly from one stage to another, execution stages communicate by reading and writing these materialized PCollections.
The Element Store is backed by Tonbo[4], An embedded LSM based storage engine. Rather than treating intermediate PCollections as transient data exchanged between execution stages, FlareDB persists them as columnar datasets that can be efficiently read, grouped, queried, and reused by downstream stages. This makes storage an integral part of the execution model. For example, consider a Beam PCollection<KV<String, String>> with the elements:
Each row represents one Beam KV element. The pcollection_id is the unique name of the PCollection that KV belongs to. Every PCollection is persisted in this way after decoding the elements sent by sdk harness or the runner transform produced PCollections.
When a node begins execution, it scans its required input PCollections from the Element Store. After completing its work, the node materializes its output as another PCollection, which becomes available for downstream consumers.
Consider the GroupByKey transform. In traditional distributed runners, GroupByKey typically involves a distributed shuffle, where keyed elements are redistributed across workers so that all values associated with the same key can be grouped together.
In FlareDB, however, the input PCollection has already been materialized inside the Element Store. The runner can perform the grouping directly over the persisted data and materialize the grouped result as a new PCollection.
This separation between execution and storage is fundamental to FlareDB’s architecture. Nodes compute and produce new PCollections, while the Element Store provides the shared interface through which those PCollections are exchanged, persisted and queried.
3. How nodes are actually executed
When the executor picks a node to run, the execution path splits depending on the node’s type:
A. For Worker Nodes (SDK Harness)
A worker node is nothing but a set of transforms and it’s the worker’s responsibility to execute those transforms. Worker nodes handle DoFn execution and run at an in-process environment, FlareDB coordinates the execution using Beam’s Control and Data channels.
When a worker node is picked up for execution, the runner first prepares a bundle descriptor that defines the stage boundaries, transforms, and coders required to encode and decode data and sends to the worker via control channel, once the worker accepts the descriptor and sends acknowledgement response, the runner sends process bundle request and begins streaming the stage’s input PCollection from the Element Store to the worker over Beam’s Data channel.
As the worker processes the input, it streams its output bytes to the runner. FlareDB decodes these elements and materializes them as a new PCollection in the Element Store. This newly materialized collection then becomes the input for downstream nodes in the execution graph.
The runner considers the stage complete after the worker has successfully finished execution and transferred all the output data via data channel.
B. For Runner Nodes
Runner nodes are Beam transforms that FlareDB executes natively inside the runner. Instead of sending work to the SDK harness, the runner reads the required PCollections from the Element Store, performs the operation, and materializes the result as another PCollection.
In Apache Beam, aggregation transforms are typically expressed as compositions of more primitive transforms. For example, Count.perElement() expands into:
Map(element -> KV(element, null))
↓
GroupByKey
↓
CombinePerKey(CountFn)
In a portable pipeline, When the SDK harness executes Count.perElement(), it first applies a keying operation to the input PCollection: KV.of(element, (Void) null) The resulting keyed elements are then passed to GroupByKey, which is a runner transform, and it requires elements to be transferred to the runner via data channel, grouped by key, and returned as grouped results before the final CombinePerKey operation can produce the count.
From Beam’s perspective this is a logical representation of the computation. However, from a database perspective, the same operation is fundamentally a grouped aggregation over persisted data. Since FlareDB persists PCollections as queryable columnar tables and the runner can directly execute the aggregation using query operations on persisted PCollections.
So, Technically:
Beam = logical programing model
FlareDB = execution engine
PCollections = persisted data
PTransforms = operations over persisted data
Tonbo = Storage Engine
Current Status & Roadmap
When I started working on FlareDB, it wasn’t even a database. The original goal was much simpler: build a lightweight Apache Beam runner. But building a complete Beam runner would involve dozens of components and could easily become an endless project, so I deliberately narrowed the scope.
The objective for v0.1.0 was straightforward: execute portable Beam pipelines with bounded sources on Global Window and implement one runner-native transform. FlareDB v0.1.0 achieves exactly that while establishing the foundation that future releases will build upon.
What’s Next:
Support for unbounded sources.
Introduce support for unbounded data by implementing watermarks, event time, windowing, and triggers.
Splittable DoFns:
Support for Splittable DoFns (SDFs). The current execution model runs a single in-process SDK harness, which limits support for SDF-based IO connectors. Future work will extend the runner architecture to support Beam’s dynamic work rebalancing model.
Stateful processing.
Implement the Beam State and Timer APIs to enable stateful processing.
More runner native transforms.
Expand the set of runner-native transforms beyond Impulse and GroupByKey, allowing more Beam operations to execute directly over persisted PCollections inside the runner.
materialized views with FlareDBIO:
Today, PCollections are materialized internally by FlareDB during node execution. In the future, FlareDB will expose this capability through a dedicated FlareDBIO transform, allowing pipeline authors to explicitly persist PCollections as materialized views.
Since FlareDB’s storage layer is powered by Tonbo, these materialized views can be stored as Arrow in-memory and persisted as Parquet files on local disk or object storage such as S3. Once materialized, they become durable datasets that can be queried directly for analytics, dashboards, and Agents without requiring a separate serving database.
Building the roadmap is going to be a long journey. FlareDB may have started with a single idea, but I hope it evolves based on the Apache Beam community feedback and participation. I want to build the rest of it with you. Come, share your thoughts, open issues, star the repo, contribute what you’d like FlareDB to do.
References
[1] FlareDB - https://github.com/flare-db/flare-db
[2] WordCount pipeline - https://github.com/flare-db/flare-db/blob/main/example/wordcount/src/main/java/com/flaredb/example/WordCount.java
[3] WordCount executable graph - https://github.com/ganesh-skumar/pipelinegraphs/blob/main/WordCountGraph.svg
[4] Tonbo - https://github.com/tonbo-io/tonbo
[5] Beam Model Rust - https://github.com/flare-db/flare-db/tree/main/beam-model-rs
[6] FlareDB runner Java SDK - https://github.com/flare-db/flare-db/tree/main/runner-sdk/java/flarerunner/src/main/java/com/flaredb/runner
[7] Maven Dependency - https://central.sonatype.com/artifact/com.flare-db/flaredb-runner
[8] Original work on Beam streams and tables - http://s.apache.org/beam-streams-tables
[9] Streaming Systems Book, written by Reuven Lax, Slava Chernyak, and Tyler Akidau.
