Solving Datomic BI Integration: Why We Built Plenish

When Eleven approached us a few years ago to help build the Business Intelligence (BI) component of their product, they faced a significant challenge: their data resided in Datomic, a powerful immutable database, while their BI tool of choice, Metabase, did not support Datalog. Eleven, an accounting platform built with Clojure and Datomic, needed an efficient way to bridge the gap between these technologies.

Over the years, we have engineered a robust and maintainable solution, allowing seamless integration between Datomic and Metabase. This journey led to the development of Plenish, a Change Data Capture (CDC) solution tailored for Datomic.

The Requirements

To integrate Metabase as the BI interface within Eleven’s product, we needed to develop a glue layer that would:

1. Make data available to Metabase.
2. Provide suitable access control, ensuring that Metabase’s access control aligns with Eleven’s user data boundaries.
3. Be easy to maintain.
4. Be performant enough.
5. Be easy to deploy.

The Challenges of Using Metabase with Datomic

Metabase is a powerful dashboard with a beautiful UI and easy connectivity to many RDBMS databases. However, Eleven's data resides in Datomic, and Metabase does not support Datalog.

Attempt 1: Custom Metabase Driver

We first developed a custom Metabase driver to communicate with Datomic. This worked reasonably well but was costly to maintain, mainly because Metabase has very little regard for maintaining compatibility with third party driver vendors.

Attempt 2: Datomic Analytics via Presto

In 2019 the Datomic team came out with a Datomic driver for Presto, which is bundled with Datomic pro, and thus allows SQL access to Datomic’s triple-store data. Given that we could rely on the Datomic team to maintain this, it seemed like a logical choice to adopt it. However, this has not been a great experience.

• Outdated Presto Driver: Presto has since forked into PrestoSQL and Trino, with Trino gaining traction. However, a Trino-compatible Datomic-Analytics driver has yet to emerge, blocking us from upgrading Metabase, which has dropped support for Presto in favor of Trino.
• Operational Overhead: Datomic-analytics connects via a peer server. Running a peer server alongside Presto introduced complexity, requiring additional maintenance and frequent restarts to recognize new databases.
• Performance Bottlenecks: Queries on large datasets suffered from severe slowdowns, leaving us with few optimization options beyond increasing CPU and RAM.
• High Resource Consumption: The combined resource usage of the peer server, Presto, and Metabase exceeded that of the entire application stack.
• Limited Access Control: There is a desire to integrate with other data sources at the SQL layer, which Presto in theory is a good fit for, but the Metabase driver for Presto only allows accessing a single “catalog” at once, at least when using the Metabase query builder. On the other hand Presto has access to all datomic databases that the peer server has access to, with no finer grained access control, meaning we can’t provide people with raw SQL access, and need to rely on the somewhat thin assurances of Metabase’s query builder to ensure no unauthorized data access.

Plenish: A Solution for Datomic

With all the above hard-won lessons, we adopted a third approach: mapping Datomic data to relational tables in a way similar to Datomic Analytics. However, instead of having "virtual tables" backed by Datomic, we actually copy the data into a relational database. Our choice of database has been PostgreSQL, because it’s a fantastic all-around relational database that is well supported and well understood. With this goal in mind, we started a new Lambda Island project called Plenish.

Plenish synchronizes Datomic to an RDBMS in real time by reading the Datomic transaction log and converting each event into a corresponding SQL command. With this simple but challenging idea, Plenish satisfies all five important requirements elegantly:

• Seamless Metabase Integration: Since data is copied to PostgreSQL, Metabase can query it effortlessly.
• Minimal Maintenance: Plenish relies on Datomic’s stable transaction log API, ensuring long-term reliability.
• Optimized Performance: PostgreSQL is performant enough for OLAP workloads, eliminating the query speed issues seen with Presto.
• Natural Access Control: Each Datomic database is synchronized to a corresponding PostgreSQL database, aligning naturally with Metabase’s user access model.
• Simple Deployment: Plenish functions as a library, enabling straightforward integration.

Future Directions

We have recently enhanced Plenish with two major improvements:

1. Support for DuckDB – an embedded analytical database, reducing initial setup complexity while improving query performance.
2. Extension Points for Contributors – making it easier to integrate new data warehouse adapters within the open-source community.

As we continue refining Plenish, we recognize that similar CDC solutions exist in the RDBMS world—such as Debezium, Airbyte, and Qlik Replicate—but the Datomic ecosystem still lacks a mature, widely adopted alternative. We welcome collaboration from engineers interested in evolving this space.

If you’re working with Datomic and facing BI integration challenges, we’d love to hear from you. Let’s build the future of Datomic analytics together.