<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>Andrew Ridgway's Blog - Business Intelligence</title><link href="http://localhost:8000/" rel="alternate"></link><link href="http://localhost:8000/feeds/business-intelligence.atom.xml" rel="self"></link><id>http://localhost:8000/</id><updated>2023-10-18T20:00:00+10:00</updated><entry><title>Metabase and DuckDB</title><link href="http://localhost:8000/metabase-duckdb.html" rel="alternate"></link><published>2023-10-18T20:00:00+10:00</published><updated>2023-10-18T20:00:00+10:00</updated><author><name>Andrew Ridgway</name></author><id>tag:localhost,2023-10-18:/metabase-duckdb.html</id><summary type="html">&lt;p&gt;Using Metabase and DuckDB to create an embedded Reporting Container bringing the data as close to the report as possible&lt;/p&gt;</summary><content type="html">&lt;p&gt;Ahhhh &lt;a href="https://duckdb.org/"&gt;DuckDB&lt;/a&gt; if you're even partly floating around in the data space you've probably been hearing ALOT about it and it's &lt;em&gt;"Datawarehouse on your laptop"&lt;/em&gt; mantra. However, the OTHER application that sometimes gets missed is &lt;em&gt;"SQLite for OLAP workloads"&lt;/em&gt; and it was this concept that once I grasped it gave me a very interesting idea.... What if we could take the very pretty Aggregate Layer of our Data(warehouse/LakeHouse/Lake) and put that data right next to presentation layer of the lake, reducing network latency and... hopefully... have presentation reports running over very large workloads in the blink of an eye. It might even be fast enough that it could be deployed and embedded &lt;/p&gt;
&lt;p&gt;However, for this to work we need some form of conatinerised reporting application.... lucky for us there is &lt;a href="https://www.metabase.com/"&gt;Metabase&lt;/a&gt; which is a fantastic little reporting application that has an open core. So this got me thinking... Can I put these two applications together and create a Reporting Layer with report embedding capabilities that is deployable in the cluster and has a admin UI accesible over a web page all whilst keeping the data locked to our network?&lt;/p&gt;
&lt;h3&gt;The Beginnings of an Idea&lt;/h3&gt;
&lt;p&gt;Ok so... Big first question. Can Duckdb and Metabase talk? Well... not quite. But first lets take a quick look at the architecture we'll be employing here  &lt;/p&gt;
&lt;p&gt;&lt;img alt="Duckdb Architecture" height="auto" width="100%" src="http://localhost:8000/images/metabase_duckdb.png"&gt;&lt;/p&gt;
&lt;p&gt;But you'll notice this pretty glossed over line, "Connector", that right there is the clincher. So what is this "Connector"?. &lt;/p&gt;
&lt;p&gt;To Deep dive into this would take a whole blog so to give you something to quickly wrap your head around its the glue that will make metabase be able to query your data source. &lt;/p&gt;</content><category term="Business Intelligence"></category><category term="data engineering"></category><category term="Metabase"></category><category term="DuckDB"></category><category term="embedded"></category></entry></feed>