"Unbundling BI" is absolutely where things are headed now that more people have agents, coding agents, agent computers to help with work.
I'm seeing this all up and down knowledge work tools. I've started treating email as a BI problem -- ETL it from Gmail and and create many different views into it and reports from it.
>"Unbundling BI" is absolutely where things are headed now that more people have agents, coding agents, agent computers to help with work.
Agreed. Coupled with bento[0] for slide decks, I think this type of project is a very welcome development, helping us move away from walled gardens and proprietary software suites.
Thanks for sharing this too. It reminds me of GistDeck a super old project that turned a gist into slides. Infinitely better than powerpoint or GSlides.
Not every visualized data point is tied to a binary decision.
e.g. when highlighting a quarterly revenue result, there is a big difference between missing the target by 1% or 50% (and of course the absolute number behind it is also important).
And on a larger scale, trajectory is also something that is easily visualized in graphs, and hard to boil down to a binary marker.
There certainly is a lot of "forced synesthesia", and more looking at charts than necessary out there, especially when decisions are then made more from the gut than based on the data one is claiming to base the decisions off, but a culture of binary-based decision making is just as bad of an idea.
I've worked in data for 15 years and I've never heard this argument. I love it :)!
Who knows, with AI this may be the future. Visualizations were created to help us understand data. Florence Nightingale published them "to affect thro' the Eyes what we fail to convey to the public through their word-proof ears."
Often a formula better represents a situation. But I'm not sure how much impact they'd have in spreading that understanding outside of a formula-minded audience.
I worked deep in big data briefly. Billions is too big for humans so we got creative how we visualized and filtered.
AI slop has a "smell" meaning there's an abstracted data filter that results in rose or poop output. Shakespeare variable says the name doesn't matter.
Sure, machines don't need visualization (although it seems that images are a surprisingly good medium to get context into agents), but humans still do. Mathematicians have plotted functions for a long time now. While the pure truth lies in the definition of f(x), the visualization helps the mind to grasp what it actually is. Humans will run companies for the foreseeable future, and so we will need visualization for the foreseeable future
Actually I think they do. Visualizations are a form of compression and focus.
All of my ai-driven plans use a tool I built to rasterize all text based visualizations. And I make the model present the rendered version to a clean room agent for brief-back. They go in circles until the diagram is good. Then, I instruct planning tools to always look at both the text and rendered version of all diagrams. I have found it to be surprisingly powerful.
Without guidance my agents tend to make me read CSVs.
Sure the data is correct, but it's obviously way easier for me to read it as a properly rendered table. Then you want to sort, group, aggregate the data. Visualize it many different ways if you will :)
Malloy is an alternative to dbt, they have a similar product called Malloyyo [1], and Publisher [3], you can also the chat with HN data demo they have [4].
DBTCharts says you can serve charts locally, but it seems they want you to use their hosting service in production. Whereas, Malloyyo / Publisher are free to use anywhere. If you like college football checkout how I visualize drive data in [2], made with Malloyyo.
Neat, but… I’ve built up skills for my agents to just generate SVG charts and embed them in the chat timelines. A library helps, sure, but you can get surprisingly good results that cover 80% of the timeline and bar chart visualizations without any libraries, and that goes for diagrams as well (mine generate .drawio XML)
Hi HN, I'm Dave the founder of Chartio (YC'10 now Atlassian Analytics), announcing today dbt Charts, an open source YAML dialect and tool for declaring and rendering dashboards.
When making dashboards with claude or other agents, a lot of free-form artifacts are created that makes it hard to audit and scale. dbt Charts is a simple YAML dialect that declares and renders a chart (think markdown but for dashboards). Along with dbt its Apache 2.0 and launched today.
We hope this language + AI help make the BI space more open with dashboards as concise auditable code. Would love any thoughts and feedback.
This is obviously a space we are very much interested in, so it is definitely nice to see approaches that attack the same problem. I haven't played with dbtcharts in depth yet but it looks very similar to DaC in principle, and also in the actual spec. I believe the industry definitely needs solutions like this to help get out of the legacy BI tools as Bİ is one of the biggest bottlenecks for AI adoption in large orgs.
Excited to see further competition in the space, nice launch!
It's neat but pretends to be more innovative than it actually is. BI has already been decoupled from everything else. People use AI to generate Excel and Power BI reports. YAML/XML/JSON - that doesn't matter. AI can generate whatever you instruct it to do.
So yes, a nice and logical development of dbt, but hardly as innovative as the blog post wants to sound. Nevertheless, I think it's a good idea that will be popular in certain circles. Hiring a professional data designer is a good idea - data visualization is very easy to get wrong.
I think I agree, but I'm not sure where Dave is claiming this is a paradigm shift in data visualization?
One of the key claims is actually that it's based on _old_ established visualization patterns from before BI tools were popular, and therefore results in consistently nice charts. The other key claim is just that it's simple and maintainable.
Sometimes tools & products can be useful as just well executed points in the known design space. You can already do all this with AI, but it's perhaps a little bit less nice and less maintainable.
AI can definitely generate whatever you tell it to do, but what to do with that artifact afterwards? If you want it to be reusable it needs to be a well-defined structured protocol/language. What good is a YAML describing a chart if all I can do is to give it back to the agent and say "read this and create a chart from it", the result will be vastly different from the chart you got the first time.
> the result will be vastly different from the chart you got the first time.
AI can generate reusable artifacts including definitions of the data sources, why not? Most BI dashboards are just XML or JSON (or YAML) files and include definitions of data sources (directly or via semantic models). I struggle to see how dbtCharts is different. Yes, your YAML schema is clean and nice, but that's because you're in the early stages :) Once you go through feature bloat, your YAML format will become much more complex.
AI can generate an XML/JSON/YAML definition of a BI report according to a spec and link the data source in whatever form it should be referenced in the file. For instance, here is a skills file for defining data sources (semantic models) in AI-generated Power BI dashboards: https://github.com/microsoft/skills-for-fabric/blob/main/plu...
Interesting, but I'm not sure how it's different from Observable Framework ( https://observablehq.github.io/framework/ ) for dashboards as code. I like that one because it's Markdown and JS instead of a language defined in YAML + templates
The use case that immediately came to mind for me was to use dbt charts as a rendering method for an MCP App.
You don't really want to give a LLM client code execution environment like Observable Framework does. dbt charts appear to validate the entire yaml input including the SQL being sent to the data source.
So I'm getting: a data vis layer that can be dynamically defined and rendered via LLM/MCP client without the headache of sandboxing a JS Runtime environment.
I like the observable framework too - very flexible, you can nail very specific visualization details by dropping down to JS/html.
Maybe this is targeted to pure SQL folks that can get lost in UI details.
PS: one issue that I have with observable framework is authentication and authorization , it requires some system to be built on top of it to handle authn.
very cool! i'd been thinking about this idea and found vega-lite to be an interesting take as well [0].
i'll compare and look at folding this into setoku for app generation [1]. right now apps are just html blobs your claude authors + a mechanism for populating them with live data. definitely hard to audit but very flexible for operators to claude together internal apps. anyway, the charts look decent but really depend on the model that's making them and don't really follow any sort of style guide (example: https://demo.setoku.com/apps/a7a1240ae0bc202c5eefa1cc). Your lib could bring some consistency and make global styling possible.
Let's say (because it's true) I've been programming for close to 40 years but in completely different domains, such that the concept of "dashboard" means almost nothing to me. Could someone explain what this project actually is, or what "BI" is or why I would want it? I mean, I assume that "BI" is the thing that "PowerBI" implements, but other than that I couldn't tell you anything.
Say you have an inventory - list of cloud database instances from all engineering teams in the org.
Is this list growing over time or shrinking or remains the mostly the same?
If the list is growing, is it growing evenly across all teams , or some teams stand out? Can I narrow down to a few teams and compare?
What about overall health of the inventory - how many instances list security or billing optimization findings, how many instances are on the version that would be out of support soon and would need to be upgraded?
A dashboard can answer all those questions immediately and make issues visible, as opposed to a long list in a form of a spreadsheet.
I'm not going to pretend to be an expert at it, but BI tools let you interrogate your data. This can be used for endless use cases for basically any department in an organization. But for instance, if you're a sales leader you might have some questions about all of the sales and marketing data in your company to focus your team's efforts for the next quarter. When you figure out which questions are actually returning useful findings, you then can take a set of these and build a dashboard which lets you easily re-examine those particular questions and their results with always up-to-date (or up-to-date-ish) data and can easily share it with the team.
well BI is I believe business intelligence, which is sort of like military intelligence, the set of practices and tools that have evolved around the need to present data to business people in such a way that they can make sensible decisions regarding their business.
on edit: So a dashboard is just one of those things were you see all the things you can do or interact with assembled together and from which you can navigate into any particular tool or view of data. Probably there would be meaningful data views on top. Like Number of Purchases versus people who started a purchase. And you could then drill down into the data that this chart represented to see how long purchases took, repeat customers, when people leave process or purchase etc. etc.
The dashboard is the entry point for how you will navigate your business data.
What drew me to the dbt project back in 2020 and what ultimately led me get a job at dbt Labs was this notion that when compared to software engineering teams, analytics teams had been vastly unserved by their tools.
In 2016 (and largely to this day), BI teams work in point-and-click interfaces without Software Development Lifecycle mainstays like source control, testing, and deployment environments.
There are BI engineers who do work like software engineers, but the gap b/w those who work like SWEs and those who don't is wide.
dbt Charts (like dbt before it did for data engineers) aims to empower data analysts to work with stakeholders in a more efficient and sustainable manner than was previously possible. The means to this end are: a succinct YAML DSL spec for defining charts, a powerful CLI that lets you validate, compile, and render the code into a dashboard spec the YAML.
Like data analysts being able to do PRs for changes to a dashboard is still rather unheard of especially if those PRs are in repos with other analytics code. Many BI vendors do ship some form of diffing and environment promotion, but virtually always these features fall short of what SWEs use every day.
Congrats on the launch! As someone who as led BI teams for many years, this looks great.. but - and you know this too, I’m sure - visualizations are only a small piece of the BI tool value prop. There’s governance, access controls, interactivity, and connectivity to semantic layers. The examples here show raw SQL but in reality we’d want something more abstract - names of dimensions and measures would probably be part of it.
So, are those other pieces going to be part of dbtTran Cloud?
hey -- Anders here, I've been helping the dbt Charts team. I loved this list you've enumerated here
> There’s governance, access controls, interactivity, and connectivity to semantic layers
I'd add to that list as well: KPIs, dropdowns, text boxes, theming, shared data sources.
If you're familiar with Vega-Lite [1], I like to explain dbt Charts as the Vega-Lite but for BI, in that it is a language that specifies all of the components of BI.
> are those other pieces going to be part of dbtTran Cloud?
Certainly there are some features that are more conducive to being offered as a managed service. However, we've built this enterprise BI tool backwards in that we've started less with commercialization in mind, but rather laying the groundwork in the language and OSS project, so that we need not reinvent the wheel over and over again.
I'm hand-waving here about a future that doens't yet exist. But in theory dbt Charts could be extended in the front end to allow for not just SQL or SL metric support, but any query language. Same for the backend, we dbt Charts can render to png and html today, but other formats are also possible!
This looks really slick! I was building something similar, so visuals are re-useable artifacts that external services know how to render, where we ask agents in whatever the interface is, slack, team's other agentic harnesses etc. and the agents receive the spec from the service, in this case dbt charts. if there was a unified spec that was agreed upon these third-party harnesses and apps would all speak the same charting language which would be super cool. I haven't read through it in detail but how does this differ from something like https://github.com/vega/vega-lite
We actually use vega-lite under the hood. vega-lite is vast tool set to build charts with static data. dbt Charts adds a whole layer on top of that, which is dashboards with repeatable SQL queries. Also dbt Charts comes with a lot of opinionated chart style ideas. So vega-lite is a toolbox for chart assembly, and we use that toolbox to build dashboards.
On the standardized charting language, that would be the dream. All agents giving the same spec when they want to build a chart or dashboard, and then having different renderers of that. That is a steep goal. Such a language would need to be simple but still extremely versatile. I'm not sure that combination exists yet. Our language is simple and fairly versatile, but not as versatile as vega-lite itself, or JavaScript code even. Maybe we can evolve in that direction though
I love ggsql, the team behind it, ggplot and the tidyverse in general!
I've been mulling extending dbt Charts to support ggplot, and will likely reach out to the team about this in the next few weeks.
The difference is that Vega-Lite, ggsql, ggplot2 are designed to create a single chart. Yes, you can use facet_wrap() to great a series of charts based on a particular variable, but arguably that's still one chart!
What's missing that is table stakes for a BI dashboard: KPIs, text boxes, dropdowns, data sources, governance, and how the charts are arranged on the page and in relation to one another.
In the next few months we hope to ship a JSON schema that represents a dashboard in the same way that a VegaLite does for a single chart.
https://openai.com/index/put-data-to-work/ was a compelling demo for me, being able to interact with a artifact and contextually EDA on a series or outliers is lowering the barrier to entry for BI exploration. I think meeting engineers where they work and being flexible and open sourcing these capabilities is really great to see
repeat after me: yaml is not a programming language. (boo, hn strips emojis, i should know that)
very disappointed that their domain specific language is just yaml. conditionals and variable binding become insane war crimes when yaml comes to town. you have a friggin llm, do better
and when we computer folks see the word language its implied that its a computing on computers language not a friggin data format.
edit: the yaml to avoid complexity that should live in the sql side can back fire, some folks I was working with last fall were evaluating if a yaml based OLAP tool would work for them, and there were some pretty gnarly gotchas from a yaml based approach. Secondarily, theres a real case to be made that having the data linkages not visible in the charting layer means that groups of related plots with different axes wont have the right data linkage without forcing a lot more ETL for what should be a quick plot if the data already fits in memory.
theres actually a very important reason you want it to be an actual embedded dsl or tiny programming language!
The reason why llms can code at all is the hugeeeee amount of RL based on the loop of 1 "write code", 2 get compile time or runtime errors,3 fix it and iterate. Data file formats dont have that feedback loop so models will fall off the rails faster. Writing code that fits a latent adhoc schema just wont work as well, or will require burning a lot more context.
from that perspective, it could just be an EDSL little library in the host language, or it could be a friggin little custom language with an interpreter and good error messages.
Fair hit on "language". I keep forgetting YAML stands for "YAML Ain't Markup Language". What we built is a declarative spec written in YAML, and that's deliberate. Think of it as HTML for boards rather than a programming language. The tradeoff is readability and structure versus flexibility.
That lack of flexibility is actually a feature for one of the bigger problems we're solving: lineage, or knowing where a number came from. If the chart layer can't transform data, the logic stays in SQL, where you can version, test and audit it.
It's also a choice about who this is for. The data community already works in SQL, Jinja and YAML every day, so there's no new language to learn.
Man, I didn't think anyone would out yaml helm, but here we are not just templating invisibly scoped files (yaml), but also SQL within them?
Looking forward to the yaml confusion when helm/dbt both see their default "charts/" directory in the same repo, or the dbt chart is set as a config map so it can be updated without rolling a new version of the full app... helm_argo is already a nightmare
rant aside, this does look super useful, and I do have CUE to help with the yamhell, but I may still prefer js/ts options so I can dynamically change the chart (like user clicking a dropdown for a different set of data). Sounds cool until the "dynamic" part of the chart shows it's limitations in crafting your ideal UX
I'll definitely be taking this for a spin, gets at that unbundling and "I need a quick chart" situations
A beacon of hope between all the vibe coded JS slop charts and dashboards! I love that this creates artifacts that are readable, maintainable and reproduce the same dashboard consistently (deterministically!), just with fresh data.
Everyone wants to come up with a clever One Spec to Rule Them All for generative UI. My bet is that the bitter lesson still bites. Models will continue getting faster and more error free at single-shot writing things from scratch with primitive libraries, and the flexibility that allows will make all of this for naught.
So this is a fairly domain specific (dashboards) spec and we intentionally avoid getting too generic.
The raw HTML/SVG or base libraries approach may well win out, but it does make it quite hard or impossible for humans to follow along and verify for instance where the numbers on a chart came from.
I think in a future where AI's doing all that verifying (or we just trust it), the AI might still prefer to use a DSL like ours because the abstraction maintains consistency, lowers maintenance, and saves a lot of tokens.
But the most helpful bits of a structured DSL are for sure still for humans. The structured format ensures things are readable and testable. Ours also enables a generative UI, which for now at least is still a much faster way to make visual edits while working with an AI, vs always through it.
I think we're still ways out until companies will blindly trust AI that the data they pulled is correct. If a data team member sees a dashboard or chart, their first questions is "Is the SQL below this chart correct?". The easier it is to see the SQL that pulled the data, the better.
This looks great.
"Unbundling BI" is absolutely where things are headed now that more people have agents, coding agents, agent computers to help with work.
I'm seeing this all up and down knowledge work tools. I've started treating email as a BI problem -- ETL it from Gmail and and create many different views into it and reports from it.
I just wrote up some thoughts on that here: https://housecat.com/blog/making-gmail-data-fast-for-humans-...
A natural followup is how to better visualize this data in chat. The DBT table component looks like it could help https://docs.dbtcharts.com/charts/tables/
Wow https://github.com/dbt-labs/dbt-charts has a great chat user / agent coding experience:
> Make charts of this with dbt Charts. Start with: uv tool install dbt-charts && dct skills intro
>
> build a dashboard of my hiring inboxes: list of candidate name / email / locale, application quality, response age
The one-shot dashboard is surprisingly good.
>This looks great.
>"Unbundling BI" is absolutely where things are headed now that more people have agents, coding agents, agent computers to help with work.
Agreed. Coupled with bento[0] for slide decks, I think this type of project is a very welcome development, helping us move away from walled gardens and proprietary software suites.
[0] https://github.com/nyblnet/bento
Thanks for sharing this too. It reminds me of GistDeck a super old project that turned a gist into slides. Infinitely better than powerpoint or GSlides.
https://github.com/nzoschke/gistdeck
Why visualize data? Graphs and charts always felt like the "show your work" in math class. It is forced synesthesia.
The rocket either lands or doesn't, safely, as expected. Green and Red lights are abstracted data plots.
The context should be the focus.
"Rocket must land at less than 0.2/mps" and pump the data into that context filter, more - reduce velocity, less - green light.
Not every visualized data point is tied to a binary decision.
e.g. when highlighting a quarterly revenue result, there is a big difference between missing the target by 1% or 50% (and of course the absolute number behind it is also important).
And on a larger scale, trajectory is also something that is easily visualized in graphs, and hard to boil down to a binary marker.
There certainly is a lot of "forced synesthesia", and more looking at charts than necessary out there, especially when decisions are then made more from the gut than based on the data one is claiming to base the decisions off, but a culture of binary-based decision making is just as bad of an idea.
I've worked in data for 15 years and I've never heard this argument. I love it :)!
Who knows, with AI this may be the future. Visualizations were created to help us understand data. Florence Nightingale published them "to affect thro' the Eyes what we fail to convey to the public through their word-proof ears."
Often a formula better represents a situation. But I'm not sure how much impact they'd have in spreading that understanding outside of a formula-minded audience.
I worked deep in big data briefly. Billions is too big for humans so we got creative how we visualized and filtered.
AI slop has a "smell" meaning there's an abstracted data filter that results in rose or poop output. Shakespeare variable says the name doesn't matter.
Sure, machines don't need visualization (although it seems that images are a surprisingly good medium to get context into agents), but humans still do. Mathematicians have plotted functions for a long time now. While the pure truth lies in the definition of f(x), the visualization helps the mind to grasp what it actually is. Humans will run companies for the foreseeable future, and so we will need visualization for the foreseeable future
Humans steer the ship. The wind and sea do the work. How we harness is what is changing.
Actually I think they do. Visualizations are a form of compression and focus.
All of my ai-driven plans use a tool I built to rasterize all text based visualizations. And I make the model present the rendered version to a clean room agent for brief-back. They go in circles until the diagram is good. Then, I instruct planning tools to always look at both the text and rendered version of all diagrams. I have found it to be surprisingly powerful.
Without guidance my agents tend to make me read CSVs.
Sure the data is correct, but it's obviously way easier for me to read it as a properly rendered table. Then you want to sort, group, aggregate the data. Visualize it many different ways if you will :)
Table and check marks in Markdown.
Malloy is an alternative to dbt, they have a similar product called Malloyyo [1], and Publisher [3], you can also the chat with HN data demo they have [4].
DBTCharts says you can serve charts locally, but it seems they want you to use their hosting service in production. Whereas, Malloyyo / Publisher are free to use anywhere. If you like college football checkout how I visualize drive data in [2], made with Malloyyo.
[1] - https://github.com/malloydata/malloyyo [2] - https://mrtimo.github.io/cfb-games/games-2026.html?%24SEASON... [3] - https://github.com/malloydata/publisher [4] - https://community.credibledata.com/hackernews
Neat, but… I’ve built up skills for my agents to just generate SVG charts and embed them in the chat timelines. A library helps, sure, but you can get surprisingly good results that cover 80% of the timeline and bar chart visualizations without any libraries, and that goes for diagrams as well (mine generate .drawio XML)
Hi HN, I'm Dave the founder of Chartio (YC'10 now Atlassian Analytics), announcing today dbt Charts, an open source YAML dialect and tool for declaring and rendering dashboards.
When making dashboards with claude or other agents, a lot of free-form artifacts are created that makes it hard to audit and scale. dbt Charts is a simple YAML dialect that declares and renders a chart (think markdown but for dashboards). Along with dbt its Apache 2.0 and launched today.
We hope this language + AI help make the BI space more open with dashboards as concise auditable code. Would love any thoughts and feedback.
Congrats on the launch Dave! I was wondering who was going to launch an attempt at industry standard here. Hope it gains widespread adoption.
Thank you Eoin!
Hi there, Burak here, creator of DaC and Bruin: https://github.com/bruin-data/dac
This is obviously a space we are very much interested in, so it is definitely nice to see approaches that attack the same problem. I haven't played with dbtcharts in depth yet but it looks very similar to DaC in principle, and also in the actual spec. I believe the industry definitely needs solutions like this to help get out of the legacy BI tools as Bİ is one of the biggest bottlenecks for AI adoption in large orgs.
Excited to see further competition in the space, nice launch!
It's neat but pretends to be more innovative than it actually is. BI has already been decoupled from everything else. People use AI to generate Excel and Power BI reports. YAML/XML/JSON - that doesn't matter. AI can generate whatever you instruct it to do.
So yes, a nice and logical development of dbt, but hardly as innovative as the blog post wants to sound. Nevertheless, I think it's a good idea that will be popular in certain circles. Hiring a professional data designer is a good idea - data visualization is very easy to get wrong.
I think I agree, but I'm not sure where Dave is claiming this is a paradigm shift in data visualization?
One of the key claims is actually that it's based on _old_ established visualization patterns from before BI tools were popular, and therefore results in consistently nice charts. The other key claim is just that it's simple and maintainable.
Sometimes tools & products can be useful as just well executed points in the known design space. You can already do all this with AI, but it's perhaps a little bit less nice and less maintainable.
[I work at dbt/Fivetran]
AI can definitely generate whatever you tell it to do, but what to do with that artifact afterwards? If you want it to be reusable it needs to be a well-defined structured protocol/language. What good is a YAML describing a chart if all I can do is to give it back to the agent and say "read this and create a chart from it", the result will be vastly different from the chart you got the first time.
> the result will be vastly different from the chart you got the first time.
AI can generate reusable artifacts including definitions of the data sources, why not? Most BI dashboards are just XML or JSON (or YAML) files and include definitions of data sources (directly or via semantic models). I struggle to see how dbtCharts is different. Yes, your YAML schema is clean and nice, but that's because you're in the early stages :) Once you go through feature bloat, your YAML format will become much more complex.
AI can generate an XML/JSON/YAML definition of a BI report according to a spec and link the data source in whatever form it should be referenced in the file. For instance, here is a skills file for defining data sources (semantic models) in AI-generated Power BI dashboards: https://github.com/microsoft/skills-for-fabric/blob/main/plu...
Interesting, but I'm not sure how it's different from Observable Framework ( https://observablehq.github.io/framework/ ) for dashboards as code. I like that one because it's Markdown and JS instead of a language defined in YAML + templates
The use case that immediately came to mind for me was to use dbt charts as a rendering method for an MCP App.
You don't really want to give a LLM client code execution environment like Observable Framework does. dbt charts appear to validate the entire yaml input including the SQL being sent to the data source.
So I'm getting: a data vis layer that can be dynamically defined and rendered via LLM/MCP client without the headache of sandboxing a JS Runtime environment.
I like the observable framework too - very flexible, you can nail very specific visualization details by dropping down to JS/html.
Maybe this is targeted to pure SQL folks that can get lost in UI details.
PS: one issue that I have with observable framework is authentication and authorization , it requires some system to be built on top of it to handle authn.
very cool! i'd been thinking about this idea and found vega-lite to be an interesting take as well [0].
i'll compare and look at folding this into setoku for app generation [1]. right now apps are just html blobs your claude authors + a mechanism for populating them with live data. definitely hard to audit but very flexible for operators to claude together internal apps. anyway, the charts look decent but really depend on the model that's making them and don't really follow any sort of style guide (example: https://demo.setoku.com/apps/a7a1240ae0bc202c5eefa1cc). Your lib could bring some consistency and make global styling possible.
[0]: https://vega.github.io/vega-lite/
[1]: https://setoku.com
You can specify Apache Superset Dashboards, Charts, ... using yaml, for example:
https://github.com/preset-io/headless-bi-blog-post-examples/...
But this is mostly just a dump of internal Superset data structures.
Let's say (because it's true) I've been programming for close to 40 years but in completely different domains, such that the concept of "dashboard" means almost nothing to me. Could someone explain what this project actually is, or what "BI" is or why I would want it? I mean, I assume that "BI" is the thing that "PowerBI" implements, but other than that I couldn't tell you anything.
Say you have an inventory - list of cloud database instances from all engineering teams in the org.
Is this list growing over time or shrinking or remains the mostly the same?
If the list is growing, is it growing evenly across all teams , or some teams stand out? Can I narrow down to a few teams and compare?
What about overall health of the inventory - how many instances list security or billing optimization findings, how many instances are on the version that would be out of support soon and would need to be upgraded?
A dashboard can answer all those questions immediately and make issues visible, as opposed to a long list in a form of a spreadsheet.
I'm not going to pretend to be an expert at it, but BI tools let you interrogate your data. This can be used for endless use cases for basically any department in an organization. But for instance, if you're a sales leader you might have some questions about all of the sales and marketing data in your company to focus your team's efforts for the next quarter. When you figure out which questions are actually returning useful findings, you then can take a set of these and build a dashboard which lets you easily re-examine those particular questions and their results with always up-to-date (or up-to-date-ish) data and can easily share it with the team.
well BI is I believe business intelligence, which is sort of like military intelligence, the set of practices and tools that have evolved around the need to present data to business people in such a way that they can make sensible decisions regarding their business.
on edit: So a dashboard is just one of those things were you see all the things you can do or interact with assembled together and from which you can navigate into any particular tool or view of data. Probably there would be meaningful data views on top. Like Number of Purchases versus people who started a purchase. And you could then drill down into the data that this chart represented to see how long purchases took, repeat customers, when people leave process or purchase etc. etc.
The dashboard is the entry point for how you will navigate your business data.
So they're trying to simplify what, the addition of new views to the dashboard?
(Anders here, on the dbt Charts team)
What drew me to the dbt project back in 2020 and what ultimately led me get a job at dbt Labs was this notion that when compared to software engineering teams, analytics teams had been vastly unserved by their tools.
In 2016 (and largely to this day), BI teams work in point-and-click interfaces without Software Development Lifecycle mainstays like source control, testing, and deployment environments.
There are BI engineers who do work like software engineers, but the gap b/w those who work like SWEs and those who don't is wide.
dbt Charts (like dbt before it did for data engineers) aims to empower data analysts to work with stakeholders in a more efficient and sustainable manner than was previously possible. The means to this end are: a succinct YAML DSL spec for defining charts, a powerful CLI that lets you validate, compile, and render the code into a dashboard spec the YAML.
Like data analysts being able to do PRs for changes to a dashboard is still rather unheard of especially if those PRs are in repos with other analytics code. Many BI vendors do ship some form of diffing and environment promotion, but virtually always these features fall short of what SWEs use every day.
Okay nice chart, but can I click in and filter into the data? Visualisations are not what a BI tool does; it's just the tip of the iceberg.
Congrats on the launch! As someone who as led BI teams for many years, this looks great.. but - and you know this too, I’m sure - visualizations are only a small piece of the BI tool value prop. There’s governance, access controls, interactivity, and connectivity to semantic layers. The examples here show raw SQL but in reality we’d want something more abstract - names of dimensions and measures would probably be part of it.
So, are those other pieces going to be part of dbtTran Cloud?
hey -- Anders here, I've been helping the dbt Charts team. I loved this list you've enumerated here
> There’s governance, access controls, interactivity, and connectivity to semantic layers
I'd add to that list as well: KPIs, dropdowns, text boxes, theming, shared data sources.
If you're familiar with Vega-Lite [1], I like to explain dbt Charts as the Vega-Lite but for BI, in that it is a language that specifies all of the components of BI.
> are those other pieces going to be part of dbtTran Cloud?
Certainly there are some features that are more conducive to being offered as a managed service. However, we've built this enterprise BI tool backwards in that we've started less with commercialization in mind, but rather laying the groundwork in the language and OSS project, so that we need not reinvent the wheel over and over again.
I'm hand-waving here about a future that doens't yet exist. But in theory dbt Charts could be extended in the front end to allow for not just SQL or SL metric support, but any query language. Same for the backend, we dbt Charts can render to png and html today, but other formats are also possible!
[1]: https://vega.github.io/vega-lite/
Yup! Big plans for deep semantic layer interaction
dbt/Fivetran announces that they will unbundle BI by bundling it with their ELT solution. Ironic to say the least.
This looks really slick! I was building something similar, so visuals are re-useable artifacts that external services know how to render, where we ask agents in whatever the interface is, slack, team's other agentic harnesses etc. and the agents receive the spec from the service, in this case dbt charts. if there was a unified spec that was agreed upon these third-party harnesses and apps would all speak the same charting language which would be super cool. I haven't read through it in detail but how does this differ from something like https://github.com/vega/vega-lite
We actually use vega-lite under the hood. vega-lite is vast tool set to build charts with static data. dbt Charts adds a whole layer on top of that, which is dashboards with repeatable SQL queries. Also dbt Charts comes with a lot of opinionated chart style ideas. So vega-lite is a toolbox for chart assembly, and we use that toolbox to build dashboards.
On the standardized charting language, that would be the dream. All agents giving the same spec when they want to build a chart or dashboard, and then having different renderers of that. That is a steep goal. Such a language would need to be simple but still extremely versatile. I'm not sure that combination exists yet. Our language is simple and fairly versatile, but not as versatile as vega-lite itself, or JavaScript code even. Maybe we can evolve in that direction though
What do you think about GGSQL, or is that not entirely aligned with what you want? They also produce vega-lite JSON as output.
https://ggsql.org/
I love ggsql, the team behind it, ggplot and the tidyverse in general!
I've been mulling extending dbt Charts to support ggplot, and will likely reach out to the team about this in the next few weeks.
The difference is that Vega-Lite, ggsql, ggplot2 are designed to create a single chart. Yes, you can use facet_wrap() to great a series of charts based on a particular variable, but arguably that's still one chart!
What's missing that is table stakes for a BI dashboard: KPIs, text boxes, dropdowns, data sources, governance, and how the charts are arranged on the page and in relation to one another.
In the next few months we hope to ship a JSON schema that represents a dashboard in the same way that a VegaLite does for a single chart.
Does that make sense?
https://openai.com/index/put-data-to-work/ was a compelling demo for me, being able to interact with a artifact and contextually EDA on a series or outliers is lowering the barrier to entry for BI exploration. I think meeting engineers where they work and being flexible and open sourcing these capabilities is really great to see
Nice to see something built for AI that's supporting both openness and interpretablilty.
Is this associated with dbtlabs?
Yes this is a dbt product - will be in the dbt cli soon, we shipped it initially as a separate tool while it’s in preview.
repeat after me: yaml is not a programming language. (boo, hn strips emojis, i should know that)
very disappointed that their domain specific language is just yaml. conditionals and variable binding become insane war crimes when yaml comes to town. you have a friggin llm, do better
and when we computer folks see the word language its implied that its a computing on computers language not a friggin data format.
edit: the yaml to avoid complexity that should live in the sql side can back fire, some folks I was working with last fall were evaluating if a yaml based OLAP tool would work for them, and there were some pretty gnarly gotchas from a yaml based approach. Secondarily, theres a real case to be made that having the data linkages not visible in the charting layer means that groups of related plots with different axes wont have the right data linkage without forcing a lot more ETL for what should be a quick plot if the data already fits in memory.
what would you have chosen for your DSL? Ideally something not from scratch
i'm not sure if that caveat is sensible :)
theres actually a very important reason you want it to be an actual embedded dsl or tiny programming language!
The reason why llms can code at all is the hugeeeee amount of RL based on the loop of 1 "write code", 2 get compile time or runtime errors,3 fix it and iterate. Data file formats dont have that feedback loop so models will fall off the rails faster. Writing code that fits a latent adhoc schema just wont work as well, or will require burning a lot more context.
from that perspective, it could just be an EDSL little library in the host language, or it could be a friggin little custom language with an interpreter and good error messages.
Oh to be clear
> dct validate <file.yml>
Or any of the other commands that read the file will quickly fail with any syntax or SQL issues. It’s not just an open schema.
Fair hit on "language". I keep forgetting YAML stands for "YAML Ain't Markup Language". What we built is a declarative spec written in YAML, and that's deliberate. Think of it as HTML for boards rather than a programming language. The tradeoff is readability and structure versus flexibility.
That lack of flexibility is actually a feature for one of the bigger problems we're solving: lineage, or knowing where a number came from. If the chart layer can't transform data, the logic stays in SQL, where you can version, test and audit it.
It's also a choice about who this is for. The data community already works in SQL, Jinja and YAML every day, so there's no new language to learn.
How is this distinct from something like Mosaic (https://idl.uw.edu/mosaic/spec/)?
Man, I didn't think anyone would out yaml helm, but here we are not just templating invisibly scoped files (yaml), but also SQL within them?
Looking forward to the yaml confusion when helm/dbt both see their default "charts/" directory in the same repo, or the dbt chart is set as a config map so it can be updated without rolling a new version of the full app... helm_argo is already a nightmare
rant aside, this does look super useful, and I do have CUE to help with the yamhell, but I may still prefer js/ts options so I can dynamically change the chart (like user clicking a dropdown for a different set of data). Sounds cool until the "dynamic" part of the chart shows it's limitations in crafting your ideal UX
I'll definitely be taking this for a spin, gets at that unbundling and "I need a quick chart" situations
how does this compare w/ evidence (https://evidence.dev/)?
All docs for evidence.dev are AI-hallucinated …YMMV
A beacon of hope between all the vibe coded JS slop charts and dashboards! I love that this creates artifacts that are readable, maintainable and reproduce the same dashboard consistently (deterministically!), just with fresh data.
Yup, that's exactly the idea.
This looks great can’t wait to try it out
Everyone wants to come up with a clever One Spec to Rule Them All for generative UI. My bet is that the bitter lesson still bites. Models will continue getting faster and more error free at single-shot writing things from scratch with primitive libraries, and the flexibility that allows will make all of this for naught.
So this is a fairly domain specific (dashboards) spec and we intentionally avoid getting too generic.
The raw HTML/SVG or base libraries approach may well win out, but it does make it quite hard or impossible for humans to follow along and verify for instance where the numbers on a chart came from.
I think in a future where AI's doing all that verifying (or we just trust it), the AI might still prefer to use a DSL like ours because the abstraction maintains consistency, lowers maintenance, and saves a lot of tokens.
But the most helpful bits of a structured DSL are for sure still for humans. The structured format ensures things are readable and testable. Ours also enables a generative UI, which for now at least is still a much faster way to make visual edits while working with an AI, vs always through it.
I think we're still ways out until companies will blindly trust AI that the data they pulled is correct. If a data team member sees a dashboard or chart, their first questions is "Is the SQL below this chart correct?". The easier it is to see the SQL that pulled the data, the better.
Exactly and that's why dbt Charts is a spec in YAML vs a python library. It ensures that the logic stays in SQL where its easy to test and trace.
thats intresting