Fabric

Critically Evaluating LLMs: What Data Visualization Can Teach Us

Announced at Build, Fabric Apps introduce a new AI-first way to build custom web apps with Microsoft Fabric as the backend. For analytics teams, this means developers and their AI coding agents now have an accelerated path to building enterprise-grade data apps directly on their semantic models. Organizations can create and deploy operational data apps that leverage the same trusted business logic and the same governance as the rest of their analytics stack. From financial planning to inventory management to pricing optimization, or really any app you can describe, your coding agent can build a custom-tailored app based on your specifications and semantic model in just a few prompts. And it's not just about polished UI. Coding agents can trivialize features that are used to take real engineering effort, like persona-specific views, custom calendar interfaces, bespoke business logic, and more.

The fix is fewer, more heavily governed logical models: curate a small set of canonical, single source-of-truth datasets that are clearly owned, consumption-ready, and discoverable, then aggressively deprecate the near-duplicates.

The fix is fewer, more heavily governed logical models: curate a small set of canonical, single source-of-truth datasets that are clearly owned, consumption-ready, and discoverable, then aggressively deprecate the near-duplicates. Physical rollups and caches still matter for cost and performance

The goal is that when an agent searches for a concept, it finds a single governed answer.

Create pairwise skills: a knowledge skill acts as a thin top-level router that allows additional domain details to load on demand

The unbook skill encodes the process a senior analyst would follow: clarify the question, find sources (via the knowledge skill), run the query, and then loop the result through adversarial review sub-agents. It also bundles a dozen reusable analysis patterns (retention curves, rate decomposition, funnel analysis) so that common requests don't get reinvented each time.

Treat skill maintenance as a first class citizen

Dashboard-based evals are auto-generated by Claude (then human validated), covering the most common stakeholder questions

A domain owner can't announce the agent to their stakeholders until their slice of the eval set clears some threshold

We then verified in transcripts that it actually read them before every answer. Accuracy moved by less than a point in either direction

Two of ours: stacking additional rounds of doc refinement past a certain point (we hit three consecutive net-negative iterations: the docs were getting longer, not better), and swapping the adversarial reviewer to a cheaper model to cut latency (it lost most of the accuracy wins, for no real speedup

Claude skill to aggressively challenge all underlying assumptions on a potential final answer increased accuracy by 6% within our eval set, but at the cost of 32% more tokens and 72% higher latency.

every response carries a footer that contains which source tier it came from

We often see companies building a significant amount of infrastructure to account for current model shortfalls that become moot once those models improve

Some of the processes we discussed may be overkill if, for example, you don’t produce much data, you only have a few consumers of the output, or your data model is likely to remain simple.

How technical is the intended audience of the output?

poor or stale documentation. Claude is exceptionally useful for closing the gap (drafting column descriptions, proposing metric docs from query patterns, flagging undocumented models in CI), but the curation and ownership are managed by humans.

ooks like.