Power BI

In contrast, by hard accuracy I refer to outputs that must meet a very precise—possibly even unitary—definition of correct. When you request a piece of code or a very specific, detailed image, there is a slim margin of acceptable outputs. Notably, soft or hard accuracy is not specific to a particular type of output or use-case, although, generally, creative areas more typically have these soft accuracy requirements. However, when it comes to outputs about data, there is typically only a single correct answer… the truth.

In a real-world scenario, people are lazy and not trained to be “prompt engineers”. While we often say that tools like Copilot or Chat-GPT accept “natural language” prompts, typically, good prompts don’t come to users very naturally. The initial prompt in the previous example describes what the user wants in more “normal” language. While this prompt comes more naturally to the user, it is less likely to return correct results because of the following reasons:

The user doesn’t specify where the order date and billing date columns are, which might be necessary since this model is imperfect (like in real-life) and these columns exist redundantly in both an ‘Orders’ and an ‘Invoices’ table. The user doesn’t specify how to get express orders, which according to business logic should be done via the Order Document Type, or Order Doc. Type. The user doesn’t specify how to aggregate the hourly difference between the order date and billing date.

In real life, however, the story might not stop here. What if:

Put another way, the more you know about a topic, the better you can tell what’s right and what’s wrong. When you know more, you’re also more aware of what you don’t know, what’s possible, and what makes sense. When using generative AI, this is extremely helpful, because you can have a more productive sparring session and less frequently fall victim to their errors or hallucinations.

Note that I’m not saying that you need to be an expert to get value from generative AI. On the contrary, I think that generative AI lowers the technical barrier and knowledge curve for many people, particularly when combined with reliable, trustworthy sources of information. However, I do think that the less experienced you are in a given area, the more likely you are to encounter challenges with the errors and hallucinations that AI returns. You’ll still get value from using generative AI, but you need to mitigate the risk that this creates.

EXAMPLE 2: USING COPILOT TO ANSWER DATA QUESTIONS

The user opens Power BI Desktop and configures it to use the F64 workspace for Copilot. They connect to a shared semantic model to start creating a report. From the Copilot pane, they can ask questions about that semantic model. Operations like querying the model are part of the "main" capacity, but the Copilot input and output contributes to consumption of the F64 capacity. The user can only access data that they themselves have access to, based on their workspace roles and item permissions. Click the image to enlarge it in a lightbox.

When submitting another prompt after changing the report page, Copilot then creates a card stating that Turnover MTD is 5bn for August 2021, which is not only incorrect, but I have no idea how it came up with this number, because when clicking “Show reasoning”, Copilot simply re-phrases the prompt.

In the first attempt, Copilot is uncertain and asks for clarification. This is quite nice; I like that this is part of the user experience. The user understand they are trying to select (or filter) to a specific currency, so they select that option. This leads to a second prompt, followed by a question from Copilot: “Which specific currency would you like to filter by?” Again, this is nice. The user responds with the currency, “WAT”, but Copilot returns an error.

TUNING AND BRICK WALLS: SETTING UP THE MODEL WITH COPILOT IN MIND

When making some model adjustments and continuing to refine the prompt, the user comes to the following result:

Copilot implies that a physical relationship is required for filtering, here… but this doesn’t seem to be listed as a limitation in the documentation. Regardless, it seems like unless we make significant model changes, we’ve reached a dead-end here, and we won’t be able to find out the Turnover MTD from Copilot.

There’s not much more to say here about data quality and generative AI, really; it’s been said many times. Like any BI tool or solution, quality is important to get value. Shit in, shit out. The onus is up to the organization to ensure that the data is fit for use in the tool, Copilot or not.

EXAMPLE 3: GENERATING REPORTS WITH COPILOT