#310: A Multi-Stop Tour of AI in Data and Analytics

00:00:00.00 | Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language.

00:00:15.33 | Michael Helbling: Hi, everyone. Welcome. It’s the Analytics Power Hour. This is episode 310. It seems like every time you turn around, you’re hearing some amazing new AI story. I guess it’s important to engage, but it can feel like what you read or hear is, I don’t know, somewhat sensationalized or overblown. Well, we wanted to bring you some reality on the ground. And we have an amazing opportunity to go in-depth with a few people from one organization to give us a bit of a tour on what they’re working on, how AI is impacting it, and their perspective. Thanks to Moe Kiss, she’s pulled together folks from across Canva to give us some insight in how they’re using AI. So let’s dive into a series of quick discussions on how AI is impacting people. AI is being used and thought of in one org. And I’m with my co-host today, Julie Hoyer. Hi.

00:01:01.17 | Moe Kiss: Why, hello.

00:01:02.21 | Michael Helbling: And, of course, Moe Kiss. Hey.

00:01:04.65 | Moe Kiss: Hey, how you going, Michael?

00:01:06.01 | Michael Helbling: I’m doing great. I’m Michael Helbling. And we’ve got a number of guests today, and they are Christian Sullivan, Nina Kumagai, Jack O’Hagan, and Flynn Alexander. And I’m excited for us to talk to each of them and get to know them a little better. Christian, we’re starting with you today. Thank you so much for first coming on the podcast. As we get started, could you give us just a… a quick intro to the work you do at Canva and your role there?

00:01:30.51 | Christian Sullivan: Thanks, Michael. It’s a pleasure to be here. So I work in the Foundational Goals Performance team as the data lead. Now, our team kind of splits itself between an ops focus, which is all about our monthly business reviews, having teams kind of accountable to their metrics and reporting on their performance all the way up to our founders. And on the other side is more of a data platforming team where we build and operationalize a lot of tooling that enables teams to better connect with their metrics and business performance. So I’m leaning more heavily into the more tooling side, but that’s kind of our core accountability at Canva.

00:02:05.92 | Moe Kiss: Christian, one of the Canva things I was trying to let my fellow co-hosts know about earlier is that our project naming is very important. So we wanted to find out a little bit about a project you’ve called ARCA, what it does, how it works, how you’re kind of using AI tooling to help you with data science, particularly in that field. So we wanted to find out a little bit about that foundational data space.

00:02:27.37 | Christian Sullivan: Yeah, so ARCA is an acronym for Agent Driven Root Cause Analysis. Now, my team kind of looks after the alerting platform at Canva, and we try to make it as easy for teams to set up alerts on any business metric they can imagine. Now, what that means is that because Canva’s scale, we have hundreds of metrics with 10 to 15 different cuts. And when we’re setting up alerting across all of these metrics, things can get exceptionally noisy. It can be noisy and really hard to figure out which ones are the alerts that we need to deep dive into that represent real business performance degradations versus those that might just be due to a seasonal event or a known marketing launch or something that we would basically expect. So it’s less of an alert, more of a notification. So what we realized was that we’re sending out hundreds, if not thousands of alerts per day, and only a very small percentage of them were actually being triaged or investigated. Right. And at the start of this year, one of our main targets was to really solve this and to get as much value out of our alerting platform as possible. That’s when we came up with the idea to plug in an agentic workflow. And that’s where kind of ARCA was born. So the way that it works is whenever an alert is triggered, as long as a team has onboarded or added an ARCA tag to their alerting configuration, ARCA will basically spin up an agent, run an investigation, perform metric decomposition, and then publish that directly back to the alert and update the alert from a triage status to in investigating. Now, teams don’t resolve things immediately from there. They can either choose to take the comment that ARCA left and record that as the reason that the metric moved, or they can go deeper and add some additional context and then update that to resolve. So the idea is that. We always want humans in the loop, at least to verify the results. And then ARCA kind of does that initial analysis and moves us from a place where we’re cold starting investigations, where an analyst or data scientist could be receiving 10 to 15 alerts per day and having to now cut that metric by a bunch of different segments, look into other related event data, look into our data warehouse to see if there’s been a load job that’s kind of gone awry. And instead, ARCA basically starts that process for them, means that they’re not cold starting those investigations and that ultimately they can reach a resolution state a lot faster. Yeah, so that’s basically how it works. Now, a lot of it was overcoming complexity just with how our metrics can work and providing that context. So we’re still only, I’d say, 1% of the way there in terms of where we want ARCA to go. But we’ve noticed some real value. And teams who have onboarded to it. And we’re starting to really accelerate the number of teams that are using it. It’s only been around for about a month now.

00:05:25.80 | Moe Kiss: So tell us a bit about some of the technical challenges. Like, I feel like we’re hearing the dream state where it’s working as intended. And the investigations are not causing data scientists to go down different rabbit holes. Like, what were some of the hard bits that you had to kind of work through?

00:05:41.69 | Christian Sullivan: Yeah, I’d say the biggest hard bit was our V1 of ARCA, which was this megalith of different agents interacting with each other, having access to Slack, having access to Confluence, to our internal company knowledge base, and then having access to our data warehouse. And being able to tell an agent how to reason between often conflicting sources of information and come to a root cause meant that, A, our results were not replicable. So every time we called the agent, we would often get a different root cause. And B, it was incredibly expensive. Because we’re burning tokens with this multi-agent workflow, trying to embed like a human in the loop kind of conversational flow. Before we realized, let’s just break this down into the most kind of core component for what you would generally start root cause analysis with, which is segmentation and breaking down by different segments. And then that gives you like a bread trail of or breadcrumbs where you can go into and further investigate based on, you know, it might be due to a country degrading in a given metric. You know, to investigate that country and to understand why that country is dropped. And so what we really found is that the lack of determinism in these workflows meant that agents were kind of going off in all different directions and publishing really weird results that ultimately wasn’t useful. So by pulling this back and understanding where can we actually get determinism, where can we say this agent needs to run this set of SQL queries instead of this agent can generate and run its own SQL queries. We can actually do that. We kind of came to a state where we’re using the agent to reason across huge amounts of data and understand interesting trends. But we’re using determinism to look at patterns where we know that every agent will investigate in this way or any human would do the same. And so it was really first removing a lot of that complexity. So taking away that multi agent orchestration workflow and just going with a single agent. Secondly, how can we have that agent spend less time kind of interacting with the person and more time picking up this deterministic workflow, publishing a root cause and being really clear as to what it’s investigated and then what it would recommend a human to do instead of trying to kind of one shot that whole thing through this overly complex investigation.

00:08:02.19 | Julie Hoyer: Did you have to spend a lot of time giving it like historical examples of like these root cause analyses that were done by like analysts throughout Canva to help it? Or like, how did you train it maybe on some of that?

00:08:15.24 | Christian Sullivan: Yeah, we definitely use that in our evals sets. So we would look for past alerts where teams have provided a root cause and use that to see if an agent was able to get it. We haven’t gone so much into providing past examples of, you know, known reasons why a metric might degrade in a given month because we always run a marketing promotion then. But that’s something that is on our roadmap. So we would like to have the ability for a metric owner to provide additional context so that the agent can pick up on that in future. And we’d also like for the agent to be more aware of other alerts that have been resolved, triaged, and documented in our alerting platform so that that can all be done automatically, that we don’t have to go through the manual process of identifying where do we see trends and patterns from before. What if we can just pick up all alerts that were resolved over the past forever and putting that into the agent so that it can understand, oh, maybe there is a seasonal pattern here. Or maybe we do know that when a given segment moves like this, it could be due to a marketing campaign or grandfathering a product or any other reason that a metric might move.

00:09:16.90 | Michael Helbling: And are you using multiple different LLMs for this, or is it all flowing through sort of one structure? Like, how is that architected on the back end?

00:09:27.00 | Christian Sullivan: Yeah, so initially we tried to run with multiple different LLMs. So we were generally using like the mid-tier model, like a Sonnet or a Terra, for the main reasoning and investigation workflow. And then the summarization was done on a lighter model, like a Haiku. We’ve now just moved everything into kind of the mid-tier model. And we’re running with GPT 5.6 Terra at the moment. But I think as we add complexity, we’ll probably consider reintroducing more of those different models. And we’ll use our evals set to basically auto-optimize that. So we might run our evals, see how they’re scoring, and then whichever combination of agents and architecture wins, we’ll just default to that.

00:10:08.64 | Michael Helbling: I’m glad you brought up evals, because it seems like that is the super time-consuming part of getting these things really tuned. Can you talk a little bit about that process as you built out this system?

00:10:22.09 | Christian Sullivan: Yeah, I think evals are probably the number one most important thing to this, because agents can sound very convincing. And you can kind of just say, oh, this looks great. I’m just going to have that set up. But until you’ve got a system where you’re repeatedly able to test the same investigations or the same kind of scenario that you would see in the real world, until you’re able to do that at scale and do that in a repeatable way, your ability to improve your product is impossible, right? Because you can only get so far as being able to improve the quality of the responses as you see it. But if you’re able to test 20, 30 different questions over and over and see not only is the agent able to answer them correctly, but is the agent able to repeatedly answer them correctly so that you can understand just how often is your agent going to veer off course on the same eval. So that’s been incredibly important to how we work today. I think we still need to build out our set of evaluations and to enrich them more, but it’s something that’s top of our priority list at the moment. One other thing as well, I’d say before building any kind of agentic workflow, starting with evals is really the most important part because you’re capturing what your agent needs to get right and what it needs to basically produce. And so once you’ve got a really clear understanding of that, then you know what you’re building towards, even if it’s just a single eval or three or five, just something to start with that basically says, this is our North Star. This is what we’re building towards helps you really identify what you’re targeting, but also constrain the scope to know what you’re targeting from the start, instead of building everything out and then trying to figure out which evals are actually going to fit your agent at the end.

00:12:06.86 | Moe Kiss: I’ve got to ask, I feel like some folks listening to this might be, oh, cool. A team like Canva can build this. You know, we’ve got heaps of data scientists. Like, what’s the time investment been? And do you look back now and be like, that was time well spent, like in terms of what it’s saving the team in investigations?

00:12:25.72 | Christian Sullivan: Yeah, I’d say the first version of ARCA probably wasn’t time well spent in what we produced, but time well spent in what we learned. But the second version, definitely. It probably took us a month to stand up end to end from, you know, having the data warehouse scheduler send the alert metadata to an agent, to an agent then performing the investigation and fine tuning that to then publishing back to our anomaly service. That probably took us about a month to build. And since then, we’ve noticed that our mean time to investigate an alert has gone down from about two and a half days to about two hours. And the quality of investigations has really picked up. And we’ve only really done this for the last month. So as more teams onboard to ARCA… We’ll get more quality responses. We’ll get more user feedback so we know what we need to improve. And I think over time we’ll get to the point where pretty much every alert is agent investigated and then documented and resolved with a human in the loop. One other thing I would say is that we made onboarding to ARCA incredibly easy. So all users need to do is tag in their alert like AI Agent ARCA and forever from that point, will have ARCA respond to the alerts. It’s not some custom configuration. It’s not, you know, pull requests that they need to make into our data warehouse repository. It’s a really simple onboarding process. And I think that’s kind of been key for us is to understand how can we, when attention is so hard to get through different LLMs and different tools that teams are producing, how can we make this as easy as possible? And that’s kind of where we’ve landed today.

00:14:06.44 | Michael Helbling: You mentioned you’re doing a lot of this on GPT 5.6 or whatever. And I think that just came out this past summer. When new models get released, what does that do to your process? Like, do you have to restart your evals? Like, what’s that look like?

00:14:22.83 | Christian Sullivan: Yeah, that’s probably where evals are also important, as well as knowing the costs of each run. So when we have new models, like we won’t jump onto them immediately. We generally wait for time to settle and we might kind of see what the response is. to those models. But the process is generally, we will understand what the model is targeted to do, understand if we need to make any tweaks on our end, and then switch it over, run a set of evals, or maybe just test out a few questions, understand on both cost and quality of the output. Are we getting a better result here? Is it worth switching over? And then we’ll decide to make that switch. In our data agents platform, though, which we use to kind of host, we have a really easy ability to switch between models and model providers. So that’s one thing that I think if you’re getting into the space, the ability to switch between and not be completely dependent on one architecture could be valuable. But I would also say that a lot of the models can perform reasonably similarly, and you’re getting updates, you know, almost weekly from each of the kind of frontier model providers. So it’s probably not the worst thing in the world to lock something in, if you have an existing agreement, or if you’re looking to build something fast, and then model kind of interoperability can come later.

00:15:45.91 | Julie Hoyer: How did you tell ARCA to respond? Like what type of language are you telling it for its output? Because I’m curious, like having the human in the loop, and it’s telling the analysts, like, is it suggesting to go look down a certain path? Or is it giving it like a confidence in it? Is it giving it evidence? Like, I’m curious the language you guys choose to make sure people are motivated in the future to like, check it still? Yeah.

00:16:07.72 | Christian Sullivan: So one thing I’ve been happy with recently, and I think this came from us switching to GPT, was that the responses are more kind of reasoned, and it will tell you if it wasn’t able to find any root cause. But this kind of goes back to the determinisms step. So we run kind of metric decomposition to figure out, you know, is there a really clear trend with a given sub segment that it’s not? Is causing this metric to move in this way? And if it can pick up that, then it’ll call that out. Another thing we do is we have a likely kind of root cause reason at the top of the alert. So what that will say is, is it a data quality is incident? Is it a seasonal event? Is it a true anomaly that needs to be investigated? And so that is kind of a signposting to basically say, oh, if this is a real anomaly, you need to look into this straight away. If it’s a seasonal event, maybe just check it, and then you can resolve it. Straight away. If it’s a data quality incident, maybe just wait till the next day. If it hasn’t loaded, then maybe raise the alarm bells and work with your analytics or data engineer to figure out, is there something going wrong in the data warehouse that we need to resolve? Or will this order resolve? And then the pros or summary kind of below that is generally just three or four lines saying, this is the likely root cause. This is what I checked. This is the metric movement that I’ve seen for this sub segment. And that’s kind of where the response is. One thing that we would like to build is the ability to resume that conversation and basically click a link so that you can have more detailed kind of documentation of all the queries that run instead of these just being accessed by us in the logs, as well as being able to resume that conversation with the agent. So you’re not picking up from scratch and you can say, hey, you investigated this. Have you looked at this segment split and being able to kind of really close the loop there easily and then have that agent published back to the alert so that we kind of own that complete investigation instead of a user. At the moment, seeing the alert running with their own investigation and published back in a separate workflow.

00:18:12.51 | Julie Hoyer: Amazing. Okay. Thank you for letting me do that. One last question. I had to know.

00:18:15.91 | Michael Helbling: No problem. Thank you so much, Christian, for coming on the show. Really appreciate it. Tim, what’s on your Black Friday readiness checklist?

00:18:26.81 | Tim Wilson: Inventory, promotions, landing pages, paid media, and apparently refreshing dashboards nervously every six minutes.

00:18:33.91 | Michael Helbling: Ooh, yes. One more thing should probably be on there: making sure your tracking actually works before the traffic shows up.

00:18:41.81 | Tim Wilson: Good point, because Black Friday is a terrible time to discover your purchase event has been broken since Tuesday.

00:18:48.01 | Michael Helbling: Exactly. Our friends at Stape have a free website tracking checker. It can scan your site and give you a personalized report in just a couple of minutes.

00:18:57.31 | Tim Wilson: It identifies tracking issues and gives you a prioritized list of recommendations. So, you know, what’s worth fixing first.

00:19:04.11 | Michael Helbling: You can also see how your setup compares with, with competitors and get estimates of where your tracking problems may be affecting leads or sales or ad spend.

00:19:12.01 | Tim Wilson: Much better to find that out before Black Friday than during it.

00:19:16.01 | Michael Helbling: Agreed. So click the link in the show description and run the free Stape website tracking checker.

00:19:22.12 | Tim Wilson: Your future holiday season selves will appreciate it. Michael, what if building an analytics app started with just describing what you want?

00:19:31.82 | Michael Helbling: Okay, that sounds suspiciously like a feature designed by people who have met analysts.

00:19:36.42 | Tim Wilson: That is the new app builder in Prism. You describe the app and before it builds anything, it asks questions to make sure it actually understands what you mean.

00:19:46.72 | Michael Helbling: Radical concept and AI that asks for clarification before confidently kind of ruining my afternoon.

00:19:54.62 | Tim Wilson: That’s the idea. Then it builds the app, checks the numbers and lets you refine everything just through chat.

00:20:02.01 | Michael Helbling: Ooh, so I could say like, okay, tweak that chart and add a filter, or, you know, clean this up and make it less ugly. Undo whatever you just did.

00:20:11.11 | Tim Wilson: Exactly. You can use your own templates or built-in ones for GA4, paid media, and more.

00:20:17.31 | Michael Helbling: Ooh, and I can preview what different users see before publishing?

00:20:21.51 | Tim Wilson: Yep, draft it, publish when ready, and use View As to check the experience for each viewer.

00:20:27.72 | Michael Helbling: Oh, finally. So looks good on my screen is no longer a deployment strategy.

00:20:34.01 | Tim Wilson: Click the link in the show description and join the Ask-Y waitlist. Use code APH and they’ll move you right to the top of that list.

00:20:41.81 | Michael Helbling: Build the app, boss it around, and ship it when it behaves. All right, up next, we have Nina Kumagai. Nina, welcome to the show. What’s your team? What projects have you been working on?

00:20:54.72 | Nina Kumagai: Thank you so much for having me, Michael. Hi, everyone. I’m a data scientist working in Canva’s Gen AI team. So I spend a lot of time looking at how people use Canva AI, and how they use it to optimize features and help teams make better product decisions with data. Some of the projects I’ve been working on recently is a prompt analytics platform. This is a project that came out of an AI discovery week, which is basically a week at Canva where everyone working at Canva doubles down on some cool projects, and there is no business as usual in that week. So yeah, that’s something that has been quite fun to work on.

00:21:41.21 | Moe Kiss: But Nina, can I say this one in particular might need a rebrand on the name, because the name of this one is called CATNIP. Can you tell us a little bit about CATNIP?

00:21:51.71 | Nina Kumagai: That sounds fun. I don’t know. Thanks, Moe. Yeah, about CATNIP, so it’s a funny one. It actually originally used to be called Cat NLP, but I think one of the PMs that I was working with gave some feedback and ended up becoming CATNIP. Basically, CATNIP is a prompt analytics platform and it stands for Canva AI, Canva AI, actually I now forget, Canva AI tool, natural input prompts, which is a very long name, so the acronym definitely helps. But yeah, this, tool was basically built under a week to help enable a lot of our stakeholders to understand what exactly users were doing on Canva AI. So to set some context as well, when we first began our department, generative AI department at Canva, it was roughly maybe two to three years ago. And at the time, our AI usage was a bit of a black box and no one really knew, what users were actually doing with our AI features. And so our prompt data was one of those really critical data points that would help our stakeholders and a lot of machine learning engineers or product managers or, yeah, a diverse group of workers at Canva to understand what users wanted from our AI, and what exactly they wanted. And so, we started to work with our AI, and what exactly different types of customers were doing with our AI features.

00:23:41.71 | Moe Kiss: So tell us a little bit about, you had this AI discovery week and you were, from memory, you had a bit of a team working on it. What was kind of the, like the project plan where you wanted to get from and to, through the course of that week?

00:23:54.71 | Nina Kumagai: A hundred percent. So yeah, initially, it sort of lined up quite well with AI discovery week, but we had this really, massive stakeholder request, and there are a lot of different questions on understanding how users were running AI presentation generations. And instead of trying to sort of answer that sort of ad hoc that I guess a lot of data scientists get, we decided to build a more scalable system so that a prompt analytics platform could help, a lot of different stakeholders resolve and self-serve some of those questions without data being a bit of a blocker and getting access to that prompt. So sort of, I guess, yeah, timed quite well with discovery week, but I’d say a lot of the groundwork happened probably one year before discovery week happened. So, given, prompt data is quite difficult to work with. We have a lot of compliance and privacy engineering and legal were folks that we had to work with a while back before we could build a system like this.

00:25:21.22 | Moe Kiss: So Nina, prompt data, like, I feel like we all have in our mind a little bit of like what some of the challenges are. I know talking to you all, I’m privileged. I get to sit near a lot of the Gen AI folks, but like, what are the challenges? What makes prompt data? So unique and different to the other types of data that data scientists are working with.

00:25:39.42 | Nina Kumagai: So the number one thing is it’s definitely sensitive data. And so when building a platform like this, we had to make sure number one, we walked alongside privacy engineering security and legal and every sort of step that we took. We had to get approval from those teams to ensure we were compliant with a lot of those, I guess, policies that we have. And number two is that I guess we had to do a lot of filtering, sanitization, making sure that some segments of data could never be accessed by anyone in the team. And of course, there were some mechanisms where if users wanted access, they had to go through quite a rigorous approval system. And we sort of had to approve every user on a case-by-case basis to make sure that there was a good reason for having a requirement to look at this sort of prompt data.

00:26:58.14 | Moe Kiss: And so we had a former guest on the show. His name’s Dr. Peter Fader. And this quote has just stayed with me for like a decade. But he said, you need to embrace the randomness of customer behavior. And anytime I was doing analysis, when you’re just like, why does this make no sense? Like, why did someone put a million dollars worth of like quotes in their checkout? I can imagine like prompt data is the absolute end degree of that, right? Where like you see all of the randomness. What’s that experience like?

00:27:26.96 | Nina Kumagai: It’s so interesting because when you’re working on Canva AI, you have this sort of idea of what users are doing with it. But when you read the prompts, it can be anything from just one word to maybe an essay. It can be anything from maybe a student asking for a cartoon or meme to an enterprise customer trying to build a branded presentation. And so that sort of range. As you just talked about, was quite difficult to work with, to be really honest. And it definitely made, I think, one maybe strength of doing a very quick prototype during AI Discovery Week was that we built the prototype really quickly. But then we kept iterating as we got more feedback from a lot of different stakeholders. And so. So every stakeholder that we had had a different sort of use case for looking at prompt data. And depending on that use case, maybe we needed to chuck in a different filter. Maybe we need to add in a mechanism for semantically searching through the data. So I guess, yeah, it was a very broad problem. And even now, I think it’s not something that I think has been like 100% fixed. It’s still, we’re still iterating on how we understand prompts and label prompts and categorize them.

00:29:12.00 | Julie Hoyer: Is CATNIP then an easy interface for stakeholders to, like, analyze the data, or was CATNIP part of… because you were saying it took like a year to get the data piece kind of ready. So what does, like, CATNIP look like, or when a user goes to interact with it, what’s that experience?

00:29:28.80 | Nina Kumagai: Yeah, so CATNIP is quite easy to use. So what was interesting was when we first built out CATNIP, I think high level I was thinking maybe product managers, machine learning engineers, maybe some QAs, but now I think we have a lot of different type of subspecialties using CATNIP and basically when you go into the interface you sort of see sort of a grouping of different filters and once you submit sort of that segment that you’re interested in analyzing, it words in maybe 10 seconds and then shows you maybe a subset of prompts with some high level insights on how things like publish rate or conversions or feedback change. So it’s a very, I think part of the reason why it sort of started small and was only used in the Gen AI department initially, but it sort of grew out over time to a lot of different departments and different specialties. We now have maybe trust and safety, security, growth, marketing, product teams using CATNIP and I think part of it is because it’s quite easy to use and it sort of can mold into what the user wants it to be where if they need to search for a specific keyword that’s possible if they want to filter down to some specific segments it’s quite easy to do as well.

00:31:16.96 | Julie Hoyer: So could you say how many people are on here trying to make like birthday invitations for their one-year-olds like that type of thing?

00:31:24.24 | Nina Kumagai: That’s right, yes. Yeah.

00:31:25.75 | Moe Kiss: Julie’s exact use case.

00:31:27.68 | Julie Hoyer: Guilty. Have done that on Canva AI.

00:31:31.48 | Nina Kumagai: It’s quite useful because I think every project we have at Canva is very different and so like maybe there are use cases where we do need to look at those sort of birthday invitations specifically and try to improve and add very niche features to our AI product to enable that. And I think machine learning engineers might use it to get like a SQL output and then use that to find your birthday invitations, or product managers might be using it to build a roadmap of where we want to go with birthday invitations. So yeah, that sort of thing is, I think, part of why it sort of took off.

00:32:18.50 | Michael Helbling: Nina, I’m very curious to learn if you’ve identified or seen any kinds of user trends emerge. Because obviously, like, prompts from users, it’s sort of a net new sort of usability interface, if you will, that a lot of companies are dealing with. And so like, there’s not any data out there really to kind of speak to like how people use it. Like when the AI first came out, of course, there was people writing about how we’re all becoming prompt engineers. But I think only a very small subset of people probably think or write prompts in that way. So like, have you observed over the corpus of the data, like any interesting trends of how users interact with or create prompts in the system?

00:32:56.00 | Nina Kumagai: It’s definitely changed. And I think part of the way it’s changed is based on how the product itself has evolved. Early on, when we first launched our Canva AI product, we actually ran a beta program. And it was only out to a certain number of users that sort of, I guess, volunteered to test the new AI product. And we had sort of like a Discord sort of chat going on, and we could talk to these customers while also looking at what kind of prompts that they were submitting. So back then, I think a lot of our use case, it was quite like nicely fleshed out data. Maybe now that it’s a bit more, it’s a bit more mature as a product, we’re seeing way more, maybe diverse prompts. And if we have like a huge education user subset, then we’re seeing a lot of education related prompts. If we’re looking at enterprise customers, we’re looking at a lot of brand related prompts. So I think, I guess, it sort of evolved on the way with the product. And I think, as well as it gets easier to use, maybe it sort of unlocks other sort of demographics to use the product as well. And so we sort of rolled out over time to more languages, to more markets. And we definitely see now that we have to translate prompts and the raw prompts are in like different languages. And so… Yeah, it opens up a whole other can of worms. Yeah, a whole other level of complexity. It’s a… Yeah, pretty complicated problem.

00:34:55.55 | Michael Helbling: That’s awesome. Well, Nina, thank you so much for taking the time to come on the show. I really appreciate it. Really appreciate your time. All right, turning now to Jack O’Hagan. Jack, welcome to the show. What do you do at Canva? And what kinds of projects are you working on?

00:35:11.09 | Jack O’Hagan: Hey, everyone. Good to meet you all, and thanks a lot for having me on. At Canva, I’m a data scientist in a team called DCV, which is Discover and Connect Value. Essentially, it’s a team which has a scope across MAU and also paid retention and upgrades as well. So it’s quite a broad team, which is kind of how this skill came about. It’s like a skill that looks across Canva and all the different experiments we’ve run. And yeah, it kind of fits quite nicely with our team having such a broad scope. We need to understand pretty much Canva as a whole, I think, yeah.

00:35:49.70 | Moe Kiss: So basically, Jack-O gets periodic messages from me being like, Jack-O, this analysis that you did last year, can you tell me about this? Hey, Jack-O, blah, blah, blah. Where he did this really cool meta analysis of, we obviously run a lot of experiments at Canva. I’m wanting to look at some of the metrics that moved. But Jack, can you tell us a bit more about how you’ve now taken that meta metrics that moved. But Jack, can you tell us a bit more about how you’ve now taken that meta analysis work you’ve done and how you’ve scaled it?

00:36:16.50 | Jack O’Hagan: Yeah, yeah, definitely. And just on that point, it’s part of the problem with having such a broad scope. You get like random messages from people from like who you’ve never met before, asking about something you’ve done. So yeah, that’s always fun. But yeah, this skill came about, it’s actually the evolution of something we’ve kind of worked on over the years. Initially, it was like a purely Python and SQL sort of really descriptive thing. And it was quite limited. And with AI, we’ve been able to evolve it and package it up really nicely, so other people can use it. But the skill itself is basically a way to look at our experiment data at Canva. And we run a lot of experiments across Canva. And often, it can be hard to understand, or even see the results of really niche experiments that might be really interesting, but it’s only known about in a really small team or in a group that might not share the results widely. So part of the skill is just surfacing interest in experiments that have moved a particular metric. And then the other part of the skill is trying to just synthesize a lot of the information from experiments, not just the result itself, but context about the experiment as well. And packaging all of that up into something that can neatly answer a question about how often a metric moves and the types of things that do move a particular metric. So yeah.

00:37:48.32 | Julie Hoyer: Who is using this skill? I mean, it sounds like you have to be very knowledgeable about what you’re asking, like what experiments or group of experiments or like, is it just data scientists using this skill? Or like, are you like, hey, and stakeholder who randomly slacked me that I’ve never met, go ahead and ask it your question.

00:38:07.40 | Jack O’Hagan: Yeah, right now it is probably more data scientists. It can be, we’ve tried to make it simple enough to use. So anyone could just prompt it with a metric and also with the group of experiments they want to run it across. And it will spit out the results for you in like a HTML file. These are the experiments that moved it, these are the experiments that didn’t move it, and then pulls out themes around the experiments. So technically, anyone should be able to run it, but it is at the minute, mostly data scientists.

00:38:40.82 | Moe Kiss: Is it fair to say like, so one of the problems, I mean, this is the problem that I faced, which is how we overlapped on this work stream, was, you know, a team potentially runs an experiment, it might move a metric that another team is responsible for, but they’re unaware. Was that actually the core problem you’re trying to solve? Or like, what was the core? How did it start out? Like, what drove this development, I guess?

00:39:04.90 | Jack O’Hagan: Yeah, that’s a good question. To be honest, yeah, that wasn’t really the core problem we set out to solve. I think the core problem for us was one of the metrics like our team specifically really cared about was paid retention or cancellations is the metric we look at in the experiments. And we had a really good idea of the types of experiments that move cancellations in our group. And we had this kind of shared knowledge just through like, sharebacks or like, discussions within our group. But we didn’t really have a good understanding of the things outside of our team that moved this metric. And when we’d seen results here and there, like a company wide sharebacks that someone had shared, we were often surprised that this experiment moved the metric that we cared about, we couldn’t really understand how it moved cancellations. And there were a lot of interesting results like that, that kind of cropped up here and there. So we wanted just a better way to be able to see which experiments will move in that metric scale. So we could feed that back into the product strategy, basically. So that was the core thing we were trying to solve. And then a kind of nice part of that has been we’ve just been able to see and keep an eye on experiments at the same time, but also come up with like a bigger strategy that’s not just as narrow focus as our team initially had, I guess.

00:40:33.03 | Michael Helbling: I’ve noticed in doing some skills work myself that sometimes when you start out, you can sometimes create a skill that makes the base LLM like Claude perform even worse than it did before. And I’m wondering, like, I’m sure you did tons of like evals and things like that. But like, were there components that surprised you that you needed to bake into the skill structure that, or not just surprised you, but like, were there any kind of light on moments that were sort of like, ah, this is how we get it to perform a little better when I when building out sort of the skill set?

00:41:04.62 | Jack O’Hagan: Yeah. That’s a really good question as well. I think, yeah, it was an evolution as we built it. And we kind of ended up in a place where we took a lot of control away from the LLM. I think at first we just tried to solve everything with the LLM with a prompt and gave it access to everything. And the results were quite hallucinatory. They weren’t very reliable. So we revisited it and tried to structure it a bit better where grabbing the actual results, was purely done in SQL. We have like experiment metadata tables that store metric results. So it’s actually quite easy to grab the results. It’s just the SQL query. It’s a little bit more tricky if the experiment didn’t have the metric applied initially. There’s a bit of Python and SQL that you need to write there, but you can still do all of that in Python and SQL. And even the metadata tables have a good set of descriptions about the experiment. They have the name, the title, and variant descriptions. So there’s some context you can get from those tables, but the problem is the context is not great. People don’t always fill it in. A lot of the time it’s null, and you can’t really use that to pull out the themes. So we really limited the LLM to just searching, pretty much. Searching like our internal documents, or even slack, grabbing context from there, and then synthesizing it. And even with that approach, we did notice some hallucinations and issues. So we pretty much instructed it to only grab verbatim quotes from, from either slack or the experiment documents. Because the good thing about the experiment documents, they were kind of used as a place for the team to discuss the experiment and set the parameters of the experiment. So if we can just grab direct quotes from there, we, we knew it could be trusted. Um, so yeah, we ended up in this place where a lot of it was actually done. We’d used some kind of like SQL and Python and then the kind of searching and synthesizing bit was done with the LLM, but limited really to direct quotes from people.

00:43:16.05 | Moe Kiss: So you’ve talked a little bit about overcoming the challenges with hallucinations, etc. What were the other technical challenges? Like it sounds, which is very top of mind for me at the moment. If anyone’s been hanging out with me, um, the, the quality of the context of the experiment. But what, like, that you had to overcome?

00:43:35.55 | Jack O’Hagan: Yeah, there were. So yeah, the big one was the quality of the sort of context and stopping hallucinations. But there were some other issues to do with the actual experiments themselves, especially looking back retroactively. Sometimes you could look at an old experiment and it had already been rolled out, but it had been continued to be analysed, like someone had just forgot to stop the analysis. And that would mean like one variant would have way more enrollments than the other. And there’d be like a split warning in our experiment platform. And often those were the results that looked really interesting. So we had to make sure we ignored those basically. We couldn’t really use them because they were sort of statistically wrong. That alongside getting the right context were probably the biggest challenge.

00:44:32.71 | Moe Kiss: So data hygiene, that fun, unsexy thing that data people care about, but the rest of the business are like, why are you spending time on this?

00:44:40.75 | Michael Helbling: Definitely. Did you ever find that the LLMs sort of stumble across something they take as a truth and then just never let go of it? Like I find that all the time where it’s like, oh yeah, we found this this one time and now it’s always true all the time. And we’re just going to apply it to everything. And it drives me crazy. But I was just curious, how do you manage stuff? Like that?

00:45:03.29 | Jack O’Hagan: Yeah, that could happen. And it happened a lot with experiments. Like the way people name experiments at Canva, often they might create a version two of an experiment that has a really similar name and really similar context to the previous one. But it could be quite different in terms of what it changed. And we noticed a lot of the time, it might place a really similar experiment to the old experiment. And you’d kind of get your wires crossed like that. So we set up some kind of strict rules where it could only use this experiment for this result. And we made sure like when we first did it, we only it ran sort of the same set of experiments every time. And it was quite wasteful. So now it’s like once we grab the context, we save the context down and it doesn’t need to run. It again. So we kind of avoid that issue a bit more, which is good.

00:46:07.40 | Moe Kiss: This is a really mean question. But a lot of this is looking post right at an experiment after. I suppose one of the things I’m always thinking about is how we influence decisions moving forward. What are your thoughts on that? Do you think there are next steps here? Do you think that us looking retrospectively is sufficient? Like, I’m just curious, I guess.

00:46:31.28 | Jack O’Hagan: Yeah. So that’s a really good question as well. And I think when I do share this with product managers or different people who might decide the strategy of where we want to go next, one thing I am really conscious of is we can really only learn from things we’ve done before. Like if we haven’t tried something, it won’t show up in the data. And it doesn’t mean it’s invalid. It might be a really good idea. So I wouldn’t want to share the results of this meta-analysis with someone and say, oh, these are the things we should only do. And these are the things we should ignore. And often some things that didn’t work in the past, like there might be a theme of things that didn’t work in the past. It can be sometimes hard to completely discount it because the reason they might not have worked, it might be to do with the execution. Times might have changed as well, that the users preferences might have changed. So I think you always have to try and approach the results with a bit of your product strategy hat on. But that is probably the trickiest part, to be honest, making sense of the data and making it apply to the future. The other thing which is kind of related to that is if you do any sort of meta-analysis like this, make sure to include all the cases where it’s neutral results. Because if you don’t, things get really bad. So I think that’s the biggest thing. If you’re really biased and you think, oh, this surface looks really great, there’s been 10 wins on it, but there’s actually been like a thousand experiments on that surface. So one thing we learned as doing this was to also include all the context of the things that didn’t just fail, but were neutral as well.

00:48:17.84 | Michael Helbling: Outstanding. Awesome. Jack, thank you so much for coming on the show. I appreciate it. All right. Turning now to Flynn Alexander. Flynn, welcome to the show. What team are you on and what kind of projects are you up to?

00:48:31.51 | Flynn Alexander: Hello. Thanks so much for having me. I’m Flynn. I’m from the Marketing Data Science Team. I primarily work across our page metric space. So looking at landing pages and how people are arriving at Canva. And I sort of like to think of myself as like the person who’s trying to tell the story of how they’re arriving, putting that narrative together and then going on. How do they get into the product? What are they doing? Are they converting? Bringing that entire journey together into like one cohesive narrative, I guess.

00:49:04.73 | Moe Kiss: And Flynn, your project, I have to say, wins the prize today for the best name. Can you talk to us a little bit about the GOAT model?

00:49:11.90 | Flynn Alexander: Yep. My coach is not going to like that you’ve said that. He’s definitely opting for the name change. But yes, the GOAT model is the Global Overview of Activity and Transitions. And essentially what it’s doing, and I will say this is probably less of a dedicated AI project and more of a traditional kind of marketing stats problem. But AI has sort of come into this and helped smooth over the process and turn it into something that is becoming a more powerful tool or becoming a more powerful tool for our kind of stakeholders to use, right? At its core, the GOAT model is applying a Markov chain to users arriving at Canva and looking at things like web pages. Um, also kind of looking at like, where did they come from? Right? So we take a page view as like a state. And essentially we say, given the state, um, what is the probability of someone going to a new state, page A, page B from that state? Like, do they go to the pricing page from the pro page? Um, and we can build a sort of network of transition probabilities. That determine the most likely pathways that someone might take as they move through the product. So that’s a really powerful tool. And it means that we can kind of get a, an overview of what our entire system looks like in one go. Right. And I would say that like this data itself is not particularly human interpretable. It’s, you know, we were dealing with thousands and thousands of web pages and entry points and, uh, a Markov chain does a really great kind of job of condensing that down just to the, the pathways that you’re more interested in. So what are the top five pathways that get someone to, to this part of the website or convert on this part?

00:51:19.45 | Moe Kiss: I remember when you were starting out this project, um, and Dr. Peter Fader once again comes to mind, embrace randomness of customer behavior. And I remember talking to you, like I’d done. Similar work previously. And one of the hardest things is like, what does top five actually mean? Because often users are so like unique and different that to even get a top five, it might only be like 0.000001% of our users. And then how is that like useful? Like talk, talk us through like how you overcame that challenge because I feel like you did

00:51:54.33 | Flynn Alexander: definitely still a work in progress, but yes, we’re working on it. Um, the, the, the way. That I would say is that like, you, you sort of have to, uh, pick your objective first, right? So like, what does success look like? And success in some cases is like just getting the most amount of people to a page, right? But they might not necessarily be the people that go on to convert. So if we have two different pathways that people head down, um, one might have a very, very low traffic volume, but the people on the coming down that pathway. Okay. A super high intent. So 90% of them go on to convert and maybe that’s the kind of people that we’re trying to target. We’re trying to get as many conversions as we want to get through the pipeline. Uh, the second pathway might have a thousand people coming through, but only 0.01 or something of them convert. Right? So you kind of have to balance what, um, what your targeted objective is and take that into consideration when you’re looking. At like the overall picture, I guess. And that’s, that’s kind of like a difficult problem that, um, a lot of, it’s probably more of a data science problem than it is a problem that you would give to your stakeholders, just given the complexity. Um, and one of the challenges that I sort of found with, with this tool was that I could put it in the hands of, of our stakeholders, but immediately it was followed by just a flood of questions and like, how do I use this? What does this mean? Um, these numbers that like, don’t really have all that much meaning to a lot of our stakeholders when they’re just trying to make practical like business decisions. Right. So I think the second half of this project, which is the part that we’re like actively working on now and trying to build out is like where like AI sort of comes in and like really helps, uh, build out what this. Could become. And traditionally, like when we first started this, we were trying to, to visualize this information through like Sankey plots, um, funnels, all that kind of stuff. Right. But it, it just, it doesn’t, uh, it’s difficult to create a system where the solution fits many different, uh, approaches, I guess. So. As I was experimenting with that. And using generative AI and the, the latest frontier models and whatnot to, to kind of help me build it. I realized that these models themselves are almost like the perfect UI to, to be using this kind of tool. Right. So they can provide, uh, all of the, you know, they’re essentially like a resident expert sitting next to you. They can help you target what you need. They can provide the parameters to, to input into the model. Um, they can then take the main pathways that are identified by that model, talk about them, apply stories and narratives to it. You can, you can essentially, if you have, uh, a pathway come back that says, you know, users go from A to B to C. I can tell the model to create a story around that. That’s easy for stakeholders to understand. And everyone knows that like generative AI is really good at that. And it’s really, really good at making stories. So this stuff becomes a lot more interpretable. Um, and that’s been like a really, really powerful realization, I think, coming out of this and seeing how we can kind of, you know, have a resident expert sit alongside and kind of get the most out of these tools.

00:55:47.46 | Julie Hoyer: So do you have it? I feel like I keep asking kind of like the user experience of these tools, but I’m so curious, like how you guys are designing and building these things to then help, you know, your different stakeholder sets, or if you guys are using them. in your own, um, analyses? Is it that you had the GOAT model and now you have, like, is it kind of like a chatbot interface on top of it, or, like, how… what does that experience feel like now that you have generative AI, like, looped in to help?

00:56:16.00 | Flynn Alexander: So, definitely preface that this is all still, like, a work in progress, and we don’t have, like, a fully productionized version out yet. But the way it works is that, like, the GOAT model itself is like an underlying foundation of, like, the stats and the math that gets done, we don’t really want the generative AI to kind of do any of the computation itself, right? Mostly because it’s quite complicated math and like hallucinations, all this kind of things will create problems. So it’s almost just like a wrapper around that where it can like plug in the inputs and then go on to like interpret the outputs, much like it would for like a semantic layer, right? So the intention, how we kind of execute this within Canva itself, I suspect will just be like internal tooling. Otherwise, we might look to partner with some of our kind of new reporting partners, maybe someone like Omni, not too sure. But yeah, it’s still definitely a work in progress.

00:57:18.51 | Michael Helbling: About a month or so ago, like a lot of noise happened in the AI space because of this new LLM that launched. It’s a decision LLM, Jev. And I was wondering, like it seemed like in initially reading about it, it may have some applicability to things like simulations and things like that. Have you looked at that at all? Or do you find it to be something that could be useful in that regard?

00:57:39.66 | Flynn Alexander: Personally, I haven’t.

00:57:40.18 | Michael Helbling: You can defer on that too if you don’t. It literally just came out. So it’s not like you have time to dig into it.

00:57:47.86 | Flynn Alexander: Okay, no, I’ll have to look into it. I’m not too sure about Jev.

00:57:51.90 | Michael Helbling: Yeah, yeah, that’s totally fine. And I think OpenAI, like a couple of weeks, They launched their own version, but decision engine versus like a text-based generative AI engine LLM.

00:58:05.26 | Moe Kiss: What’s the difference, Helbs?

00:58:07.38 | Michael Helbling: Now you’re putting me on the spot. The tables have been turned. I’m like, I need to go look this up. So a traditional LLM generates text as a responses or code or things like that. A decision LLM generates a probability or a decision that this is likely to happen or not happen. So in other words, like, is this true or false? And you give it a scenario and it says, probability is probably about 0.7.

00:58:31.00 | Moe Kiss: Is that not just adding a probability to the answer though?

00:58:34.30 | Michael Helbling: Yeah, it basically is. It doesn’t, the point is it doesn’t generate text and therefore it’s like much cheaper to use and that kind of stuff. Anyways, let’s not go down into it too far. Sorry, Flynn, to put you on the spot like that. Just curiosity.

00:58:48.76 | Flynn Alexander: No, no, it’s totally fair. And I mean, it sounds like it could be something that we potentially could use, but at the same time, you know, we have all of this like abundant, rich, kind of like page impression data and like click data available to us already. So for us personally, like it’s very easy for us to just generate those like actual probabilities very quickly.

00:59:13.26 | Michael Helbling: Yeah. And that was the other thing, because we did a show a while back on simulations, and I’ve always felt like this is an underutilized tool for a lot of marketers. And so this is a very fascinating topic. Just in its own right. So like how did you kind of leverage? I guess you leverage historical or existing data to kind of build out your model for the simulation itself, I assume?

00:59:39.42 | Flynn Alexander: Yeah. So I mean, that’s the great part, right? Like one of one of the big projects that we’ve had to push as part of this is just making sure that the underlying data itself is like super, super clean and normalized. Right. So If you’re going to ingest anything into a Markov chain, the state and what the definition of a state has to be very consistent. And that means that making sure how we are tracking page impressions, page views, and potentially in the future, we could even extend this to include CTA clicks and interactions on the web pages themselves. But all of that needs to be put into tables in a way that’s incredibly clean, structured, and consistent across the entire business. And I’m sure that anyone that has worked in a business as large as Canva knows that consistency in engineering across the entire business is always a bit of a challenge. So that’s definitely something that has been top of mind for us. And we’ve been working on getting that foundational data layer super robust so that we can ingest different time periods And we can compare across time periods. Because it really is just looking. You grab one month of impressions from a year ago versus one month of impressions from now. You can facilitate comparisons.

01:01:16.87 | Moe Kiss: Yeah, there’s really a lot of potential for the tool and how you could use it. I’m going to do the mean thing to you that I did to Jack-O. I’m always thinking about the future, right? So, like, we’re trying to understand the opportunity of, like, a new page, surface, whatever it is. We obviously don’t have historical data, but I do feel like this data set could actually potentially be quite useful in helping understand the likely performance of something based on similar attributes. Is that something that you’re thinking about?

01:01:41.11 | Flynn Alexander: 100%. And that sort of builds into the other work that we’re doing around building up that foundational layer and improving just the way that we’re looking at the data. And I think it’s around categorizing pages, right? Because if you are able to categorize pages efficiently and make sure that groups of pages are kind of associated, then if we are to release a page in the future that is in line with other pages that are currently performing, yeah, we could definitely do some kind of projections or simulations to kind of understand how that page might perform. And how it would fit into the existing network of pages, right? Because essentially, that’s just adding a new state.

01:02:35.05 | Michael Helbling: Awesome. Thank you so much, Flynn. And I want to say a huge thank you to start to wrap up to all of you, Christian Sullivan, Nina Kumagai, Jack O’Hagan, and Flynn Alexander. Thank you all so much for coming on the show, sharing your insight and knowledge. I think to me, the coolest thing was as we were talking, I know we literally only scratched the surface at a very high level, but even seeing, sometimes when you’re working with new tools and AI, you sort of feel like you’re on an island, sort of, like, making stuff up as you go along. And so hearing from your experiences, certainly for me, it’s highlighted and identified a bunch of things. It’s like, oh, yeah, I have something similar. And so that experience was top-notch for me, especially. And so thank you very much. I’m sure our listeners also got a lot out of it. So thank you. This is sort of an unprecedented look. I don’t think I’ve ever seen anybody do a thing where they dive into like different topics and categories and teams within one company and kind of look and see where AI is impacting it. So really appreciate you all taking the time. And I know if you’ve been listening, you probably have your own questions and we would love to hear from you. Our listeners, you can reach out to us. And the best ways to do that are through our Measure Slack chat group or LinkedIn or via email at contact at analyticshour.io. And if you’re listening, we’d love it if you’d leave a review on your favorite platform and let us know how you like the show that way as well. And if you’d like a sticker for your laptop or your water bottle, we’ve got some and we can send them to you. Just reach out on the form that’s on analyticshour.io. There’s a special form just for that. And we will send you one out. All right. Well, I have loved this episode. Can’t thank you all once again so much. And I know I speak for both of my co-hosts, Julie and Moe. No matter where you’re applying AI in your business, keep analyzing.

01:04:26.31 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at at analyticshour, on the web at analyticshour.io, our LinkedIn group and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst.

01:04:44.27 | Charles Barkley: Smart guys want to fit in. So they made up a term called analytics. Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition.

01:05:03.07 | Julie Hoyer: Rock flag and weird Canva project names.