#298: Listener Questions Answered Live from Marketing Analytics Summit!

00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour.

00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language.

00:00:15.56 [Jim Sterne]: The Marketing Analytics Summit is pleased to present these four amazing podcasters.

00:00:24.80 [Jim Sterne]: We have Michael, the self-effacing every-man consultant who knows far more than he lets on. We have Moee, who flow all the way in from Sydney, Australia. She’s the determined, driven practitioner, the one to stand up and say, well, that’s all very well and good, but how do we actually make it work? There’s Val, who co-founded the Consultancy facts & feelings, because we have strong feelings about our facts and no facts about our feelings. And there’s this guy named Tim. Ladies and gentlemen, take it away.

00:00:56.72 [Michael Helbling]: Hi, everyone.

00:00:58.22 [Michael Helbling]: Welcome to the Analytics Power Hour, this episode 298. And we are recording live at the Marketing Analytics Summit in the beautiful Santa Barbara. You know, 21 years ago, my very first Analytics conference was right here in this city at this conference, formerly known as E-Metrics. And similar to today, it was a conference full of smart, engaging, and passionate people learning together about how to solve the problems they were facing day-to-day in the analytics industry. A lot has changed. A lot has changed. But persistent through all of that is this beautiful analytics community. And the shared learning, this podcast was actually created to encourage and celebrate. So to that end, I have my co-hosts with me. And we have put out a survey to gather your questions. We’ve been asking you during the conference. And so we have a number of them to get through, and we’ll do our best to answer them from some of the perspectives we bring to the table in our various roles. So let me introduce them. Moee Kiss.

00:02:13.44 [Michael Helbling]: Hi, folks.

00:02:14.44 [Michael Helbling]: Hi.

00:02:15.44 [Michael Helbling]: Director of Data for Product, Canva.

00:02:19.44 [Michael Helbling]: And of course, Tim Wilson, Head of Solutions at facts & feelings.

00:02:24.16 [Michael Helbling]: Hello.

00:02:25.48 [Michael Helbling]: And Val Krull, Head of Delivery at facts & feelings.

00:02:29.04 [Val Kroll]: Hello, hello.

00:02:30.04 [Michael Helbling]: And I’m Michael Helbling, my president of Stacked Analytics.

00:02:34.72 [Michael Helbling]: Okay.

00:02:35.72 [Michael Helbling]: What a privilege to be here with all of you in person. It’s been a wonderful couple of days. So let’s just dive right into it.

00:02:45.36 [Michael Helbling]: So we’ve got a question.

00:02:48.24 [Michael Helbling]: This one came from James from Namer’s Children’s Health. What’s the best approach to establishing yourself as a partner and not an order taker when working with stakeholders?

00:02:59.12 [Val Kroll]: James is the one who has the most trouble with the uncomfortable pause. So that’s why he looks the best.

00:03:09.76 [Moe Kiss]: Oh, I assumed that was your cue. Anyway, we were having a conversation earlier about taking notes over lunch. And if you don’t want to be perceived to be the person in the room taking notes, then don’t take the notes. And I think it’s really jarred with me because I have this belief, I mean, I always know that I come back to everything that Cassie Kazarkoff says about just be useful. And I am sometimes fine. I’m still like, I might be the most senior person in the room and sometimes I do still take notes. Part of that’s my brain, but part of that is also that I want to be useful and the people in the room are normally smarter than I am. And I have much more interesting things to say. So I actually kind of struggle a little bit with the question in and of itself. I think one of the things that maybe means that it’s OK to still take notes, but is also adding your voice to the conversation.

00:04:02.44 [Michael Helbling]: I think, I don’t know, I just, again, we were chatting about this at lunch.

00:04:06.20 [Moe Kiss]: When you don’t contribute to the conversation, when you’re not willing to be accountable for what you say, I think that’s when you kind of, you’re not the partner, so to speak.

00:04:15.36 [Tim Wilson]: I mean, I think it’s, I agree.

00:04:17.00 [Michael Helbling]: I think it’s don’t be an order taker.

00:04:19.12 [Tim Wilson]: I think to me it is, and there have been a lot of discussions at this conference, a lot of the discussion in the industry were at large around how are we using AI and there to be curmudgeonly and a little cranky about it. But a lot of is, how do I take more orders? How do I take more orders and produce more output? And that misses, to me, the fundamental part of becoming more of a partner is to step into their shoes, ask the questions, write it down, be genuinely curious. Don’t be trying to get to, how quickly can I turn this into a ticket? And I think that has been, AI has nothing to do with that. That has been a 20-year problem in our industry when we sit around and wring our hands and gnash our teeth about why aren’t we brought in. We produced the next dashboard like they asked us to, we gave them recommendations, but did you ever actually sit down up front and say, I just, I’m an analyst, I’m curious, I want to understand what you’re grappling with. And there have been, even at this conference, there have been some discussions around that. But I, I lay awake at night fretting about the industry being, I want to be a partner, but I behave like an order taker. And it’s like, it’s simple, just be curious, go ask them, get inside their heads and start asking them, how can I have some problems?

00:05:41.96 [Michael Helbling]: Yeah, okay.

00:05:42.96 [Moe Kiss]: I kind of want to call a bit of bullshit on it though, because I do, I get-

00:05:46.88 [Michael Helbling]: On brand?

00:05:49.72 [Moe Kiss]: I get what you’re saying, and I hadn’t really thought about like AI driving more of the order taking. I actually feel like my experience might be a little bit different. I think the thing that I just keep observing is data folks don’t want to be accountable. I had this conversation at the back of the room earlier, it’s like, they want to be like, here’s the numbers, don’t look at me, like I don’t want to be on the hook for the recommendation that I’m making. And I don’t know, I don’t think that’s an AI thing. I think that’s just a, maybe it’s a 20 year old industry thing.

00:06:19.68 [Tim Wilson]: I think it is, I mean, I actually agree that it’s like, we make recommendations and they’re not following them. And a lot of times like, where did this start? Well, we asked them what they, we asked them what their business question was. If you’re asking them, even if you’re saying, I’m asking what the business question is, there is a human element of saying, I am your partner, I am in this with you together. So I’m just pushing for moving upstream and not jumping to the data and jumping to the solutioning and jumping to, did I produce the published report that made a recommendation and then I’m upset.

00:06:53.84 [Michael Helbling]: So you didn’t fix that?

00:06:56.88 [Val Kroll]: Yeah, I’m just kidding. I would say like to get started with that, it’s not even always about like anticipating all of their needs. I think it is being curious and just thinking about the motivations and the stresses that that person has in their role, like what pressures are they facing? And I think that this has come up a couple of times too about how do we de-risk the decision that’s upcoming for them or where should they spend that next dollar and do a little bit of prep work before you walk into that room to show that you’re trying to empathize with what they’re dealing with because their job is hard too. It’s very different than yours, but starting from that place, I think is a good way to kind of say like, we’re in this together, we’re side by side, not sitting across the table and I’m just going to push something on you at some point.

00:07:38.52 [Moe Kiss]: But I think the difference there is you talked about preparation and I think a lot of the curiosity can come without necessarily absorbing stakeholder time. And I worry sometimes that the push is like, oh, well, I haven’t got enough context. My stakeholder is time poor. So, you know, that sort of thing where it’s, I actually think there’s a lot of personal responsibility that data folks need to take to be like, I have enough information available to me throughout various, various Slack channels, emails, et cetera, to have some of that curiosity before even asking the question of a stakeholder.

00:08:10.68 [Tim Wilson]: 100%. 1000%. I see your 100% and raise it. But I think if you’re thinking about it, you should be saying, this is my understanding, but here’s something that I can’t. I listened to our quarterly conference call and there was something that doesn’t make sense to me. I tried to figure it out. It doesn’t. I think you, marketing manager, might help me explain this because I can’t square that circle. So, yes, showing up and that certainly worked with analysts who say, well, what are my lists of discovery questions? What is your most challenging business problem? If that’s the way you show up, they’re going to be like, I have to explain every minute.

00:08:44.24 [Moe Kiss]: 25-year-old data slanted slightly because that’s kind of how they show up sometimes. No age.

00:08:49.84 [Michael Helbling]: No age. All right.

00:08:51.84 [Moe Kiss]: We’re going to move to our next question. This is the whole episode.

00:08:53.84 [Michael Helbling]: Sorry. And this is coming from a member who’s here in the audience, Michelle Kiss.

00:08:59.12 [Michael Helbling]: Whoo.

00:09:00.12 [Tim Wilson]: Any relation?

00:09:01.12 [Michele Kiss]: None. Completely the same person. Complete coincidence. Yes. Confuses everybody. Okay. So, first of all, my question has two parts. There’s a pre-question that is required for setting the context.

00:09:15.12 [Tim Wilson]: Yes, they’re related.

00:09:16.52 [Michele Kiss]: And so that we can interpret your answer. Okay. So, first of all, the pre-question is where do you guys stand on AI? Are you a skeptic or a fan? Okay. So, that’s the context. Then, what things have you personally found that it is useful for and reliable for? It has to be life.

00:09:42.88 [Val Kroll]: Okay.

00:09:43.88 [Michael Helbling]: I’ll go first.

00:09:44.88 [Val Kroll]: So, I would say that I’m a skeptic, but that’s just kind of like my nature. I don’t think there’s anything new or special about that. That’s just kind of who I am as a person, but I’m excited at the same time. And I think I’ve been inspired by a lot of the conversations we’ve been having over the past couple of days that have gotten my wheels turning about some different things that I can try when I get back to my desk. So, I would say more on the skeptic, but still excited, still optimistic. I don’t think, you know, world’s ending just yet. One of the things that I have been playing around, this is recency bias, but just thinking about how do I start at like day 60, day 90 with the where I’m at with an understanding of the business context for a potential prospect conversation. And so, we had been doing a lot of process effects and feelings very manually to kind of go through and listen to like most recent earnings calls or pulling out, you know, press releases or seeing what was publicly available just to kind of understand what is the context, what is the competitive nature, where are they competing and using AI and building out some gems to build that out for me. And so, I’m asking it to give me some tables and comparisons of where do they compete on what message is? Is this about pricing? Is this about quality? And so, really trying to figure out like how do I get into that context so that I’m not asking like question number one, what are your goals? It’s like, let me start with something because then I’m going to be able to ask much richer, deeper conversations. And so, again, recency bias, but that’s been the most fun one just to kind of understand outside of an analysis, what’s something I can do to show up in those conversations to kind of position myself to be a better partner.

00:11:26.88 [Moe Kiss]: Mine are not sexy. Like I don’t… That wasn’t sexy. Build me a table of your competitors. Well, it depends what you think is sexy. Yeah, I mean, I do a lot of leadership-y things and those things tend to be quite admin-y. I do think the one… Oh, I forgot your first part of the question. I’m excited and terrified. I would say I’m… I don’t… Fan feels like a strong word. I do like a lot of the potential and the personal gains in the way that it’s changed how I work, but I have this like level of fear, which I think is pretty rational and normal.

00:12:05.76 [Michael Helbling]: It’s a pretty big shift.

00:12:07.40 [Moe Kiss]: And yeah, a lot of the ways I use it are not sexy. But I do think the thing that has been really nice for me personally is feeling like code is not so far away. Like it’s been a couple of years since I wrote code very regularly, and there are things that now I feel much more comfortable. I’ll go do a quick Here’s the Queer I Want to Vibe code something.

00:12:26.56 [Michael Helbling]: And I also know enough to know if it’s not correct, which is important, but it makes

00:12:32.44 [Moe Kiss]: it feel like more approachable to a skill that’s pretty rusty. The rest of the stuff is like making my snarky comments in Slack, seem less snarky, like meeting summaries. Like, yeah, it’s not sexy.

00:12:49.48 [Tim Wilson]: So I’ll say I’m a skeptic and a fan. So I’ll punt on the first question. My skepticism comes from I’ve been developing this observation that I feel like we are as analysts, we are often saying this is the thing the tool can do. This is my hammer. So now I’m going to go figure out what the nail is, and I’m going to tell myself that the nail is going to solve a problem that has nothing to do with the technology. So I have found it to be very useful on debugging code, kind of piggybacking off of what you said, because coming from having written code, debugging it. It is very, very useful. I get very, very nervous when I have Vibe coded and yes, it brings results out. But I’ve now seen in the wild how that can go awry. I actually find that it’s very useful just from a thought clarity. And I think that hasn’t changed. That probably took me three times listening to Jim Stern, give various presentations about writing prompts for it to take. And it was the first time I heard it now, kind of everybody saying it of some of the keys with writing prompts, but prompts, I think by writing. So saying this forces me to organize my thoughts and then I’m going to trust and verify what comes back. That is much more on the help me clarify my thinking, because if you’re helping me clarify my thinking, just like a human being who I think, no, that’s wrong. That’s bullshit. You can challenge me, but you may be wrong. I am not, and which wasn’t the, where is it? Not useful, but I, anytime it’s like, it’s going to basically do glorified anomaly detection and spit out insights. Yeah, I’ll go to the mat for a while on that. And I’m pretty triggered with various claims to have it do that. Michael, what happens after you finish a GA4 analysis?

00:14:43.32 [Michael Helbling]: Oh, traditionally, I guess I paste screenshots into a doc, rename it final, final, uh, you know, do not change final, uh, lose the source query and then wait for somebody to ask, can we break this down by campaign? Terrifying. Please stop. I would love to emotionally and professionally.

00:15:05.12 [Tim Wilson]: Well, that’s why ask dash, why.ai just released the Prism Cod co-work connector. It brings the whole Prism brain to your GA4 big query data. Ooh, the whole brain analytics, agent harness, skills, memory engine, the works.

00:15:24.32 [Michael Helbling]: Wow. So co-work doesn’t just answer questions. It remembers context, uses repeatable skills, keeps analysis

00:15:30.82 [Tim Wilson]: organized, exactly. You’re, you’re picking it up. Your co-work based analyses are accessible in Prism, organized, traceable, auditable, and ready to use with your other data sets.

00:15:43.06 [Michael Helbling]: I love that. Cause currently my audit trail is mostly like, oh, I know I had a reason for doing that. I can’t remember what it is.

00:15:50.54 [Tim Wilson]: That checks out. The connector also ships with ready to run funnel and cohort skills right out of the box.

00:15:57.86 [Michael Helbling]: So I can ask for retention by acquisition channel and not immediately enter a fugue state.

00:16:02.58 [Tim Wilson]: Right. And every analysis becomes a shareable page. Prism auto-generates the dashboard page right from co-work. Oh, so the answer doesn’t die in a chat thread. That’s right. It lives as a reusable and shareable analysis. Well, that’s very rude to my old workflow, but fair. We’ll go to ask dash, why.ai. That’s ask dash the letter y.ai and sign up for the wait list. Yeah.

00:16:27.70 [Michael Helbling]: And use code APH and that’ll get you pushed to the top of that wait list. Ask dash, why.ai code APH. I like it. Co-work.

00:16:37.46 [Michael Helbling]: It’s GA for analysis, but with receipts, which was not part of the question.

00:16:43.74 [Val Kroll]: We’re only on our second question. You can’t, you can’t get triggered yet.

00:16:47.70 [Tim Wilson]: No, I welcome it. I was triggered 36 hours ago.

00:16:50.14 [Michael Helbling]: I think it’s okay. I go through life triggered. All right. We have another question from someone here in the audience. So I’ll hand it off to Jen Coons.

00:16:59.22 [Jenn Kunz]: It actually follows up with what Tim was just saying, perhaps a bit of it. Do any of the ways we promote and use AI in the industry make you feel icky? Do you have a threshold of lines that you don’t want to cross when it comes to AI?

00:17:12.98 [Tim Wilson]: Can I add, I’ll add a new one to that. I am not a fan of the note takers in meetings and doing the summaries, which I know is super controversial just inside. And because to me, I’ve watched and I’ve watched this time and time again, how much that drives laziness and not paying attention in the meeting and not editorializing. And it is flat summaries. I’ve worked with some clients where they have literally said, oh, you weren’t at the meeting, we recorded it. Here’s the Gemini summary. And it’s just not useful because it’s not telling me what the human beings in the room were thinking about. Let me throw one other, I’ll be quick. Because, Jim, when you did your closing note, you know, yesterday, which was very good, not going to be too useful for people who were listening to this, but there was a lot of talk about using AI to ramp up junior analysts and build lots of different, use different tools that kind of let them kind of do self study. And I wound up wondering about where do we teach the junior analysts, how to actually relate with people and have the creative, collaborative process. And so there’s something there, too, that makes me nervous that we’re at a conference right now, like are we on some logical trajectory where we think we’re all just going to sit at home and just have AI do everything that needs to happen. There is a very, very real part of this job that is human and communication and collaborative creativity. And I get very nervous that people are not recognizing the value of that and trying to have AI replace it. That was really deep.

00:18:56.02 [Moe Kiss]: I was going to talk about meeting notes.

00:18:58.38 [Tim Wilson]: Go for it. Do I need to move away?

00:19:01.22 [Moe Kiss]: No, but so for me, I actually find the meeting notes summaries incredibly useful. And the biggest thing for me is, like, as someone who self-declared

00:19:10.78 [Tim Wilson]: has ADHD, can you square the meeting notes with the taking notes by hand?

00:19:15.98 [Moe Kiss]: I do. So I do still often also take notes. But the taking notes for me is me, number one, it helps me pay attention in the meeting. And number two, it helps me retain the informations. Like if we’re talking about especially something complex, I won’t necessarily fully hear it. And then the next day, I sometimes do look at my own notes. I won’t necessarily look at the notes. But what I find useful is the next steps. There is always like, no, when you are trying to corral like 20 people,

00:19:47.02 [Michael Helbling]: you can be like, what is the big deal?

00:19:51.42 [Moe Kiss]: What is it? Why is that the big deal?

00:19:53.02 [Val Kroll]: Tim, just fell off the stage for those of you listening. That was a Redd Foxx impersonation for anyone who walks.

00:19:58.82 [Moe Kiss]: It’s like, this person’s going to follow up with this. By then this person’s going to follow up with it.

00:20:02.46 [Tim Wilson]: And because it’s actually fucking terrible at that, it just goes through and says, this is what it was. And you need the human editorial person saying, what are the real next steps? That’s where it’s trying to go through a discussion.

00:20:15.70 [Moe Kiss]: To get back to the question, though, of what makes me feel ick, what makes me feel ick is things being shared that have not been properly vetted and QA and meeting notes fall into that category just as much as analysis or write up does. Like that’s a bit that I get stressed about. Is it analysts are like, yes, I can pump out more stuff. I’m going to automate this report. I’ll send it out. I’m never even going to look at it. And I’m like, oh, that feels uncomfortable.

00:20:41.98 [Tim Wilson]: But isn’t the meeting notes is asking people to do a lot because they’re like, I got to get the meeting notes out promptly. And it takes an enormous amount of diligence to say, I am truly going to read through these and modify and write my own little summary and write. So it’s like, you can paint the picture, but watching what actually happens with the people I’ve worked with, all of a sudden I’m like, oh, this was just barfed out. And I’m sure they scan through it and I’m sure they told themselves, yeah, that seems about right to be fair.

00:21:13.50 [Moe Kiss]: I just take the like the here are the action items. I don’t send the whole summary. Is that worse or better?

00:21:19.86 [Tim Wilson]: I well, if you take them and you say, yeah, that looks about right. I think that’s a problem because I think there is much more often. There is the person who’s sending those out should have a responsibility to say these are the things that really need to happen. There’s a level of prioritization and wording and body language and and adding net new stuff that I know that Joe said that Joe was going to do this. But the reality is I know in our organization that Joe is going to have Mary work with him on this and putting that sort there, there is something in that maybe I’ll get off that.

00:21:56.54 [Moe Kiss]: There’s a lot of nods, so I’m interested to hear more Tim. Normally, I don’t have this kind of live feedback that suggests maybe you’re right.

00:22:02.62 [Michael Helbling]: That they’re agreeing with me.

00:22:05.66 [Michael Helbling]: I think I’d go in a little different direction, which is I get sort of this icky feeling or there’s a threshold. I don’t want to cross with AI in terms of relating to it. And what I mean by that is some of the LLM big companies put out research where they kind of take what the LLM is doing and equate it to emotion and telling you that basically if you behave around your AI a certain way, it may actually impact its performance and that’s sort of what they’re seeing. And I think it’s a really dangerous thing and I think it’s also tricky to talk about because I don’t want to advocate for being mean to your AI or whatever. But as humans, we anthropomorphize things really a lot. And so I think we very easily buy into this idea that I make my AI feel bad if I yell at it or I make it feel good if I tell it does a good job. But in reality, it feels nothing. And most importantly is human to human interactions. I modify them a great deal based on what I know of that person and the empathy and the intuition I’m getting from that conversation.

00:23:13.46 [Tim Wilson]: So if someone’s struggling, I modify empathy and modifying Tim’s going to need a definition of hang in there,

00:23:22.74 [Michael Helbling]: and so you adjust to that person like if you’re giving feedback, for instance, whereas if I’m giving feedback to an AI, I want to be as direct and succinct as possible without having to kind of couch it in a phraseology or terminology that keeps it secure in its own quote unquote emotions, which are not real. And so for me, that’s sort of a weird line that I think I’d love for us to avoid

00:23:48.58 [Michael Helbling]: as we approach AGI.

00:23:51.86 [Moe Kiss]: Can I ask a crowd question? Only if you can figure out how to get a mic to them. No, I was going to make you count the hands for a rough estimate. Well, if you each take one and we take the average of each of your estimates, maybe we’ll have a decent score. OK, creating an agent based on your stakeholder one, stakeholder two, stakeholder three, so that you can tailor your comms and have a persona built out for each of it. Like, I want to know how we feel about the ick factor of like, is that icky or is it thoughtful? Because anyway, I can finish my thoughts afterwards.

00:24:30.66 [Tim Wilson]: Sam Bert, would you like to answer that question? There was a whole session on that.

00:24:35.58 [Michael Helbling]: So here’s my take on that, because I learned about that yesterday in a session and I actually really quite liked it, because I look at AI as a tool to take information and position it in the best possible way for the audience that you’re presenting it to, much like you would present maybe a slide deck to one person and a narrative to another based on how they consume or prefer data or information to be consumed.

00:25:05.82 [Moe Kiss]: Two notes. Just to be clear, I’m not picking on Sam. That was like very persona based. I’m talking about like someone in your team creates one that’s like, this is Moee. This is how she receives information, like very personalized. I think that’s a bit different.

00:25:19.66 [Tim Wilson]: So I think and I think there’s two aspects. So I think Sam’s and yours. One, I think if it actually forces the person who’s creating it, this is where we to actually really think about like to create it, you have to think about what do I need to? What do I know about Moee? What have I seen about Moee? Can I ask Moee something?

00:25:40.18 [Moe Kiss]: So that’s like, but do you think that they would or do you think they would just be like, I’m going to upload 50,000 conversations, a bunch of Zoom transcripts, whatever of interactions with this person and you tell me what you think they’re going to like.

00:25:52.42 [Tim Wilson]: I think that’s going to be less effective.

00:25:54.06 [Moe Kiss]: Yeah, it would probably would be.

00:25:55.66 [Tim Wilson]: And then the second part of that, I completely lost what my other thought was. So on Brent.

00:26:04.74 [Michael Helbling]: Well, a lot of these questions are tailored around a bunch of information. I uploaded about you three. No, I’m just kidding. OK, we have another question coming from someone who’s unfortunately not here. Joe Domoleschi, if you were stripped of your fancies tech stack and could only provide one specific deliverable to a stakeholder to prove the value of marketing analytics, what would it be? Live dashboard, a PDF report, a slide deck, a meeting with actionable insights, etc. What would you do?

00:26:34.66 [Val Kroll]: So one specific deliverable to prove the value of marketing analytics,

00:26:38.46 [Michael Helbling]: just to clarify, that’s what it says.

00:26:41.14 [Michael Helbling]: Yeah, OK.

00:26:44.66 [Val Kroll]: The results of an A.B.

00:26:45.66 [Michael Helbling]: test. So Format can be anything.

00:26:50.42 [Val Kroll]: I mean, well, he said deliverable. OK, so I would I would do a presentation, tight narrative results of an A.B. test. That would be my that’s my no explanation.

00:27:02.30 [Tim Wilson]: I think my not might not be a prove, but might be to convince or define. I would probably go to some sort of compelling story that I was comfortable. I might have pulled data from various sources. I might have run an A.B. test, but I would actually tell a really strong narrative and maybe with slides. Maybe not. I think that would actually be more convincing.

00:27:30.10 [Val Kroll]: Like, why does said deliverable, that’s your constraint.

00:27:33.70 [Moe Kiss]: But see, OK, this is the difference when I heard deliverable. I was like, it can be anything. And it sounds like you’ve both interpreted that in like agency consulting land very different to me, because I would have said if I could pick anything, it would be an MMM. Like, I can talk about that for weeks and months. I mean, at some point, it becomes valid. But it would probably be an MMM. Like, I’d love to go through an experimentation tool, but I feel like that maybe is not in the spirit of the question.

00:28:00.90 [Tim Wilson]: I don’t know. Yeah. What is it? What is a deliverable?

00:28:03.18 [Michael Helbling]: We could get that was not the question.

00:28:06.90 [Tim Wilson]: No, it’s what is the deliverable is. So now I found sound like a consultant. But I mean, on an MMM being super compelling, and I’m sitting like 15 feet from Jim Janolio. So and having heard him talk like you can you can conduct a great MMM. You can you can build it and then you can deliver it horribly and doesn’t show anything or you can communicate it really, really effectively. So it is kind of what is a deliverable and how effectively is it created and delivered?

00:28:35.26 [Moe Kiss]: Or we could just combine all three and then we’d like be winning. If we’re perfect, you didn’t answer yourself.

00:28:43.18 [Michael Helbling]: I’m going to ask the next question.

00:28:46.10 [Michael Helbling]: From a member of our audience, Bryce Preslicka.

00:28:52.14 [Michael Helbling]: Preslicka. Preslicka.

00:28:58.58 [Brice Praslicka]: All right. So I have a client that had some major data issues previously. Once we got on to the project, we’ve cleaned things up. We’re in a much better state now. The problem is one of the clients, POC’s continues to act as though we have unreliable data. And even worse, we have a member of our team that continues to use words like discrepancy and lack of trust and keeps using words that don’t really help us accurately convey that it’s reliable now.

00:29:24.78 [Michael Helbling]: So I’ve sent memos.

00:29:27.38 [Brice Praslicka]: I have tried to coach them to not use certain words, but with both an internal team and a client that don’t trust data that is in much better spot now, how would you go about trying to regain trust?

00:29:40.26 [Val Kroll]: In the role of the client you’re supporting, are they on the business side or are they in an analytics role?

00:29:45.82 [Tim Wilson]: They’re in the business side. Well, now you have to provide an answer because he was just clear. So here, I’ll kick us off.

00:29:54.14 [Michael Helbling]: There’s no time for cash measures. Jeez, we’re in a world of AI now and we need to move quick. First thing first, stop inviting the internal person to the meetings. So they can’t screw you up. Second, take charge of those meetings and tell the client that you know what you’re talking about. And it’s time to make decisions and get off the pot. What are they afraid of? No, I don’t know if I could pull that one off.

00:30:16.50 [Michael Helbling]: But but, you know, start to form the communication

00:30:20.46 [Michael Helbling]: and put it into the positive realm so you get past that moment.

00:30:24.18 [Tim Wilson]: I I mean, this is going to sound easier than it is in practice. But to me, one of the things that AI has not helped we have struggled with for for 25 years is businesses that are looking for certainty and precision when there is actually they’re operating under conditions of uncertainty. And Jen Kunze’s presentation yesterday, like it’s like people think that, oh, the data, the data was never complete. The data was never perfect. It’s really no better, no worse. We can always point to stuff that’s not working. So that like so to me, once you’re having the discussion about is the data right? You’re losing if you’re using discrepancy. If it’s, you know what? Hey, can we reset? Can we really nail down the one or the two or the three biggest decisions you’re trying to make? Don’t worry about the data. Forget about the data. Yeah, it’s going to be involved at some point. I think a lot of times what happens, if you can really get them saying,

00:31:24.70 [Michael Helbling]: what I really want to know is, is meta delivering results?

00:31:29.62 [Tim Wilson]: And they’re like, can’t you keep crunching the data from meta? But I know there are gaps. Instead, you may say, really, let me talk to you about what a geo lift test is. So it kind of goes back to that order taker versus partner. And it’s it’s tough. They’re still working with you. So they have some level of trust. But I think we wind up fighting the fighting on the wrong ground. We were on the ground of like, no, but the data is good enough. Well, no, but this and go, I know it’s an asterisk and don’t use this word. It’s like, instead of like, we so quickly lose sight of it’s such a tangible thing to point to that the data has a problem. And we forget to say, what’s the what do we really want to know and try to elevate that conversation? I often I think it’s like, that’s actually not the right data set for it. Anyway, we keep chasing the wrong data set to most answer that question.

00:32:24.50 [Val Kroll]: I like that a lot. And the one point that you mentioned that I just expand upon is I think the really rooting yourself in like, what level of certainty is really required to answer this question? How much time do I have to turn this around? Is it something you need tomorrow? Do I have a couple of months before you’re going to make this call and really just trying to line up the various different methodologies that are at your disposal to bring that to bear to bring the right evidence to that question? You know, Paula’s presentation, like merging those different sources of evidence to really kind of paint that picture. I think can like shake people loose from like focusing on, you know, what percentage of, you know, people are opting out from whatever cookie banners and things like that, right? So I think like just not trying to play on that, like move the battle, I guess, not play on that turf and kind of say, hey, like let’s just focus on like Tim was saying, like those top those top questions that you’re really grappling with. And let’s think about how much business risk we really be introducing if it wouldn’t be perfect. So like, let’s think about the various ways we can kind of go about it. And sometimes just injecting a little creativity, if you will, into the methodologies, into that conversation can kind of get them excited about some different ways. Maybe it’s a user test. Maybe it’s, you know, it’s not going to be something we’re going to look for in a table as an example.

00:33:38.66 [Tim Wilson]: So be sure to record the meeting and send them the meeting summary. Oh, for fuck’s sake.

00:33:43.70 [Moe Kiss]: I actually stayed very quiet on that one because this is an area I think Tim generally is normally right in. Wait, why is everybody shaking their head now? But I do sometimes and I’m obviously in-house, so it’s quite different. I do find if I have a stakeholder like that, I will almost always have one-on-one time with them. And I’ll normally have some questions around like, what would have to be true for us to use this data source

00:34:10.90 [Michael Helbling]: or, you know, are there other data sources

00:34:14.34 [Moe Kiss]: that we could use to supplement the information so you’d be comfortable enough making a decision with what we have? Like kind of trying to tackle it almost one-on-one, because especially soon as you get in a meeting with a bunch of people and everyone’s like, oh, well, this data’s wrong. So, you know, we’re all stuck here and then everyone whinges about it for the rest of the meeting. It like it stops being productive. And so I would almost be trying to like really partner with that, like the biggest doubter of the group especially

00:34:35.54 [Michael Helbling]: and build up that relationship and really work with them on

00:34:39.54 [Moe Kiss]: some of the methods that Tim and Val are talking about here so that then they can also become your advocate, hopefully, over time.

00:34:46.78 [Michael Helbling]: Excellent. All right. Here’s another question we got. Everyone at my company is being tasked with showing specific efficiency improvements they’ve delivered using AI. I’m an analyst who supports marketing. What are some ideas you have that I could do for that? Don’t make me do the Hollywood Squares one again.

00:35:14.78 [Tim Wilson]: I just feel like this. I’m starting to feel like we’ve beaten this particular horse.

00:35:21.86 [Michael Helbling]: What do you mean, AI or efficiencies?

00:35:24.62 [Tim Wilson]: Well, well, yeah, I mean, the AI piece and the big, I guess the my my qualm with the efficiencies and I totally recognize the person asking the question. They don’t have control over that. That’s being pushed down and it’s an organizational challenge. But efficiency is like producing more with the same or producing more with the less or producing the same with the less whatever. And it’s like more what and and moving down the path of saying, well, we’re we’re producing more dashboards faster. We’re responding to requests faster and everybody feels resource constrained and like we can’t hire we can’t double our headcount. So AI is going to help us keep it fixed. And I think this is where I’m feeling like I’m beating a dead horse and I am the dead horse. I don’t know that that has this idea that if we its volume, volume is the issue that we just need to generate more.

00:36:24.74 [Michael Helbling]: But that volume of whatever we’re producing is going to someone.

00:36:30.38 [Tim Wilson]: Like there’s there’s value in the friction, which does not make me an AI skeptic. I just think the efficiency part is really, really tricky. You know, I think I’ve seen that in articles that that’s happening across a lot of companies are saying, we’re just trying to make this as a actually Jim talking on, I think day one was showing like this is just a chase for headcount reduction and something’s not right there.

00:36:57.26 [Michael Helbling]: So you you managed to answer that without giving this poor person any tips on how to do their job.

00:37:03.30 [Val Kroll]: They should have been a marketing analytics woman. Yeah, that’s right.

00:37:06.06 [Michael Helbling]: There’s a lot of tips there.

00:37:07.18 [Moe Kiss]: Can I jump in, though?

00:37:08.02 [Michael Helbling]: He helps.

00:37:10.26 [Moe Kiss]: I learned this recently and I’m still kind of reconciling it. So I’m going to obviously tell a bunch of people the exact advice I got. And we can all try it out, report back to me. Uh, I got some advice recently, essentially, like sometimes you just suck it up and you do it and a lot of conversations around AI at the moment are about productivity gains, not quality gains. And I find that just it gives me the ick. However, there comes a point where sometimes you’re in a position and you just suck it up and you do it and you go, oh, my God, my team saved five hours a week. Here’s all the dot points, send it up to leadership, move on with your life. And then you get your team together and say, OK, let’s have a conversation about how we improve the quality of our work. That’s what matters here. So sometimes the signal you send up doesn’t have to be the same as the signal you send down, but you better be smart about how you do it and don’t get caught.

00:37:59.30 [Tim Wilson]: But you’re setting yourself up to actually send a better signal up down the road, right? So I love that for doing both.

00:38:04.54 [Moe Kiss]: Yes, obviously, team that was the grand plan.

00:38:06.74 [Tim Wilson]: Check the box, but then separately say, but here’s the real value we got. And yeah, yeah.

00:38:11.74 [Michael Helbling]: And we’ve been battling a problem like this since forever. I mean, there used to be a time in our industry when we thought if we collected every single piece of data, we would somehow magically know more. And it sort of is a redux of a similar way of thinking. And so we have to kind of manage through it effectively. All right, we’ve got another question, and this one comes from also

00:38:34.90 [Michael Helbling]: someone in the audience, Sam Burge.

00:38:39.18 [Tim Wilson]: I think Michael surprised himself and didn’t realize he should have already been on the move.

00:38:42.34 [Michael Helbling]: So we’ll fix that in post.

00:38:48.00 [Sam Burge] How do you think analytics teams will look differently from today with AI?

00:38:54.98 [Tim Wilson]: In the future. I mean, I I hope that on the one hand, they are still spark, curious, business thinking, technical kind of have a broad set of skills. So I don’t think the teams will necessarily look a whole lot different.

00:39:16.54 [Michael Helbling]: How they work, they’re obviously going to get, I guess, efficiencies

00:39:21.90 [Tim Wilson]: and changing ways of working and become more prepared.

00:39:25.98 [Michael Helbling]: But that’s a tough one.

00:39:28.22 [Tim Wilson]: Why did I jump in and answer that?

00:39:29.54 [Michael Helbling]: I don’t think I had a good I was just buying time for one of you guys

00:39:32.38 [Tim Wilson]: to say something smart.

00:39:34.18 [Moe Kiss]: I don’t know about. OK, this is my guesstimate.

00:39:37.66 [Michael Helbling]: Yeah. And this is what I’m observing within my own team.

00:39:43.58 [Moe Kiss]: It seems like folks are kind of splitting a little bit. There are the folks that are going much more technical. I would almost, to some degree, even a little bit more specialized. And then there are the folks that it feels like the chasm between the technical and more the like business facing generalist is getting a little bit wider. I don’t necessarily see that as a bad thing. I think when folks have a particular strength in one direction,

00:40:06.02 [Michael Helbling]: like we should encourage that.

00:40:07.54 [Moe Kiss]: And that’s a great thing. I do think it makes it harder for like teams and how they work together and all of that sort of stuff. I had a massive rant a little earlier today because I am sick of reading very shitty, long documents which were based on someone’s shower thought that never should have left the shower, but now is in a four thousand word document and being flung around our organization. And then someone else comes in and writes 50 comments on it. And then someone’s like, over to you now, Moee. And I’m like, what do you want me to do with this? I am now doing all of the thinking work of having to read it. And it’s pretty like watery garbage, having to like respond, also trying to figure out what you want me to do with this because it was a shower thought bubble. And what my hope of where we get to is that it actually helps us think more critically, not less. I think we’re in the shitty stage right now. I’m optimistic. I’m going to bitch about the shitty stage we’re in right now because it is shit, reading all these documents. But I am optimistic that with time it will help us think better about what we do. Like, I don’t know, you’re creating a hypothesis, like instead of having to tap your co-worker, you can like sense check, have I got all the key components that I need to have a really strong hypothesis for this experiment?

00:41:27.82 [Tim Wilson]: No, I think even like knowledge management, the ability because AI is helping so much with unstructured data, the calling through what has happened in the past. But it feels like it is an elevated role for what a great analyst should be doing five years ago is still the same, which is being deeply embedded in the business context and the business needs. It’s been historically very hard to get the historical what have we done. So I think some of those like the preparation, the pulling this together, using the tools, but I don’t think it should change from what I think great analysts are doing, which is still having a deep connection to the business. It’s a shifting kind of tool set.

00:42:13.26 [Moe Kiss]: Can I just also add, Sam, one of the things I actually loved about your presentation was the idea also that we can change the format very quickly to suit different types of people. Like I am an audio person. I would absolutely listen to a podcast on business metrics.

00:42:28.58 [Tim Wilson]: I just remembered at the second point I was going to make back on that question.

00:42:31.22 [Michael Helbling]: All right, fine.

00:42:32.66 [Moe Kiss]: Was this like from 10 minutes ago or? It was, but it was on that.

00:42:36.02 [Tim Wilson]: It was asking, trying to figure out the best way that’s what somebody would, how to respond to them. So when you said the audio person, how often do people actually know themselves? So for all the listeners or anyone who wasn’t in Sam’s session, it was the, you know, there’s the marketing person who says, I just want to have my cup of coffee and listen to the podcast. And there’s a little trigger in me that thinks sometimes we seldom really know ourselves. So differentiating between somebody who thinks that’s what they would want

00:43:07.38 [Michael Helbling]: and someone who actually that would be effective.

00:43:13.02 [Tim Wilson]: And that had rang true from what we’ve dealt with for a hundred years. We’ve had people saying, I just need a dashboard that does X and we deliver them the exact dashboard. And they’re like, this isn’t helpful. Where’s this other thing? So there’s this other layer that I think analysts historically have needed to follow. And the same thing opening up all these different formats is great. But what somebody says would work. And I think you have to deliver it to them. But figuring out like, does that actually work? And giving them the opening, if they said, oh, I thought that would be really cool. And you tweaked and tuned the tone and the content and everything, but giving them the out to say, you know what that actually didn’t work. I don’t know that I wouldn’t have known it wasn’t gonna work until I actually tried it for a while, which we have not done in the industry very well forever. We get in sort of a whiny mode of saying, we’ve given them all these dashboards and they’re not using them. And it becomes this adversarial thing because, and then if we ask them, don’t you want these dashboards? Well, they ask for them. Of course they’re gonna say, yeah, yeah, yeah, this is really useful. We haven’t figured out how to say, no, that mechanism didn’t work and it’s okay. And we need to have the trust and we need to try something different. So we can go back and that was my other point. Maybe we should let, I’ll get a word in and twice.

00:44:35.90 [Val Kroll]: Back to how do we think teams will change? I think that there’s gonna be some new muscles that are built. I think one of the things that we had been talking about this conference is how this has given us a lot of energy and excitement. And I think that there’s been some creativity injected, which I think is just fun. Even if we’re just talking about little things that we do on the side that’s not ready for prod, but it’s just kind of like stretching us in some new ways, which I really appreciate. The other thing that we were also talking with you about, Sam or you and I were chatting about,

00:45:03.38 [Michael Helbling]: is the communication skills

00:45:06.74 [Val Kroll]: and how that’s gonna be improving. Because I think how often have you been in a conversation, even over the past couple of days, perhaps where people are just ragging on their stakeholders, like, oh, they’re so dumb, they just don’t get it, right? And it’s like, when you’re prompting and you’re talking about something you need the AI to do for you and it totally misses the boat, and you’re like, oh, geez, I’ve totally forgot to give you this piece of context. Of course you didn’t understand what I was trying to say. Think about your poor stakeholder that you were not giving that context for how many years. The video of, we were talking about this, the dad with his son and daughter about making the peanut butter and jelly sandwich, about like, take the bread out of the bag and he’s like trying to, and he like ends up like putting the knife through the whole loaf of bread, whatever, because he was just trying to follow the directions. But I think it can help us build a little empathy for our stakeholders, because like, they’re not wired the same way we are, they don’t have the same background that we do. And so I think it’s one of the byproducts that will help us become better communicators and hopefully have a little bit more empathy for people who don’t make all the immediate connections

00:46:06.82 [Michael Helbling]: that our brains do just because we’re nerds.

00:46:10.50 [Michael Helbling]: I think organizationally, I think we’ll see departments flatten out a little bit. Like over the last 15 years, we’ve become super specialized in a lot of different disciplines, because analytics is actually multidisciplinary. And I think AI will push that back together a little in a lot of organizations. And then those organizations over time will start to realize there’s still a need for some of that specialization on the fringes, and they’ll find ways to bring it back in. But I think we’ll all express experience, some compression where an analyst will go back to being able to code something and also write a data pipeline and also go access the data lake and also build a dashboard and a great visualization. And those were all things that like analytics people were attempting to do 15, 17 years ago. And then we realized we needed specialization. We needed a data engineer. We needed analytics engineer. We needed a data visualization expert. And I think we’ll begin to flatten those out with AI.

00:47:13.50 [Moe Kiss]: That doesn’t worry me a bit though.

00:47:15.54 [Michael Helbling]: I don’t say it’s good or bad. I just think that’s what will happen.

00:47:18.94 [Moe Kiss]: Like I feel like sometimes folks are over indexing on the generalization at the moment and thinking that, I don’t know, a product manager can do a data scientist job, and I mean, some product managers is doing engineering jobs. Also interesting choices. So I think we’re over indexing on the fact that folks can generalize. And I heard there’s a like-

00:47:39.06 [Michael Helbling]: Outcomes follow expertise, even with AI. Okay, we have time for one more question. And that’s our last question. And we have someone who is an audience member who’s going to ask it and it’s Jim Stern.

00:47:52.54 [Michael Helbling]: My question is, what are the first three skills

00:47:58.46 [Jim Sterne]: that analysts have mastered that are going to be successfully taken over by artificial intelligence?

00:48:08.18 [Val Kroll]: I wish we could get a question about AI. No shade, Jim. Just looking at Michael. First three skills, say it again. First three skills. Sorry, I was being an asshole. First three skills.

00:48:24.66 [Michael Helbling]: What are Tim’s favorite skills? So, probably like the, what is it? The ink to data ratio. So that probably going to be the first thing AI takes over.

00:48:37.94 [Val Kroll]: Data pixel ratio. Data pixel ratio.

00:48:39.98 [Michael Helbling]: I was paying attention, Tim. SQL, I don’t write SQL anymore.

00:48:46.82 [Tim Wilson]: I am so much more on the debugging SQL debugging R. I think debugging coming over really, really quickly. I think a second one would be QA’ing or validating or vetting the results of an analysis, having that the thing that you’re supposed to go to another analyst or try to come at it a separate way. I’m not sure what the third one is. I think it’s a lot of things that are going to be supplemental that we should be doing. Like not doing the QA, but giving me the list of, check my logic. Check the things that I, so maybe it’s not what the junior analyst is doing. I think it’s what the junior analyst ideally is working with another junior analyst or senior analyst to look over and review. There’s not a whole lot that I see a whole hard, whole hog hand in the keys over on.

00:49:42.94 [Michael Helbling]: I came up with three, but I don’t know if they fit the criteria. We’ll have to go from there, but that’s going to be where we have to wrap up. And I want to say first, a huge thank you. To Jim Stern for organizing the marketing analytics summit. 45 years. And save your applause because also to all of you for being here and bringing your energy and your questions, your insights and experiences in a world changing daily with AI, it’s the people, the community and human connection in our industry. It feels all that much more special and crucial in these changing times. And obviously there’s a huge audience also listening and we’d love to hear from you too. And you can reach us at our LinkedIn page or the measure slack chat group or by email at contact at analyticshour.io. Please leave comments, ratings and reviews on whatever platform you use to listen. We do read all of them. And Tim brings them up in meetings.

00:50:48.46 [Michael Helbling]: And I think-

00:50:50.70 [Moe Kiss]: It makes us set KPIs.

00:50:52.62 [Michael Helbling]: Yeah, it’s terrible.

00:50:55.18 [Michael Helbling]: I can’t wait till AI replaces that. All right, and I know that I speak for all of my co-hosts, Val, Tim, Moe. When I say, no matter the question or challenge

00:51:06.06 [Michael Helbling]: you’re currently solving, keep analyzing.

00:51:09.30 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at analyticshour, on the web at analyticshour.io, our LinkedIn group and the measure chat slack group. Music for the podcast by Josh Crowhurst.

00:51:26.98 [Charles Barkley]: Those smart guys want to fit in. So they made up a term called analytics. Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition.

00:51:46.18 [Jim Sterne]: Ladies and gentlemen, that is so much fun. Thank you so much for be adding the spark to the end of the marketing analytics summit. You guys are awesome.

00:52:02.62 [Tim Wilson]: Rock flag and AI gives me the ick.

00:52:06.42 [Jenn Kunz]: Yeah.

00:52:07.26 [Michael Helbling]: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah.