[music]

0:00:05.8 Announcer: Welcome to the Analytics Power Hour, analytics topics covered conversationally and sometimes with explicit language.

0:00:05.8 Michael Helbling: Hi everybody. Welcome. It’s the Analytics Power Hour. This is episode 245. And welcome, welcome. The Analytics Power Hour, it’s an advice show for the modern data professional. This is gonna be fun. We asked all of you to send in your questions and send them in you did. So on this episode, I’m joined by all of my co-hosts to answer some of these questions. I’ve been told that this show in particular, we will be taking very seriously, so none of the traditional banter and humor will be allowed. So, in no particular order, let me introduce myself. I’m Michael Helbling. I also have co-hosts, the awesome and talented Julie Hoyer. Welcome, Julie.

[laughter]

0:00:58.2 Julie Hoyer: Hello.

0:01:00.0 MH: Awesome.

0:01:00.6 JH: How are you? [laughter]

0:01:00.9 MH: Yep. Remember.

[laughter]

0:01:01.0 Val Kroll: We’re not even two minutes in.

[laughter]

0:01:02.5 MH: Yeah.

0:01:02.8 Moe Kiss: Oh, wow.

[laughter]

0:01:06.3 MH: But I think the laugh track is a little premature, but let me introduce the quintessential analyst, Tim Wilson. Welcome Tim.

0:01:18.6 Tim Wilson: Alright. Happy to be here.

[laughter]

0:01:20.4 MH: At this point we’re like a traveling circus. Alright. The wise Moe Kiss, welcome Moe.

0:01:29.6 MK: Hi folks.

0:01:30.9 MH: And of course, the convivial and clever, Val Kroll. Welcome Val.

0:01:36.0 VK: Did ChatGPT write this intro, Michael?

[laughter]

0:01:39.0 MH: No, I actually did Google synonyms for friendly and smart to get the alliteration. Yes.

0:01:47.8 VK: I like it.

0:01:49.2 MH: Alright. Well, okay, supposedly we all know the format, but let’s just review. We have listener recorded questions. We’ll play one and then we’ll all make fun of Tim’s answer for a few minutes. We will all answer and figure out. We’ll have some thoughts on some of the questions. So let’s see where this goes. Before we start, we all want to give a huge thank you and shout out to all the listeners who submitted questions. We couldn’t possibly include them all, but we deeply appreciate all of them nonetheless. Let’s do this. Let’s get to our first question.

0:02:21.2 Lori: Hi, my name is Lori and I’m calling from St. Louis, Missouri. My question for you is, how do you know when an analysis is good enough? I work for an agency so it’s very fast paced and often the time that I can spend on any given analysis depends on what else I’ve got in my queue, but I always, always feel like it needs just a little more time. I think the most influential piece in my thinking about this is Tim’s blog post on assumptions and just the idea of deciding when you have turned enough data stones and when it’s time to stop turning them over. I really like the idea of calling those out as limitations, but I still, I’m wondering if you have any advice for determining when you have turned enough data stones. I think that sometimes I operate out of fear that someone’s gonna poke holes in my analysis and that’s really not the best way to be creative and to think about things. And so yeah, I’m just wondering if y’all have any advice. Thank you so much.

0:03:14.8 VK: So to be clear, Lori read Tim’s content and walked away with more questions than answers questions.

[laughter]

0:03:24.0 MH: Well, in an agency environment, the first thing you do is you work on it until you’re out of billable hours for that particular task. No, I’m just kidding. [laughter] I’m joking. I’m joking. That’s a joke. Please do not do that. That’s not a recommendation. Alright, Tim, what’s the answer?

0:03:40.0 JH: But she did say the word…

0:03:42.2 MH: Oh, go ahead, Julie. Sorry.

0:03:43.3 JH: I was gonna say she did say the one word that came to mind for me first, which was assumptions, and my kind of thing is like once you start to build up too many assumptions, you’re going down a path of an analysis, and if you’re making assumptions for your stakeholder, to me, that’s my sign of like, I need to pause and stop. Because even though you may think, oh, I feel like maybe I don’t have enough to take to them of a story, depending on how open-ended or vague this question or request was, unfortunately we fall into that trap a lot. Yeah, if I have to start making assumptions for my stakeholder and what they want to determine which direction or how much further I go, that’s usually when I say I need to probably pause where I’m at and check in with them.

0:04:22.6 MK: See, I don’t know. I feel like maybe I have a different approach and it might be very internal, I suppose, specific. But part of it is like, at what point do we have enough information to make a decision? If we had more data, would it possibly change the decision we’re gonna make? And so for me, a lot of it is the timeliness and the impact or size of the decision that we have to make. So like, oh, Tim’s screwing his face off already. But I don’t know if that’s a… Because I’m in client side, it’s very different. You’re close to your stakeholders and sometimes you’re like, okay, I know that they’re gonna make a call on this by 10:00 AM Wednesday. So I’ve got to do what I can by then, and potentially, if it was a huge decision, I would push back if we didn’t have enough information. But generally speaking, it’s the stakeholder reality that caused me to stop or start. Alright, everybody pack on.

0:05:22.5 TW: No, no, no. When I was listening to that, to me what I was hearing, the word that popped out to me was fear. And I think I definitely identify with the fear because, say that it’s like, oh, they have to make a decision Wednesday morning, so I need to give them enough by Tuesday afternoon. And the fear is that you say, “I think I’ve given you enough.” And they say, “Oh, but what about,” and that was a stone that you didn’t turn over. And that’s kind of the balancing act. But I think Julie, your point, I was thinking like, when you start to make those assumptions, that’s the opportunity to reach back out and say, “Hey, here’s what I’ve done.” Which I think, Moe, your point is great that it’s a lot easier if it’s an internal Slack or you can reach out to him and say, “Hey, I’m working on this.” One, you’re just letting them know you’re working on it, but also say, “I could go down this path. It’s going to take me some time. It might mean tomorrow instead of this afternoon.” And that is a little tougher in an agency. But I love that she’s like, I don’t think it’s right to be operating out of fear. That’s a nerve wracking way to go.

0:06:34.6 VK: Julie, I’m reminded this is probably the third episode where I’m bringing this up, but the approach document that you worked on with Ryan and Sam to outline what are some of the risks that we already foresee, getting clarity upfront about what data sources we’re using, what time period, and just so that you’re really starting from that foundation of understanding and opening those doors of communication for what you just called out, Tim. But yeah, again, and that was helping it frame as a hypothesis, ’cause that’s also helps it get really clear as to like, how do we know we have enough to help validate the question that was at hand? But yeah, I’m just gonna keep mentioning that ’cause that was just such a beautiful document you put together.

[laughter]

0:07:17.1 MH: Nice.

0:07:18.5 MK: Can we rewind though on that fear component? Because now that Tim’s brought it up, I can’t un-think about it. Is this something that all analysts face? I’m just, I’m thinking of a very specific situation where I did a piece of analysis. There was a pretty tight timeframe. It wasn’t impossible, but it was pretty tight and I had to make an assumption at one point of a particular rate. It was like adoption rate or something like that. And at my last company, our leadership were all ex-consulting. So they went through every single line. It would be like, why’d you make this assumption, this one? And at some point I had to be like, this is what it was on other devices. And I gave my reasoning, and he was like, “Well, I don’t agree with it and here are all the reasons why.” And I’m like, “Okay. Well, we can change it.”

0:08:07.2 MK: But I remember having this really visceral reaction to it of like, oh, I’ve really fucked up here. And is part of this evolution as an analyst just realizing you have to make assumptions or you have to sometimes do something at 80% and it’s not actually, I think the fear is very natural, but it’s about learning ways for you to be comfortable with it. Because guess what, we can’t do everything a 100% of the… We can’t do things perfect every time. It’s just not possible and we have to make assumptions, otherwise we can’t take a step forward. So, is part of just your development, learning ways that you can get comfortable in that fear?

0:08:51.7 TW: Well, but you also hit on the being able to, “Well, I don’t agree with the way that you decided to do that.” And you can say, “Well, I gave my rationale. I’m a human. I made my judgment call. Now, would you like me to go back and redo it?” ‘Cause I think that’s another little hook, is, and maybe I was catching this as she was speaking as well, wait, we have this tendency to want to say, I’ve done it. I’ve handed it off. And if it’s not everything that they need, then I have failed. And definitely, and I think in any in-house or external, there is a tendency to say, I want to check this off my list and be done. You guys have lived with… I keep wanting to get stuff done, whether it’s… And then you’re like, oh, but one more thing. And it’s like, ugh.

0:09:36.2 TW: So I think that’s the other piece is to, you can turn over more stones later. If you’re going in with the mindset that given the time, the scale of the decision, the number of assumptions I was making, this felt like a good stopping point. If I go in saying, if I need to do more, I failed, then you’re gonna fail and that fear’s gonna perpetuate. If you go in saying, this is my best shot and this is my best spot to check in. I might be done and that’s great, but I might not be done, but I’ll have more information. I’ll know which stones, which additional stones I’m overturning, and hopefully you’re helping your partner realize that this is a collaboration. This isn’t throw it back and forth over the fence.

0:10:22.3 JH: I think follow ups are inevitable, is one thing. And like, there’s always going to be something new coming up where they probably have a tweak on their question or a new meeting the next week. So it’s kind of like, to your point, you have to be comfortable that it’s going to be an ongoing conversation. So it’s not a failure that they’re not like, oh, this is exactly what I needed and I’m done with this conversation. Talk to you next month. Yeah, it’s probably not gonna happen.

0:10:43.1 MH: Business decision finalized. I think another thing I’ve done and I like doing personally is I like grabbing somebody else with context for that analysis and showing them what I’ve done and see if they can find anything to gut check it. So here’s the business question I’m trying to answer or the hypothesis and here’s what I’ve done so far. What holes could you poke in this? And just spending 15 minutes with somebody else sometimes really helps build your confidence to then say, okay, yeah, this is in great shape. We’ve looked at it, or it’s like, ooh, I didn’t think of that. Okay, I’m gonna quickly do some additional work around that piece of it. So sometimes that can also be like, yeah, a second pair of eyes could be helpful.

0:11:23.5 JH: I like that. Yeah.

0:11:25.9 VK: That’s a good one.

0:11:26.3 MH: Because when you’re working alone, sometimes I do this anyways, you just really get like stuck in your own way of thinking about it and then you forget like, oh yeah, we could be this whole other low way of looking at it that someone else can bring, so.

0:11:39.1 VK: Great question, Lori.

0:11:41.1 MH: Yeah, super great question.

0:11:42.9 TW: Yeah. Starting strong.

0:11:44.1 MH: I wish we knew how to answer it ’cause it was like, oh yeah.

[laughter]

0:11:51.9 MH: It’s time to step away from the show for a quick word about Piwik PRO. Tim, tell us about it.

0:11:56.2 TW: Well, Piwik PRO has really exploded in popularity and keeps adding new functionality.

0:12:03.3 MH: They sure have. They’ve got an easy to use interface, a full set of features with capabilities like customer reports, enhanced e-commerce tracking and a customer data platform.

0:12:14.2 TW: We love running Piwik PRO’s free plan on the podcast website, but they also have a paid plan that adds scale and some additional features.

0:12:21.6 MH: Yeah, head over to piwik.pro and check them out for yourself. You can get started with their free plan. That’s piwik.pro. And now let’s get back to the show. Let’s get to our next question.

0:12:35.4 Cade: Hello, this is Cade. I am a lowly analyst who has worked with several of the hosts on the Analyst Power Hour. My question is related towards dashboarding. I have built mini dashboards during my time as an analyst and have gotten advice even from some of the hosts here as well on some of my dashboards. Previously, there’s been an episode that has discussed dashboarding, and my question is, how do we get clients to care about their data rather than just saying, here, give me this report, this is what I want because my stakeholders need it? How can we help them build something that is functionally usable that they can use continually in their organization? I feel like it’s really easy to get siloed where I am told to build something, I build it and then never hear back from them again.

0:13:29.1 VK: So to be clear, Tim, you’ve helped Cade with dashboarding and he still has questions?

[laughter]

0:13:38.6 VK: I’m noticing a theme.

0:13:39.0 MH: Wow.

0:13:42.2 TW: This is good. The what’s going to happen is Michael will never be able to use that dastardly Q word ever again because this will be the one that removes that adjective.

[laughter]

0:13:54.8 MH: This is a good question.

0:13:55.9 VK: It’s another good one.

0:13:55.9 TW: To be fair, and Cade, we’ve never… I’m also a lowly analyst. We’ve never had a chance to work together, but dashboards are basically things to be thrown into a void. They serve no real business purpose. So, just make as many as you can and as pretty, you have to make them really pretty and then it doesn’t matter who ever looks at them. It’s just, could they make it into a business development pitch at a later point is the real value they bring.

0:14:30.1 MK: Oh my gosh.

0:14:30.2 MH: Okay. Who else has a real answer to this question?

0:14:31.9 VK: Michael’s on fire. [laughter]

0:14:33.3 MH: I just woke up and chose violence. I don’t know.

0:14:37.0 TW: Actually, Moe, what is the role of dashboards within Canva?

0:14:44.4 VK: Ooh, good question.

[laughter]

0:14:46.4 MK: I find this question quite challenging because I still feel like dashboards are the bane of my existence, and it doesn’t matter how much I read, what things I test out, it still just feels like the most frustrating thing ever. And the reason I say that is because there is the Eric Weber school of thought of like, you don’t build for a person, you build for stakeholders and the types of questions they would wanna answer, but genuinely, sometimes you need to build something very specific and bespoke just to shut a person up and be like, here is the exact thing you asked for. Please don’t tell me that my team don’t deliver the stuff that you want because you literally put a set of requirements out and we delivered it for you. And then there is the bringing the horse to water or whatever. Like you build it and then they don’t look at it and then you’re really frustrated. Can anyone tell that I’m having a bit of drama with dashboarding at moments just based off my pure tone of voice?

0:15:51.9 MH: It’s not coming through at all, Moe. That’s good. You’re really balancing it out.

0:15:56.3 MK: And then there is the like, what if this dashboard is cut this way and that dashboard’s cut that way and then people get confused because they’re like, this number doesn’t match that number. And you’re like, because you asked for very bespoke things and therefore they are cut different ways. So the summary is, I need everyone else’s help on this one because I am feeling pretty similar to our lovely listener.

0:16:18.9 TW: I am teed up to swing at this one, and this was funny ’cause I actually emailed Cade back when he submitted it ’cause I was like, I literally have a post going live tomorrow called Dashboards Must Die, except, long live the Performance Measurement Dashboard. So I think, and this came from when I was at Adobe Summit and somewhere dashboards came up. I was talking to an analyst and and it got the immediate, yep, dashboards are where data goes to die. So I think that dashboards, and I absolutely blame BI vendors and then analysts and companies buying into their pitch for it. The dashboards get treated as this grand dated democratization thing that, oh, we’re gonna build this for you and you will have access. Actually, when I started search discovery, there was a really good analyst working with a really good client and the monthly report was done by building a Data Studio dashboard for the monthly report.

0:17:19.9 TW: And he had to spend a lot of time ’cause it was like the dashboard had to have a narrative and it was a monthly report. And it was just the investment to build dashboards take time to build them well. And if you then say, I’m gonna build it and make it as my interface to all of the data so that then the business user can answer every question, it will fail horribly. So I think we, I have a very strong opinions that I think dashboards are fantastic when you have, they’re just being used to tightly show the how something is performing. If I want to be able to monitor my campaign and I have clearly defined KPIs with targets and I want a tight single screen, Stephen Few’s Information Dashboard Design says dashboards fit on one screen, and the number of platforms out there that say yeah, it’s one screen ’cause you have a scroll bar, or it’s one screen ’cause you have 18 tabs across the top.

0:18:21.8 JH: And the scroll. [laughter]

0:18:23.8 TW: And you can put all these filters and you can do drill downs. I’m like, that’s not a dashboard. That’s teaching them to use an Ad hoc analysis tool that you’ve put some constraints around it. So for me it’s, let’s don’t expect that if we give the magic, if we give a dashboard with enough features on it, that now the business will go away and they’ll be able to self-serve and answer their questions. And if they’re not looking at it, it’s their fault or it’s the dashboard’s fault. It’s like, it’s a misaligned expectation. So, that’s where if they need to be able to self-service on the data, figure out a way for them to self-service on the data with whatever platform they’re using, if there is something that is, the performance is being measured, and they’ve done the work to identify clear goals and how they’re measuring progress against those goals, and they’ve set targets, then build a dashboard, but keep it tight. I mean, there can be a couple little filters in there. So, okay.

0:19:18.0 MK: Okay. I have a follow up question.

0:19:21.2 TW: Nope, we’re out of time. So I’ll just go ahead and play the next question while I climb down from my soapbox.

0:19:29.4 MK: I agree with your soapbox speech, so that is an interesting place to start from.

0:19:35.0 TW: What? Do you agree with me?

0:19:37.1 MK: Yes.

0:19:37.3 TW: Okay.

0:19:39.0 MH: We’re all shocked, frankly.

0:19:41.8 JH: Hashtag quintessential.

0:19:46.7 MK: I obviously am in this painful place right now. What is then… Because, like I said, I do agree. Let’s have that as the starting point. What it is, then, in your view, the best way to communicate the like, are we on track? Or the like, I’m going to say the I word, insights coming from the dashboard? In your dream state, what does that look like? Because someone has to interpret the data and share it.

0:20:16.8 TW: I don’t think insights ever come from that.

0:20:18.1 MH: Yeah, I’m with Tim on this.

0:20:18.2 TW: I think insights rarely come from dashboards. I think that’s…

0:20:21.8 JH: Preach.

0:20:23.2 MH: Yeah, as someone who is forced to try to generate insights from dashboards and hated my life…

0:20:29.4 MK: Okay, sorry. Maybe I used the wrong language. We want to know every month if we’re on track. So our dashboard will tell us that. Like generally speaking, you have your target. It goes, we’re hitting it. We’re not hitting it. I’m not necessarily expecting the dashboard to answer the why, but when you share, we are on track, yes or no, the next question will be why? So how do you best communicate the why?

0:20:53.8 TW: Okay. If you’re on track and the answer is yes, we’re on track, and they say, why, then the response is, why do you fucking care? You had a plan…

0:21:02.9 JH: No, it’s what did you do…

0:21:03.5 TW: You had a target, you hit it. Why? The methods you’re going through…

0:21:10.2 MK: Fine. What worked? What are the extra opportunities say to hit that target even better?

0:21:15.9 TW: Yeah, I fundamentally think you’re fucked if you’re down that path, and I think most companies are. So, yeah, I think that’s a broken conversation.

0:21:23.6 MH: Well, I think there needs to be a separation between the dashboard itself and those questions, because I think those questions are great. Like, hey, we’re crushing it. How do we crush it harder? Is a great question. That’s not a question to answer in the context of a dashboard. It’s a question to answer in the context of an analysis or something like that, where we go…

0:21:44.0 MK: Okay. But how do you share that? Do you send out ad hoc analysis whenever you like do a deep dive? Do you have a cadence for it? Like, if that is what you’re trying to share back with the business, what does good look like there?

0:22:00.0 MH: This is a can of worms, because I’m recently big fan of statistical process control because of the episode we did with Cedric Chin. So the first thing… Oh, I’m riling Tim up so bad right now, I can tell. All right, Tim, go ahead. You’re smarter than me anyways.

0:22:20.4 TW: Well, no, I actually challenged that. I mean, this is the biggest. Oh, and, Michael, you gave me, before we were recording, you gave me shit saying, like, and this is why you started your own company.

0:22:31.1 MH: No, no, listen. On this, I actually have… We agree. We just don’t ever say it the same way. But we actually do agree on this.

0:22:41.6 TW: I think things are getting thrown to the analyst. I mean, use that simplistic example of, here are the results. Just say we’re not crushing it. Say that we’re missing. And then if the response is, oh, my dashboard showed me that we’re missing it. That’s a problem. Great. You’ve actually cleared one good hurdle, objectively and quantitatively shown, even on dashboard, you’re missing it. And then if the business says, okay, analyst, it’s your responsibility and your responsibility alone to tell me why. I’ve now asked that question. Come back to me when you can answer it. That is where analysts are absolutely destined to fail in the long run.

0:23:23.5 MH: Yeah, but the problem, Tim, is that you don’t get very far being a Bartleby, the Scrivener around that, and just being like, I’d prefer not to, because…

0:23:32.6 TW: No, that’s not…

0:23:34.2 MH: That’s a literary thing. I did a literary thing.

0:23:38.6 TW: I mean, I think the business needs to have some… I want to talk with them.

0:23:45.1 MH: I don’t disagree, but the path is convoluted, is all I’m saying.

0:23:51.3 VK: Tim’s not saying to say no.

0:23:51.8 MH: I’m not saying no at all.

0:23:51.9 VK: Tim’s saying, let’s have a discussion. Because usually it’s the business who is the one that’s in charge of the decision that got us to the place that we are at.

0:24:01.2 MH: Right. Yeah, agreed.

0:24:01.3 VK: So, they probably have a lot of insight into why they’ve made the choices they did over that past whatever time period.

0:24:07.1 JH: Yeah, and instead of having to look at the data as an analyst and be like, oh, it looks like you did X, Y, Z. It’s like, can you foren into X, Y, Z?

0:24:10.6 VK: Yeah, forensics.

0:24:10.7 JH: Yeah, like, I don’t want to go discover what you did six weeks ago by the time I’m giving you my monthly report.

0:24:15.1 MH: ‘Cause then we’d come back with the analysis and be like, it looks like you did X, Y, Z. Like, yeah, we know. We did X, Y, Z. It’s like, or maybe you just should have told me that before I spent the time.

0:24:24.3 JH: Well, analysis says, don’t do that.

0:24:28.0 VK: Next question.

0:24:30.2 MK: Okay. So, hypothetically, we’re not asking the analyst to say why, or the analyst will potentially include some type of write up which has heavy business consultation. And you definitely see companies that do this well, where, like product managers or product marketing managers or marketers are getting involved in helping answer the why. The question still stands. How do you best communicate that then to leadership?

0:24:53.5 TW: I think it’s very organization and situation specific. I think trying to make it a… And this is not me injecting a bunch of like, well, now this becomes super complicated in a production. I think the flip side is when you prescribe that it’s this way, then you shove everything into it. Even the people who put insights on dashboards, and, I mean, there are listeners who are doing that, and, oh, my God. Like, let me guess. You wind up with insights saying that we missed our target last month, or you have insights that are just noting that a spike happened on a chart, or you actually have something interesting to find. But since you only have a two inch by two inch square to put the insights, you can’t elaborate on it So, so many things wrong with that.

0:25:38.8 TW: So I think there are times, because there’s even a why? Like, the natural question of why is a natural human curious. Given a magic wand where it takes zero resources and zero time, I always want to ask why, and somebody immediately gives me the answer. But there’s another level of saying, of having that back and forth, and then it’s like, well, when could you change? What could you change? How much does it matter? Oh, so I could do something in 20 minutes and shoot you an email, or, hey, we have a status call next week. By the time of that call, you know what? I’ll have two slides put together that show it to you. I’ll email you beforehand. I’ll have the charts after. So to me it’s a much healthier way. It’s like we don’t ask when we’re going to answer the phone. We answer the phone when the phone rings and we’re available, like two factors, and we know who’s calling maybe. So, okay.

0:26:38.7 MK: I do agree to some level. What I would say, though…

0:26:41.9 TW: It’s fading.

0:26:42.5 MK: No, what I would say is like, it sounds like, and I could be mis-paraphrasing, that you share when there’s information to be shared and you ongoing kind of negotiate what the right format for that is. The issue is, when you’re dealing with leadership, they have set cadences, they have calendars that are hard to get into. Sometimes you have a meeting once a month, and that’s your only chance. They have an expectation often that there is a regular cadence. And especially in the early days with a new stakeholder, you kind of can’t say to them, hey, I’ll flag with you when there’s something you need to know. I built you a dashboard just like… Do you know what I mean?

0:27:26.0 TW: I will absolutely turn that around. So the regular cadence, there is a dashboard that says, here you’re doing… Because you worked with me to identify what good looks like, that you absolutely 1000% that is structured, rigid in whatever cadence. Maybe they’re even available on demand, but still in your regular cadence, you pull it up and show it to them. But the difference is they say, oh, but now you need to tell me all the insights about that. If you flip that around and just say no in that meeting, we’re having a forward looking discussion about what are you going to do? What questions do you have about it? What are your concerns? What are the risks that you see? What are the things that you hope to actually know and understand by the next time we meet? You’re engaging them and saying, what do you want and need to know? And then you’re actually answering questions that they are ready to, and that can still be on a regular cadence. I think it is 1000% broken. And also, what happens in 90% of organizations, to say no, you show up at the weekly report or the monthly, whatever your cadence is, and you’re supposed to just deliver magic.

0:28:33.8 TW: And, I mean, it even sounds like what you’re saying, like, yeah, you know what? I haven’t figured out how to make that work. Yeah, because it’s not gonna work. I used to do this with agency when I was in an agency as an analyst within a digital agency. And you can make that pivot. It’s rough for the first month. I mean, the complaint that would get was, you find some stuff, sure, it’s low hanging fruit. You found some things, you tell them to do it. Well, then they don’t do it for various reasons, because it’s not actually practical. They don’t get around to it. They don’t follow through. The next month, you can’t say the same thing again. So you say something else. I had a…

0:29:06.0 MK: I think we’re saying the same thing, though. You would use that regular cadence to understand what questions they want answered the next time.

0:29:18.4 TW: And I would also then be able to answer the questions that they asked the last time. Like, if that first meetings are tough time…

0:29:24.8 MK: But you’re not showing up saying, I’m gonna deliver you insights. It’s like, we have a good relationship and an understanding of…

0:29:29.8 JH: It’s a facilitation tool, is the dashboard. Like, how do you get people to use it? You facilitate a conversation with it, I think is kind of what we’ve all gotten to.

0:29:38.4 TW: Your takeaways are, these are the questions, here’s how we’re going to answer them. And then the next time you say, as you remember, these were the questions. We could partially answer this one. We couldn’t answer this one. We have a test plan to answer this one. Here’s the answer to this one. We’re having another discussion. We’re also tanking on this. Why do you think it is? We think it might be this. But you’re setting up your plan going forward from the last one.

0:30:00.9 MK: Yes.

0:30:02.7 JH: And I think dashboards aren’t for business stakeholders. They’re for analysts in general. I feel like business stakeholders ask for them, but if they ask for them, they think they’re going to go look at data and actually analyze it and get an answer. And if I throw it back to Kathleen when she was on the show, providing a number is not the same as analytics. Analysis is a verb of you actually inferring from the data an answer, so to throw that out there, too. I think it’s for analysts.

0:30:29.6 TW: But I think it’s fine for them to look at and say, is my campaign on track or is my website delivering what it’s supposed to do? And even, can I slice it and look at it for mobile versus desktop? Or whatever my little bit of… But also being clear with them saying, if there’s a question you have and you can’t answer it, that doesn’t need to be add another feature to the dashboard. That is where… And the BI tools, they are absolutely out there. Some of the digital analytics platforms say, yeah, you teach them how to drag dimensions in. I won’t name the…

0:31:00.8 MH: But if you do get to the end of building a dashboard and you have a little extra square on the page, yeah, definitely. Just drop in a pie chart, something nifty. All right.

0:31:10.6 VK: Well, one other quick thing that Cade mentioned, just, we can touch on it quickly, that he said that gave me a ping in my stomach, which was, I’m told to build these dashboards, which makes me think that he’s a little disconnected, potentially, in this scenario, from the stakeholder he’s delivering it to. And I’ll tell you what, there’s nothing you could put on that page that’s going to turn out to a successful situation, because it’s going to be through conversation and some of the facilitation you were talking about, Julie, moderating that discussion to make sure that whatever makes it into that is worth it and valuable. So, get yourself in front of those stakeholders, Cade, and get some sharp elbows.

0:31:48.3 MH: And thank you, Cade, for giving us a question that none of us have any problems with or any historical demons about. So, great job.

0:31:57.6 VK: Demons. Unpack your trauma with the Power Hour crew.

0:32:00.9 MH: That’s right. Yeah, we’re on pace to get through four questions at this point, so, I mean, we’re really doing it. All right, let’s do our next question.

0:32:12.3 Rusty Rahmer: Hey, Analytics Power Hour crew. This is Rusty Rahmer. I have a question for you. I think at some point, everybody wrestles with the career decision to work for an agency, a consulting firm or the brand/client, etcetera. While your examples and opinions of the various topics you’ve covered on the show often reflect those difference experiences that each of you have had, I don’t know that I have ever heard an episode where the differences of working in those different positions has been explicitly compared and contrasted. So, what’s it like working at an agency versus a consulting firm versus the client? I would love to hear your collective experiences on the topic and even perhaps the differences in your experience with small, mid and large sized versions of those different business types.

0:32:56.7 MH: I’m not going first this time.

0:33:00.0 VK: That was a great question from Rusty.

0:33:03.4 MH: Yeah, it was a really good question.

0:33:06.2 VK: The first question, if we want to break it down, is agency versus consultancy, which is an interesting one, I think. And sometimes those words are kind of interchanged, but I think that if you’re getting specific about it, that they do mean different things. Do we want to start there?

0:33:18.0 MH: I’m not going to say anything that we can publish.

0:33:23.3 TW: Well, walking down memory lane, Michael and I, we don’t remember which coast it was on. It was definitely at an eMetrics where I had a contention with Michael that it was impossible to be successful as an analyst within an agency.

0:33:40.0 MH: Yeah, eMetrics, New York, I remember.

0:33:40.2 TW: So defining an agency as creative agency, or a media agency, or a digital agency, because the incentive structure is… The agency’s incentive is show success to get more money. And so you are running uphill so hard. So then there are companies, and I’ll name, like Further is one that is like an analytics agency. There’s much, much more of, to me, a consultancy where it is like helping with the data piece. So that’s the distinction. Michael, and I think we’ve decided that my memory is correct on this, that you said, no, no, no, I think we can be successful, and like a month later, you were gone.

0:34:26.7 MH: Well, it wasn’t that fast, but there was a period of time… I was pretty new back into the agency world, and so I had a plan I was trying to do, and then about a year later, I was like, yeah, Tim, you’re right, it’s not possible. So, yeah.

0:34:45.1 TW: And you went to Search Discovery, which was much more consultancy.

0:34:48.8 MH: Yeah, and we built out the consulting model at Search Discovery, which more like, how do we solve business problems as opposed to support paid search or whatever it is that agencies do. And I think that’s one aspect of it. I think the other things are like, working across consulting firms versus client companies. And Tim, I know you’ve got a lot of experience, and Val, you’ve got a lot of experience with that. Moe, you too. I don’t have any recent experience with that. Last time I worked at an industry company was 2010 or something, ’11, something like that. So, somebody else can answer how it’s different than a consulting firm.

0:35:32.6 VK: I’ll say that on this side, ’cause I had two in-house roles in digital analytics before I was on the consulting side. And the one thing that I missed is that I don’t get to see always where the questions come from, like those early stage, water cooler mentioned in the last five minutes of a meeting before it becomes something that someone would talk to their agency partners about. And then I also kind of miss, sometimes, the accountability on the back end. So after you deliver your analysis and you give your recommendations, when you’re on the consulting side, like this kind of like job done, and you check in on that and you cross your fingers and you hope that some of that sees the light of day and it all comes together. But being in-house and being accountable for those outcomes, sometimes I really enjoyed that pressure because you’re the team or the person or a part of a group that’s picking that path forward for the business. And so, I enjoyed some of that accountability ’cause you don’t always see it. But it is nice to not have to be a part of the politics.

0:36:32.9 MH: Yeah.

0:36:33.2 MK: Yeah.

0:36:33.9 VK: You can just kind of pop some popcorn and you watch all that unfold.

0:36:37.8 MH: You’re like, as a consultant, I’ll let y’all just argue about that.

0:36:42.6 VK: I’m just gonna go on mute.

0:36:45.3 JH: Yeah, wait it out.

0:36:48.1 TW: But the flip side is you don’t have the time and the opportunity to sort of work that not, not necessarily play the politics, but figure out, find your partners who you can go and kind of play with and do cool stuff and kind of avoid the other ones. You’re kind of stuck with the hand that you’re dealt in consultancies a lot. And I certainly, there have been times where I thought, boy the, my primary contact is making me run through quicksand to do this. If I could just talk to the person on the other side or talk to the person on the other side without them in the room, without the intermediary in the room, I think we’d be doing phenomenal things.

0:37:32.0 TW: So I feel like consultancy, you’re often running with a couple of kind of lead boots. One may be the constraints of the consultants you’re working in and what their motivations and incentives are. And the next is whatever their entry point is. And you’re like, the real value would be over there. Whereas inside with a company, and Moe, this is where you can be like, yeah, right, Tim, you’re seeing the grass is greener on the other side. You may still have the people who are problems, but you can also, a little easier to get the direct lines because you’ve got a year or two years that you’ll be working with them, not the end of this agreement, when we have to sign a new engagement to find them.

0:38:16.3 MK: Conversely though, you’re gonna be working with those people as long as you stay at the company.

0:38:19.5 VK: As long as you will challenge.

0:38:20.0 MK: So, if they’re really difficult, if they’re really difficult, they’re staying in your life and there’s nothing you can do about it. It’s not like you get to deliver it and be like, job’s done. Yeah.

0:38:33.4 TW: The joy of not firing a client, you don’t have to fire clients. You just don’t re-up with them. Because contracts are rarely five-year contracts, you just say, “Yeah, you know what, my bill rate, yeah, I’m just tripling it.” In which case, I would suck it up.

0:38:46.7 MK: One thing I will say that I actually really miss about agency, and when I say agency, I mean analytics agency, ’cause I didn’t work at another type of agency, is the pressure. I actually think that sometimes pressure is a good thing, and especially a good thing for an analyst. And our first question from Lori is like, sometimes you have to stop ’cause you don’t have any more time, and I feel like when you’re in-house, one thing that I struggle with is like, you can push back and be like, I need more time, or we’re not gonna get to that this season or whatever, and the deadline can move, and I actually don’t always think that’s a good thing. Sometimes it is absolutely a blessing, but sometimes it’s also a curse, because you’re like, I don’t know, you want people to be motivated and fired up and that sort of thing, but they don’t necessarily have the pressure there.

0:39:41.6 TW: This is why, Moe, it seems like you’re always just kind of like, where should I spend my time? Things are moving so slow that I just, you know what? I’m gonna turn over a few extra stones ’cause I don’t wanna knock off at 3:00 this afternoon.

0:39:55.6 MH: I don’t wanna skip over the other part of Rusty’s question, which is also dealing with the relative size of those organizations, ’cause I think that actually has a pretty big impact as well. When you’re working in a small company or a mid-sized company versus a really large company, it can have a very different feel and impact, and actually on both the consulting agency side as well as industry side. And I don’t know, we spend a lot of time at Stacked Analytics thinking about like, why do we like this client so much versus maybe some other clients, and what are the attributes and what is the nature of this relationship that’s like making it feel like such a good thing?

0:40:35.7 TW: Have you thought maybe you should just be delivering work instead of pondering the relationship?

0:40:40.5 MH: Well, I think what our clients are paying for is to ponder these things.

0:40:42.2 JH: Shade. Whoa.

0:40:46.1 VK: Shorts fired.

0:40:46.9 MH: Wow, Tim. Just wow. He starts up a competitor and then just goes right after us. It’s crazy. I never said a bad word.

0:40:56.8 MK: But Michael, can I talk about this large, small thing?

0:41:00.1 MH: Yeah, yeah.

0:41:00.8 MK: One thing I have noticed, ’cause obviously Canva has grown in size. When I was there, we were a bit under 500 people, now we’re over 4000, and the data science field is like hundreds of people. One of the things I’ve noticed that has actually been really difficult to manage is like, you have to bring more people in the room constantly. So like, we were doing analysis on something and it’s like, oh well, you got to get someone from this department and someone from that department, and the lead analyst from that department to make sure we’ve covered with the angles. And suddenly we were in a working group call with like 12 data scientists. And I’m like, we don’t need these many people. Like in my mind, I’m calculating the hourly right of these 12 data scientists being like, can we just have two or three people work on this and then everyone else can QA it? And that stuff gets really difficult, like the deconflicting as you get bigger.

0:41:49.6 MH: Yeah, larger organizations have the scaling problem, where you get more and more specialized in terms of what your role is, what you cover. Whereas in a smaller org, you’re able to wear lots of different hats and potentially move around in lots of different problems. Even if it’s not necessarily like your main skill set, it’s sort of like, “Well, you’re the best we’ve got, so you figure out and do what you can with it.” And then later on in life you’ve got, well, we’ve got a PhD whose dissertation was basically this problem. So they’ll touch it, you don’t need to worry about it. That kind of stuff happens.

0:42:26.4 TW: And that’s a large company that’s young. Like Moe, like yours is kind of digital first. Like, you take the large companies. Like when I work for a big insurance company that have been around for 100 years and they had acquired these and that, and so the calcification within those… And I think the same is for large… I mean, consultancy is the big ones have a very sort of structure, this is the way that we work, and I have yet to find an analyst… A lot of times it’s analysts who were at another company that got snapped up and acquired into those larger consultancies, and they’re like, this is just brutal. And bureaucracy is sort of an over-simplification of what it is. I was talking to somebody who work at Google even who was saying, I’m supposed to do this thing. It’s clearly defined, and I need to reach out to this other team, and it’s the same thing, Moe.

0:43:21.4 TW: She said, by the time we get to the meeting, there are 17 people in there, and there’s a combination of people who have touched different systems to CYA, I need to bring… Oh, I’m coming, then I need my other expert to come along and listen as well, and three people are working… My time when I was at the insurance company, it was like, I was like, could we just… If we got three people pointed in the right direction and let them go, we could get this done. Instead, we have piled so much shit on it that it will take it a year and a half to do a tiny little thing. So I know I have a bias towards the smaller, I get to a few hundred and then I start or I get to, I don’t know, five, I’m struggling.

0:44:17.8 MH: Yeah, no more co-host on the podcast or else Tim’s gonna be able to handle it.

0:44:25.3 VK: He’ll lose it. I remember, my first time out of college, I worked at a super small boutique market research firm, and I just thought that’s the way market research was done, that’s the way you interact with clients. And I went to TNS, which is a WPP, and it could not have been more night and day, and I lasted seven months at that second large agency. It was not a good fit for me. But I will say that, then I went to UBS later, which is like, what, over 60,000 people, and I loved, like what you were talking about, Moe, with some of the networking, ’cause if someone told me no, I would literally just scroll through active directory to find people’s job titles who might sympathize with my cause…

0:45:04.6 MH: That’s amazing.

0:45:04.7 VK: Put a meeting their your calendar and be like, “Hey.” I didn’t you know if they spoke English half the time. I was like, maybe they’ll see that I’m from New York and assume that we can communicate in this meeting. And, yeah, sometimes that was how I got things done.

0:45:20.1 MK: I love that. I love that.

0:45:21.2 MH: That’s amazing, Val. Living out the better to get forgiveness than it is to get permission mentality. That’s so awesome.

0:45:28.5 VK: Love it.

0:45:28.9 MH: Alright, why don’t we move on to our next question?

0:45:33.6 Rickey Messick: Hi. This is Ricky Messick from Nashville, Tennessee. With AI, it seems now more than ever, we need to improve in the art of asking good questions. I see two different audiences for these questions; for humans, business stakeholders, customers, co-workers, and for an AI, like prompt engineering. So how do we get better at asking good questions? Bonus, how do I ask good questions as I get older and more crotchety and less inquisitive? This might be directed a little bit more to Tim, but love you, Tim.

0:46:09.0 TW: Oh, wow.

0:46:10.7 MH: Ooh, boy.

0:46:10.9 VK: Oh, Ricky.

0:46:13.0 MH: No, Val, where is your one-liner about… No.

0:46:16.5 VK: The quintessential analyst?

0:46:20.3 MH: No, it was like, “Oh, yeah, Ricky used to work with him and he still has a question.” No.

0:46:25.0 VK: Ricky used to work wit Tim and he got even crankier because of it?

0:46:28.8 MH: Yeah.

0:46:30.9 TW: My work here was done. He was this laid back chill guy and I was getting crankier.

0:46:39.8 MH: Well, Ricky, I’m glad you asked. I always viewed this is sort of a volume game, just really pepper him with everything you can think of, and something will stick. No, I’m sorry. That’s the joke answer first.

0:46:52.3 MK: I don’t know. I guess, okay, yes, I get, with the rise of AI, we need to ask better questions, and like Ricky points out, with prompt engineering, specificity and good instructions are even more important. But like, is it really any different? Haven’t we always been on this kind of evolution to try and get our stakeholders to ask better questions? Maybe AI is putting a boot up our backside, but I would say, I’m constantly working with stakeholders to be like, how do we get rid of the low-fi stuff and and answer the most burning interesting questions for the business? And like, how do I help stakeholders get there instead of saying, can I have, to refer to earlier, this dashboard with this filter? And I don’t know, I feel like Taylor, the episode we did with Taylor recently, Taylor Guthrie Buonocore on asking better questions was such a good way of thinking about the type of questions and the way we engage our stakeholders. I don’t know. Tim is jumping to get in here and…

0:47:56.9 TW: No.

0:47:58.0 MK: Oh, I thought you were like…

0:48:00.0 TW: No. I think it’s great. I think, on the one hand, it doesn’t change… I think the prompt engineering, and Michael, I think has been, of the five of us, the most sort of playing and trying and exploring, and I was a little slow, but now I feel like I’ve been sort of… There is a science to prompt engineering, and lots of people have written how to get better answers out of AI. And I think there are some kind of tactical techniques for that, so there’s…

0:48:31.2 MH: Yeah, there’s some things to it.

0:48:33.0 TW: Yeah, pretend you are a whatever, what do you need to know? Whatever. So that’s like, okay, you can go Google how to do prompt engineering. I think, Moe, I’m largely with you that, to me, it just comes back to curiosity and the business focus, that asking, we talk about, what are you trying to achieve? Or what’s keeping you up at night? John Lovett. I don’t think somebody’s really agonizing they don’t have a dashboard with the click-through rate in a timely fashion. They’re thinking about, how do I make my email more effective? Well, now my question is like, well, what is it your expecting email to do? So to me, there’s a hand in hand with, you’ve got to stay curious and you’ve also got to be comfortable knowing that you shouldn’t have the answers. I think that’s where an analyst will lock up saying, I don’t wanna ask that question. It’s a dumb question. Which, one other aside, Malcolm Gladwell, it was on his revision of history feed, but it was on the… Another, I’ll put in the show notes.

0:49:41.3 TW: He was talking about his father and how his father felt he was an expert on super advanced mathematics, the Bible, and a third thing, and he said, he didn’t think he was an expert on anything else, and he would just assume that he was the dumber person in any conversation and would ask questions. And then Gladwell went on to say, and then if he was… Oh, and gardening. He was like, if he was talking to somebody who knew something about some type of flower, he would assume that person knew more than he did as well. And Malcolm Gladwell, who I’ve got some qualms with occasionally, but I can’t fault his mind. Like when he speaks extemporaneously or in conversation, he’s just curious, which means he asks questions, which means the people he’s asking questions of want to answer them. So, I don’t know. I think it goes to being curious and not feeling like you are asking a dumb question and then you’ll ask good questions.

0:50:41.8 JH: And if you feel stuck too of like, I know I should ask more, I feel like it’s really important to realize, what you’re trying to ask more about is probably context around the question. You don’t have to just think of random other bespoke questions. Like to get deeper information, it’s okay to ask more about the background, where they got the question from, or is that coming from so and so a leader you know the name of? Like, it’s okay to just pull at some threads to get that connective information, and I do think that is helpful. Because even going back to the AI prompt stuff, you guys were just saying, to be really good at that, you have to give the AI enough context to better answer your question. So same thing, when you’re talking to a stakeholder or a client, if you’re trying to get them to ask you better questions or you’re trying to clarify a question that they’d asked you to give a more helpful answer, you’re pulling on that context piece of, can I get some more background? Can I get some helpful hints around where the heck this came from? Who’s actually looking for this answer, so then you can better go serve the client or stakeholder?

0:51:44.6 TW: And Taylor did say that. In episode 240, there was the… You pulled that one out, and I was like, oh, that was coming from the discussion with Taylor, like, what if you’re like, it’s dead air and I don’t have any more questions? And that was a great discussion you guys had on what to do with that.

0:51:57.4 JH: Yeah, so good.

0:52:02.1 TW: Yeah, episode 240.

0:52:02.2 MH: Yeah, that’s a good one.

0:52:02.7 VK: Gold.

0:52:04.3 JH: Oh, yeah, her golden question, where me and Moe both gasped, like that was amazing. She’s like, “Yeah, is there anything you feel like I should be asking you?” Which is a great one.

0:52:14.7 VK: Which I immediately used in the client meeting the next week, and people were like, Oh, there were oohs in the room.

0:52:20.8 JH: Audible oohs.

0:52:20.9 VK: It wasn’t an ooh kind of meeting and there were oohs.

0:52:26.5 MH: But it is interesting, ’cause I do think sometimes when developing this as a skill set, it is sort of like there are tactics you can learn. So sort of like prompt engineering where you kind of learn like, okay, here’s how I navigate to get the AI to produce the result I want, people aren’t obviously the same as an AI, but you can learn different things that actually give you a leg up, like those types of questions, which are just so fabulous in their design. And I think another one from that same episode was, Val, continuum questions, which I think you talked about how you used one, and then I was like, I’m using that. So I use that in a one-on-one right after that, and it surfaced all this incredible discussion with the person, which we would have never gotten into if I just asked like, “Hey, how are you doing?” So it was like, “Give me the, on a scale of 1-10.” “Oh, interesting. What’s driving that number?” It’s incredible. And so that’s where learning a skill around some of those things and picking up some of those skills then actually then gives you an enhanced capability.

0:53:34.5 MH: And again, I think with people, we have to be careful not to be cynical about it, but they’re profound in their ability to do that. So I’m always looking for those, like how to do that better, or how to pick up new ways of building questions. I remember Wil Reynolds, who’s the founder of Seer Interactive, he has a huge list of client questions he’s gathered over the years, which he’s shared with us at one point, and probably, I think you shared out. We’ll put in the show notes. But it’s a great set of questions. And you put some of those in there and they’re just like those ones where it’s like, in having conversations with people, asking the right question can really put you in a whole different category in people’s minds. Because they’re like, “Wow, people don’t often ask that,” or, “They don’t think to ask like that.” And those are the things that just can set you apart. It’s hard to remember them all, So I’m thankful to Wil. He’s put a lot of work over the years and to putting… It’s a pretty big list. You can’t obviously ask everything question in that list, so you kind of like, what I do sometimes is I go through that list before a phone call, and I pick up two or three I’d like to try out, and then I pull them into the conversation. So, thanks, Wil.

0:54:42.1 TW: I’ve also had the, when having a conversation, sometimes thought, what would I need to know that I could best represent that person if they weren’t here in the context of what I’m trying to learn?

0:54:50.9 MH: That’s good.

0:54:53.0 TW: Like, I wanna be making decisions. This kind of goes back to Lori’s question as well. If I’m gonna be trying to figure out which stones I should most un-turn, I need to be able to represent them as best I can comfortably. What do I need to know to do that? And it’s not so much that you’re trying to play the game to become them, but it does sort of force a degree of empathy of, I really want to understand where they’re coming from. What do I need to know to actually understand that. Another kind of mindset.

0:55:27.9 MH: Alright. We’re having such a blast, we’re making a game time decision, we’re gonna do a second episode so that we could keep going over these questions ’cause we’re obviously not getting to them fast enough. And they’re awesome. They’re awesome questions. And some of the answers are okay too. But this is so much fun, and I hope, as you’re listening, you’re having fun too. We would love to hear from you. Even if it’s not in this format, we love your questions. You can tell we love your questions. We’re doing a whole show with some of your questions. So please feel free to reach out to us. The best way to reach out is either through the Measure Slack group, where a lot of us are, our LinkedIn page, and you can also get it to via email at contact@analyticshour.io. Well, this is fun. Like Julie, Val, Tim, Moe, I’m not the only one, right?

0:56:20.8 TW: Yeah, yeah, and a couple of us are drinking this time, just like the old time, so.

0:56:25.0 MH: So we’re gonna wrap up, and obviously, no show would be complete without a huge thank you to Josh Crowhurst, our producer. Thank you, Josh, for everything you do for the show. We look forward to getting back to all these submitted questions in a future episode, and we’ve had a blast. So, thank you all so much for submitting them. Thank you, Julie, Tim, Val, Moe for being a part of this episode answering these questions. So, remember, whether you’ve got questions or whether you’ve got answers, keep analyzing.

0:57:00.7 Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at @analyticshour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst.

[Outtakes]

Podcast: Download | Embed

Subscribe: RSS