#086: Avoiding Analytics Rabbit Holes

[music]

00:04 Speaker 1: Welcome to the Digital Analytics Power Hour. Tim, Michael, Moe, and the occasional guest, discussing digital analytics issues of the day. Find them on Facebook at facebook.com/analyticshour and their website analyticshour.io. And now, the Digital Analytics Power Hour.

[music]

00:28 Michael Helbling: Hi, everyone. Welcome to the Digital Analytics Power Hour. This is episode 86. Round and round the mulberry bush, the insight chased the hypothesis. The hypothesis was null, and pop goes the insight. Okay, but seriously, sometimes getting a great insight to use could feel like a rabbit trail, and that it feels like it goes around and around. How do you avoid these analytics rabbit holes? And yeah, we have probably covered a topic similar to this before, but we were drinking during that episode, and so far, we’ve only just started drinking. So, Tim Wilson, my co-host, how you going?

01:13 Tim Wilson: Michael, how’s it going? [01:15] ____ man.

[laughter]

01:18 Moe Kiss: What was that?

01:19 MH: See, in the show notes, all I have there is, chuckles, just chuckles. [chuckle] And of course, the star of our show, Moe Kiss.

01:36 MK: Hi. That was quite the introduction today. Do I get that every episode from now on?

01:41 MH: No. [laughter]

01:45 TW: Never again.

01:46 MH: That was so much effort.

01:49 MK: That was the first and only time, hey?

01:51 MH: I’ll need auto-tune to do it again. [chuckle] So, let’s talk about analysis of competing hypotheses, because that’s a great starting point and it has been a topic of the talk you gave that I saw you give at Super Week and I think maybe at a couple other places. And it inspired me to buy the book that I think you got that from, which maybe is a good place to start, unless that’s your last call, which we won’t talk about it again until later. [laughter]

02:18 MK: Yeah. That would be super awkward. But you can actually, if you’re game, you can actually get the book online for free from the Central Intelligence Agency. You just need to be willing to go to their website.

02:31 MH: So if you’re an EU citizen, could you tell the CIA then, to delete all information about you and they’d have to comply because of GDPR?

02:43 TW: I don’t know. What’s 4% of the CIA’s revenue? It depends on who they’re shipping arms to? I don’t know.

02:50 MH: Oh, he’s got into a whole… This is a good topic.

02:54 TW: And yet, we’re gonna claim we haven’t been drinking yet.

02:57 MH: Okay. Well, I just said we just had started drinking, and we’re very fast drinkers. Okay, maybe it’s better we go back to talking about the name of the book, which of course escapes me, but Moe, it is…

03:10 MK: The Psychology of Intelligence Analysis. The guy who wrote the book, his name is Richard Heuer, and he was an intel analyst at the Central Intelligence Agency in the 70s. And he got really into, I guess, the school of thought around how do analysts come to the conclusions that they reach and their recommendations. And he started sharing this methodology within the CIA and eventually, they published his book, which is how most people have become aware of it. But the thing is, the actual methodology is really based on the scientific method, and cognitive psychology, and then also decision analysis, which are all things that are super important to us in the analytics community. So I’ve kind of amended his work, and used it for some of my own analysis. I mean, it’s definitely… I don’t use it in its purest sense, but I’ve adapted the principles to make it work for me.

04:06 MH: Nice.

04:07 TW: So what are those steps? What’s the process? I mean, I will say that, I saw it… I’ve only seen you present on it once, and that was at Super Week and I would say my mind was a little bit blown on a few fronts.

04:19 MK: Really?

04:19 TW: It’s taking… ‘Cause if I… If we say the scientific method, which I think is a great place to start. I mean I obviously have been a huge fan of saying, “Start by articulating a hypothesis. A hypothesis is a tentative assumption about how things might work. Then use data to try to invalidate that.” Which I think is another one of your points, that what… We’ll avoid that rabbit hole for now. Whereas, ACH really puts some real rigor and meat behind that and sort of forces you to go in a sequence of steps of walking through that, and being very clear on… I guess the piece is, is that we’d like to say we have a hypothesis. We go and pull some data, and we get 27 data points and 27 of them point in the same direction, and that’s… There’s competing data as you’re trying to analyze your competing hypotheses, right? And so there’s a lot more… It kind of… It handles the rigor a bit more. And so now I’m talking kind of in the abstract, but I think that’s where sort of the matrix and those seven steps are worth relaying to the best of your ability in an audio format without a white board or a slide deck.

05:31 MK: Of course. Yeah, I actually agree because there’s a few points that I want to drill into about some of the steps that I think make this process really strong. But basically, the first step is that you get all of your stakeholders in a room and you brainstorm all of the hypotheses. And you really… You start out with the question that you’re trying to answer, about, “Why is this happening?” So, why is conversion rate increasing on our app, or why did sales go down at this week in time? I mean, you can literally start with any question that you wanna answer. And you brainstorm all these different hypotheses. But the thing that’s, I guess, useful about this approach is that you then go out… So you approach your data collection from a different perspective. Which I think… And I do this often, where it’s like, “Okay, I need to understand some stats about our app. What is everything I can find on the app?” This is really different, because you now have your list of your hypotheses and you go, “What data point will disprove each of these hypotheses?” And you only collect that data. You don’t need to collect everything under the sun, only the data points that will help disprove hypotheses.

06:37 TW: It could be multiple data points for each one of the hypotheses, right? And some of those are gonna overlap between the hypotheses, but I’m getting ahead of myself.

06:45 MK: So they will occasionally overlap. And I mean, in Richard Heuer’s case, the way that he uses it for intelligence analysis, they almost always overlap, because they have less choice about what they collect. Often, it’s like, this is what we’ve got, so we need to make a decision on it. But for us, we can be a little bit more targeted about what we collect. Then, step three is about preparing the matrix. And this is the bit that it’s probably worth looking at his book to kind of reference, so that you can see a visual explanation of it. But basically, you put all of your evidence down the side and your hypotheses across the top. And this is the step that really differs from how we work as an analyst, because typically, we have multiple hypotheses and we try and play them against each other. So this hypothesis is likely because of this, and this… And we try and compare them to each other. But in step three of this process, you actually take each piece of evidence…

07:43 TW: Step three being the matrix, right?

07:46 MK: Yeah, yeah. Putting your matrix together.

07:48 TW: Just to slow down a little bit. When you said the rows are the evidence, that’s kind of the data points or the, “We found this data.” That’s in your rows across the top or your hypotheses, and so now you’ve got cells. Every cell’s an intersection of evidence or data and hypotheses. Okay.

08:06 MK: Nice. You drew a great visual there, Tim.

08:09 MH: Truly spectacular.

08:10 TW: If they could only see my hand gestures while I was doing it as well, it would’ve been amazing.

08:14 MK: Yeah. I often, I normally just use a Google sheet for this. That’s the easiest way to do it, so that your evidence is in the first column and your hypotheses are across the top in the first row. And you work across. So you get each piece of evidence and you see how it compares to each hypothesis, which is quite different, like I said, to what we normally do. ‘Cause we try and keep all the stuff in our minds and play them against each other. And then, step four is probably the one that for me is where you start being like, “Okay, this process is really strong.” So step four is about analyzing the diagnosticity of the evidence. I always get really proud when I say diagnosticity correctly. But basically, the example that I think of is, if your CEO comes running downstairs, and is like, “Oh my God, Moe, why did sales drop yesterday?” The fact that sales dropped yesterday is a really important piece of evidence, or so I think. But the problem is, is that that piece of evidence is consistent with so many different hypotheses. It could be because our warehouse burnt down, it could be because our website went down. It could be because our competitor has a huge sale. It basically supports every single hypothesis, which actually means it has no diagnostic value. Tim, you look stunned. You okay there? [laughter]

09:37 TW: No, I’m good.

09:38 MH: No diagnostic value?

09:40 TW: In that example, though, if that is a piece of information that it went down, does that not trigger the whole process of brainstorming a hypothesis, why it would come down, where that’s kind of the outcome?

09:50 MK: It does, but the fact that, say, sales went down by 9% is not a very good piece of evidence. That’s what I’m trying to say. So in answering that question and trying to figure out if a hypothesis is possibly true or not true, it’s not a very valuable piece of evidence. Because it’s consistent with so many things.

10:12 MH: Right. So what does one potentially prove versus not prove?

10:16 MK: Yeah.

10:16 TW: If you look at that grid, that matrix, and you go across, and this is me trying to remember, there is evidence that actually does not support or refute the hypothesis. So you just leave that cell blank, basically? What actually goes in the actual cells? Is it you wind up putting in these… At these intersections, this evidence disproves, is evidence against the hypothesis. Is that literally all you’re really flagging?

10:44 MK: Yeah.

10:44 TW: So you’ve got a lot of empty cells. But you are coming at it from a… Not which piece of evidence supports the hypothesis. You’re saying which pieces of evidence, because you’ve gone through the rigor of saying, “Which data could I collect to disprove the hypothesis?” So it should be a bunch of things that would disprove if a piece of evidence particularly supports the hypothesis? Do you flag that or do you kind of disregard that?

11:10 MK: So basically, when you work across, you’re gonna put a notation in each cell about whether a piece of evidence supports, disproves, or is not applicable to that particular hypothesis. So I use a plus for support, a minus for disproves, and N/A for not applicable. I mean, you can really use whatever notations you want, like a tick or a cross, or consistent, inconsistent. But I definitely, I find the pluses and minuses, for me, are a little bit easier for your eye to see, particularly if you’ve got a pretty complex matrix. Richard Heuer doesn’t really stress doing this when you have more, four or five hypotheses, but I’ve totally done that, and it’s not an issue.

11:50 TW: Do you assess the strength with which it supports or refutes? I mean, is there a concept of a plus plus or a minus minus? Or is that kind of a later step?

12:00 MK: Well, the minus minus is not really possible, right? Because either it disproves it or it doesn’t. There isn’t like, it disproves it more. But I do use a plus plus if a piece of evidence is particularly compelling for a certain hypothesis. There’s actually, in Richard Heuer’s book, because obviously, when you’re talking about the CIA, they deal with a lot of missing information, or this thing is likely to occur, is a possible piece of evidence that they might have. They go into much more advanced methods of then looking at the probability of some pieces of evidence which have yet to happen. But basically, the way that they use it is that they could talk about the likelihood of an event occurring as a piece of evidence as well. Which is, yeah, if you go through the book, I think that’s really interesting. We really necessarily need that in analytics, in our community, but I do like that kind of more advanced step too.

12:55 MH: So you have to start with something observed though. Or some… You can’t just say, “I’d like to learn more about this,” for this to work. Okay. Yeah, I just wanna be clear about that.

13:05 MK: It’s really… You’ve really gotta have a question that you wanna answer.

13:10 MH: Well, yeah. Our video reviews improving conversion, this is what you could then lay out a bunch of hypotheses to kinda show that. As we were kind of bantering in the notes, why I brought up inductive versus deductive reasoning in terms of its uses. Because deductive, we start with like, video is good for engagement. And then we say products that have video reviews maybe have a higher conversion rate, thus proving my original theory right, so I deduce down. Whereas, inductive is, I observe a pattern and try to create a generality. That’s, depending on how you perceive through that, your hypothesis we might be moving up or down sort of the deductive or inductive reasoning kind of ladder, if you will.

13:57 MK: Yeah. I did think about that. I can’t really recall a specific place where they discuss inductive or deductive. And I think the key difference is about that’s trying to help you reach a conclusion or it’s about your reasoning process, whereas this is more about… The next step is literally you trying to disprove your hypotheses. It’s not…

14:17 MH: Well, right. Both can be true because you move to a set of hypotheses in deductive reasoning to prove or disprove your generality that you started with. And inductive, you see patterns and try to use them to prove or disprove to move to a generality. It seems like in this context or in the idea of intelligent signal gathering, you’d be using the lowest level up most of the time, just in terms of like, I am looking through all this data to… Look at how I can’t say data right.

14:50 MK: I know, I was gonna say. [laughter]

14:50 MH: I got caught in the middle. Data, data, data. I don’t know, data. You move through all this data to look for patterns or signals and then put it into your matrix. And maybe not really the other way around, starting with, video is good. Let’s think about all the places we could apply it.

15:15 TW: I find myself, from kind of the topic writ large and I tend to always go to, I’m way more comfortable and focused when I’m working from a hypothesis. And I think there’s one way to avoid data wandering or kind of analytics rabbit holes, is when somebody says, analyze the website or analyze our check-out process. I feel like I… And I have to catch myself and I always say, wait a minute. Stop. Convert what you’re doing into one or more hypotheses. I think it’s kind of common in the industry to say, where you’re trying to answer business questions, part of the ACH methodology says, yes, let’s make sure we have a business… It can be any business question, I think, and then we’re gonna use this technique to say, “We had one question, let’s generate multiple hypotheses.” Instead of translating one question into a hypothesis, this gives you the ability to say, “Let’s take that question, now I have the freedom at the front.” And say “Let’s cover all the hypotheses.” And maybe I’m gonna prioritize some of them ’cause some of them are just stupid, and I need to get to a finite number. But it’s a more rigorous way to get from one question, many hypotheses, and then evidence to try to disprove.

16:32 MH: That was my other question, was, do you have to do the brainstorming with the stakeholders, or could you just do it with a friend? [laughter]

16:40 MK: Well, there is another step that we haven’t talked about yet, which…

16:44 MH: Oh, okay. Well, maybe we should talk about that.

16:46 MK: Which is called sensitivity analysis. And part of the step of sensitivity analysis is that you basically go back to different stakeholders in the business. Particularly, I have two product managers that I love to call my black hats. And basically, you challenge your assumptions. You say, does this piece of evidence actually disprove this hypothesis? Have I made a reasonable judgement here? And that’s one of the probably the best things I think about this process, is that, in my view, sometimes if you go to your stakeholders and you’re like, “I’ve got this hypothesis, I’ve got this piece of evidence. I don’t know.” I see it as you don’t have confidence in your findings or you’re not sure of yourself, or whatever. Part of this process is actually going back to your stakeholders and being like, “Do you agree with my assumptions? And if you don’t agree, let’s address that.” And you can actually re-jig your work based on feedback. And I actually think that’s a real strength of this approach, is that that’s not something that we typically do very well because, like I said, it makes you look like you don’t know what you’re talking about. This whole process is… Actually, this is about making your analysis stronger.

17:54 TW: When you’re also their… You’re implicitly getting their engagement and buy-in to what you’re doing. You’re not saying, “I’ve analyzed all the data and this is the conclusion.” You’re kind of opening it up, and I think that’s a huge challenge for analysts. Analysts I think sometimes feel like, because they’re getting told that by their stakeholder, they’re saying, “I have a question. Go off, crunch your data. Come back when you have the answer…

18:23 MH: The answer to my question, yeah.

18:25 TW: As opposed to… And it’s, I’ve never… It’s literally never backfired on me to have a meeting, where I’m like, “I’m not at the answer yet. I’m finding some things. I’d like to get your thoughts on them as well, ’cause you’re a subject matter expert, and I’m a subject matter expert, but I don’t wanna go and interpret something in a way that there’s some pieces of information you have that I don’t have.” And it’s never been a problem. I’ve never had anybody said, “What the hell? I expected you to have the golden answer.” As long as I’ve set some level of expectation, saying, “Hey, can I grab 15 minutes? Can I grab 30 minutes?” And before I say, “Now, I’m gonna spend the next four hours digging further into this. Hey, if I’m at a critical point… If I’ve got this piece of evidence that if the way I’m interpreting it seems pretty cut and dried, I’m not gonna just go spin up some big presentation. Let me go check that piece of evidence.”

19:20 TW: I guess, the other thing, when it comes to involving the stakeholders and the hypothesis identification… So, I guess, I’ll make a comment and then I’ll have a question. The comment is, “The more we get stakeholders thinking in terms of hypotheses and understanding a process and a methodology that isn’t necessarily what they would just do if they got sent the data and started charting it, I think that builds a lot of power and credibility for the analyst over time.” But back to where Michael was kind of hinting towards a question of how often do you… What does that look like? Is it eight people? Is it three people? Are they used to this? Did you have to educate them that, “Hey, this is a process and we’re gonna try it. And here’s the exercise.” How did you get to doing this the first time, and now… Like you said, you deviate from the process in places. Is the brainstorm hypothesis one of those?

20:16 MK: It depends on the business question, right? I mean, if it’s like, “Should we invest $50 million in some AI thing?” Then I’m gonna have half the business of stakeholders, so it’s gonna be all of the execs. It’s gonna be a huge decision. And when I get to sensitivity analysis, that’s gonna be really important. But if it’s something small, I think, I used it for some analysis I was doing on conversion on different app types, I included the product manager who looks after the checkout. And that was kind of my person that I would bounce off, and I bounced off another analyst as well, because it wasn’t like that was a huge multi-million-dollar decision. So the amount of stakeholders you have and who you involve. But definitely, the first big piece of work I used this for, I had all six product managers and product owners and the head of product involved, and even some of our UX team, and we had a brainstorming session about the hypotheses.

21:16 MK: And thank God, because they thought of stuff that I would never have come up with. They know the product so much better than me, and there would have been hypotheses I never would have even reached without their input. And then when I got to the sensitivity part, I probably went back to the three that were the most involved. But that’s the thing I also like, I love collaborating through my work, and the idea of an analyst sitting quietly in the corner crunching away on their own, I don’t love, but having lots of other people involved and it being an [21:48] ____ process, to me, is really appealing.

21:50 MH: Yeah. I like involving another analyst, at least, maybe, in the initial part, but not necessarily…

21:56 MK: Just got the stakeholders.

21:57 MH: Not the business, yeah. Depends. It all depends. You can have great stakeholders, and so it makes the process really work. I just find that if you were gonna actually brainstorm a set of hypotheses, first brainstorming as a process is not well understood or respected a lot of times. And then a lot of people don’t understand the nature of formulating a hypothesis, either. And so that it seems like all that pre-work needs to be done before you could effectively engage people in this process.

22:24 TW: I have sometimes, even on my own, tried to come up with a handful of… Seed some hypotheses for two reasons. One, to show… Well, three reasons, maybe. One, to say, “We need to put some thought into this before I run off and start pulling data,” ’cause that gets very expensive. Two, I have invested my time and energy already, whether it’s by myself or with another analyst in trying to come up with some of these. And then three, it’s kind of saying, “But I’m not,” Moe, exactly to your point, “I don’t have subject matter expertise in everything. So I suspect that what I’ve come up with are not the best hypotheses. Some of them might be pretty good, but now I’ve modeled what I want from you. Can we talk?” ‘Cause I don’t think that… Brainstorming hypotheses does not need to be a half-day session.

23:15 MK: Oh no.

23:18 TW: And it’s kind of, again, it does, Michael, to your point, it goes to the relationship you have, where their heads are stakeholders. They’re stakeholders, that means they have a stake in the activity. So they should be engaged. Opening it up to, “Well, we’re always gonna invite these 15 people,” when 10 of ’em could give two shits about whatever it is you’re talking about. Or, “We’re gonna brainstorm a hypothesis for all our questions.” So I think from the overall topic of avoiding analytics rabbit holes, I think ACH is this one very specific method that I think has all sorts of stuff in it, but we’re kind of also getting to… Well, part of it is putting some up front thought, engaging with the stakeholder, not telling yourself that because they asked a question, you have to go find an answer.

24:08 TW: If you don’t have the right information… I’ve had that happen, where somebody’s come to me and said, “Well, they said they want this data.” And I’m like, “Who wants it and what do they want it for? And they’re like, “I don’t know. They just said they want it.” And, okay, sometimes that person’s an asshole, or you need to show him that you can pull the data and you need to from a relationship perspective, but I think there is a tendency, for analysts, as pleasers, we wind up in rabbit holes ’cause we’re like, “If we just go down every possible rabbit hole, Peter Rabbit’s gonna be at the bottom of one of them, and we’re gonna be able to deliver him on a platter with… ”

24:44 MH: The Golden Nugget.

24:46 TW: Yeah. [chuckle] So I think we just sorta hit on… From a very specific methodology, we hit on some broader principles.

24:54 MK: Well, I think that’s the biggest thing that I’ve learned from ACH. I don’t use this every week, probably not even every month, but that’s not the point. What I’ve learned from this process is how to better structure my analysis, because I am the type of person that goes down rabbit holes. And it’s like, “Ooh, there’s a shiny object over there.” And, “Ooh, I’m really excited about this.” And like, “Did I think about that?” And even just the way the methodology about how you collect your data, how you present it, how you challenge your assumptions, that learning, for me, was so valuable, so that even when I’m doing smaller or different tasks, I’m still applying those learnings to the other work that I’m doing. And that’s been huge for me, personally.

25:35 MH: Yeah, having a methodology that improves your own personal rigor is probably really helpful.

25:41 TW: Part of the reason… I’ll latch onto… ‘Cause more informally, but very deliberately, when I find myself getting a little overwhelmed with, “Oh, then I’m gonna slice it this way, and then I’m gonna slice it this way.” This can be the downside of programming in R, is like, “Oh, I can write a script that will just power through and slice it 12 different dimensions, every which way, because there are so many ways I could chase it. And I try to catch myself and say, “Wait a minute.” When I’m about to go into that kind of manic, chase-everything, I will actually stop and say, “Put the data aside and try to write down what my plan is.”

26:24 TW: I mean, in some ways, it’s almost like doing a decision tree in the business school format, of saying, “What’s the first thing I need to look at? What’s the branch?” So that I don’t say, “I’ll just chase everything,” ’cause then that’s just gonna give me an overwhelming amount of data and I’m gonna be lost. So I will stop and say, “These are all the ways that I think I’m gonna look at it.” I guess, in an R Notebook, or you could do in the Jupiter, if you’re doing Python, you’re using Jupiter, actually writing that stuff out. If you’re gonna be pulling the data and say, “You know what? That’s one of the beauties of this is I can put heading: I’m gonna do this. Heading: I’m gonna do that.” And just writing it down, as painful as it feels, sort of forces some organizing of, “These are the six things that I’m gonna chase, and here’s why.” And I find that, more often than not, that exercise, it is kind of like trying to write a well-written anything, you iterate through it, and I wind up organizing my approach before I’ve actually pulled any data.

27:30 MH: Yeah, and you’ve thought through it.

27:32 MH: I wanna say something really quick because, Tim, you brought up something really good, but there is that situation where you’re just sort of sitting there, looking at your data, and we talk about this like, “Don’t ever do that. Don’t just go through your data, looking for an insight. It’s way too much,” but I do think there is sort of a thing that I found useful, which is actually not looking at the data, and it’s a step before maybe the one you described, Tim, which is sitting down and thinking, and maybe brainstorming with others about things I would like to know, that I do not know, that I think would be drivers or important things to know. So a lot of those things. And then you then build-in your data exploration plan behind that, but it’s like… ‘Cause I think sometimes analysts are like, “You need to go create an insight.” “Okay, let me go… ” Crack your knuckles, open up Google Analytics or Adobe Analytics Workspace, and, “Let’s go find an insight at this data.” And you’re like, “Where do I go?” And I think that’s, before you open it up, you think back, and I just like the process of being like, “Well, what do I not know? And could that be useful to me if I knew it?”

28:45 TW: I had a case… ‘Cause this is a simple example that, I think, is along those lines, which is we look at free commerce sites, we look at conversion rate. And we sort of know we get conditioned to looking at the funnel. Like, “Oh, what’s the funnel look like?” And then we wind up with a funnel in our report, and funnels are just a horrible visualization. They’re actually one of the worst possible metaphors, ’cause they’re just… They’re actually…

29:09 MK: It is. It implies the journey’s linear.

29:12 TW: Oh, it’s bad on so many fronts. It implies, “What happened?” A funnel literally takes everything and forces it through. When we look at a funnel, we’re losing stuff. So we’re actually… The funnel is like a…

29:23 MH: I had a really great joke about that, though, back in the day, when I would put a funnel in my presentations and be like, “A funnel has to be part of every analytics presentation.” [chuckle] But… And I’d just be like, “Here’s a funnel… ” because it has to be part of every analytics presentation. And then, on the next slide, we would do a circular flowchart and be like, “And also, a circular flowchart, also has to be part of it.”

29:48 TW: And a 2×2 matrix.

29:49 MH: Okay, sorry.

29:50 TW: So now, we’re heading down a topical rabbit hole [chuckle], but we can look at e-commerce, that if you go from… Came to the site and purchased. And I don’t care if you’re talking about session-based or user-based, there is a process where you have to create a cart, you have to add something to the cart, maybe that happens at the same time. You have to start the checkout. So all the steps in the funnel, they are required steps along the way, and sometimes, we don’t actually look, “Well, what percentage of our traffic creates a cart, but doesn’t start checkout,” which is kind of a different twist on looking at a funnel… Not so much funnels, but stepping back and saying, “Okay, I want to get as many people to purchase as possible. I can break down the ‘Came to the site to purchase,’ into a sequence of steps. I’m gonna sit and think through, ‘What do I think, before I’ve looked at the data, what do I think the conversion is between those steps?’”

30:49 TW: Do I think that people, once they’ve started the checkout, do I think 90% finish, or do I think 10% finish? And come up with what my guesses are. So it’s a little bit of a decomposition of the overall flow… Or maybe not… Breaking it down into steps. I don’t know, but I think there’s a degree of saying, “Instead of diving into the data,” I mean, Rusty brought this up, Reimer, when he was on a few episodes back, where he said he knew a guy who would just say, “I’m just gonna make up a number. If I’m doing this analysis, I’ll just make up numbers, and see if I can come up with solutions that would [31:27] ____.

31:28 MK: In the context of a meeting, he would come up with a number. So someone would be like, “What’s the percentage of customers who abandon at checkout?” And he’d be like, “86%. So what are you gonna do with it now?”

31:40 TW: Right. So that’s using that technique as a means of saying, “Is it really worth chasing this?” So, I guess, I’m twisting that a little bit in saying, “Well, if I come up with what I think my best guess is, that actually grounds me. One, how unknown is it? And two, when I actually pull the data, I’ve given myself some context of saying, “Wow, that’s way higher or way lower than I expected, maybe that does tell me something.” I don’t know.

32:08 MK: Yeah, but I still feel like that’s a semi-rabbit hole.

32:13 TW: Yeah.

32:14 MK: Whether it’s way higher or lower than what you expected, is that useful? What are you gonna do with that, that it’s higher or lower?

32:23 MH: Here’s how I’ve used it in the past. And what we were doing was, yeah, you look at conversion rate between the different steps, but then you also look at the site and what people are doing as drivers or non-drivers of moving between steps. And then your analysis focuses on, “What am I trying to do to move someone from step two to step three?” Whether that be looking at a product versus adding it to their cart, and all the things that could be associated with that. So for instance, video reviews would be about moving people from product views to product add-to-the-cart.

33:00 MK: Okay. I’ve gotta chime in here. So at Conversion XL, two weeks ago… I’ve spent a lot of time talking about this, and one of the main points that I make is that it’s actually about experience and not conversion. And the reason I say that is, and everyone knows that I harp on about cross-device all the time, right? So we have Sally, who’s a customer, and she is sitting on her mobile phone at night, on the couch, and I don’t know that she’s on the couch, but I know that she’s doing this at night at about 8:00 to 10:00 PM. And she is looking at three dresses, which she loves. She sends photos to her girlfriends, decides which one she wants. And then the next day, she comes back on her work desktop and she buys said dress. Now, if you were looking at a funnel, or if you were looking at conversion rate on device, she might have had a fantastic experience that night browsing, but I just feel like in this industry, we focus so much on, “Did they buy something?” and not, “At some point, they will buy something because they had all these little great experiences that added up to one amazing experience.”

34:04 MH: Well, Moe, you are presupposing that I’m talking about a session-based conversion metric, which I’m not. Boom!

34:14 MK: Oh, okay. Boom! [laughter]

34:17 TW: Well, I think, we have very successfully gone down a…

34:20 MH: Rabbit hole?

34:20 TW: A rabbit hole on the topic. [chuckle]

34:25 MK: Can we actually get back to the topic, though?

34:27 MH: Yes.

34:27 MK: Because there’s one thing that we haven’t really talked about in a great bit of detail, and that’s about the concept of disproving your hypothesis.

34:37 MH: Ooh, I like this one.

34:38 MK: So the scientific method is really based on the fact that you try and disprove something, and you only tentatively accept what you cannot disprove. And I’m not sure… That’s one thing that I love about ACH, that I’m not sure we, as analysts, do very well, is that we often… I mean, we’ve talked about confirmation bias till the cows come home, that’s not necessarily an expression, is it? [chuckle] No? Okay, good. Good, I’ve got everyone on board. We’ve talked about confirmation bias so often, but what we’re talking about is… That’s the same with the rabbit holes, you go down here, being like, “Is this gonna be higher or lower than I expected?” rather than, “What number tells me, is this a bad experience? How many customers, or what percentage of customers have to have this so that we know that is not a great experience?” And that’s where I think the funnels and analyzing drop-off and stuff becomes valuable, but the point is it’s still about disproving your hypotheses, and that’s kind of what we miss sometimes.

35:40 TW: So I feel like the disproving hypotheses, typically, in the statistics world, is disproving the null-hypothesis and the null-hypothesis is almost always, “There is nothing there.” And so I feel like going back years, we’ve learned, “I’ve failed to disprove the hypothesis,” and the response is, “Well, if you failed to disprove the null-hypothesis, then you proved the actual hypothesis. And now we’re back off in the land of our lack of understanding of uncertainty.” But that’s not the case.

36:18 MH: But, I guess, in spirit, I really like this step, Moe, because A, embracing uncertainty, Tim, your words, and this concept of just having, I call it humility with data, which is, “My idea doesn’t have to be the right idea.” The data is gonna be the thing that tells me what’s actually happening as best as I can figure it out. And then it goes in and says, “Okay, I’m not gonna… ” And this is where, I think, deductive reasoning kinda gets a bad rap because a lot of times people use it with bias to basically say, “Here’s my generality, and I’ll only use data that proves my generality, my generalization.” You can use deductive reasoning correctly, but this idea of going through and really systematically just beating up each hypothesis, and kind of trying to make it wrong, I think is really… It’s an honest thing. I really like it. But you have to… I think it takes having kind of this scientific mentality of, “I really don’t have a dog in this fight. I really just… ” Is that a saying in Australia? “Till the cows come home? You don’t have a dog in this fight till the cows come home?” Okay.

37:30 MK: Yup. [chuckle] We do, but adult conversation makes me sad, but okay, that’s okay.

37:37 MH: Alright.

37:37 MK: I know the expression.

37:37 MH: Sad dog references. [chuckle]

37:42 TW: That dog don’t hunt, Michael.

37:42 MH: Oh boy, oh boy. But you know what I mean? Like… That’s why I love this step so much because I think that’s where we can really embrace the idea of uncertainty as analysts and be in the flow of the data and I don’t know… I get real metaphysical at this point.

38:01 MK: But it does come back to only tentatively accepting what we cannot disprove and the example that I talked through at Super Week was that at the end, we had two hypotheses left, so it could have been either of them. We hadn’t reached certainty, it’s that we didn’t have the data that we needed to disprove one of them to know that one of them was likely true. And that’s what I like. It is bringing an acceptable way to have uncertainty into the analytic space, which typically, our businesses want certainty. They want an answer, they want, “Yes, this is going to happen. No, this is not going to happen.”

38:39 MH: I feel like that’s when we get an opportunity to look for and shine a different light, so in all of our quantitative data from digital analytics, can we step over to survey data or research data or some other data that then allows us to look at a different angle at the same set of problems and see where maybe there then becomes sort of this…

39:06 TW: Would that set you up, Moe, if you get down to two remaining hypotheses, they’re still competing, and you realize you need to come up with data you could collect that would disprove this one but not this one, and data that you could collect, and so you kinda restart the process. Maybe that’s not what the business wants. They want you to bring the answer and you can say, “Look, based on this rigorous process… ”

39:33 MH: The answer is collect more data. [laughter]

39:34 TW: But I’ve narrowed it down, it’s all these things we’ve disproven. We don’t have enough evidence to nail it to one of these two, so either as a group, we roll the dice. Maybe that’s where we do an AB test, depending on the nature of them. Maybe it’s one where we say we need to collect some voice of the customer data or something else. That’s still got much more focus than I think how we historically tend to operate.

40:01 MK: So basically, that’s pretty much what happened. We had two hypotheses left. One we never would have even considered if we hadn’t gone through this process, and it was a quick fix. It was something that, as a business, we could address pretty easily. So knowing that it’s possibly or likely to be true, we just fixed it, and that alleviated that as a possibility. And then the second one was a huge… Basically, it had a huge question still hanging over it, and that led to my next big piece of analysis, which ended up uncovering a whole heap more insights. So I think it’s… Yeah, it’s about… It really gives you direction that isn’t just like, yes or no. It is like, okay. To do this… To address this one, we’re gonna do this, and for this one, we’re gonna do that, and I don’t know, I really liked the way that it really came up with a plan of attack for us.

40:53 MH: I love this, and actually, what I’d love to do right now is have us try a real world example of doing this process. Okay, so I’m gonna brainstorm a hypothesis, which is that people listening to this episode have ideas or things they would wanna share.

[laughter]

41:14 MH: And you guys, what are your hypotheses about this?

41:18 MK: That people stopped listening about 20 minutes ago.

41:20 TW: Yeah, that it’s…

41:21 MH: Aw, that’s… Okay. Well, we can disprove that hypothesis potentially. Okay, that’s one hypothesis. There’s no wrong… There’s no bad ideas.

41:30 TW: Oh, we’re brainstorming.

41:31 MH: Yeah, we’re brainstorming, Tim. Sorry if I didn’t accurately reflect a positive reaction to your brainstorm idea.

[laughter]

41:42 TW: I think this process can work.

41:43 MH: Oh, okay.

41:44 TW: Just not here. I have a hypothesis this is a very good process, just not as the way it’s being tongue and cheek applied at this juncture.

41:53 MH: So next, what we’ll do is collect data. The best way for us to collect data is through our Facebook page or Twitter or the Measure Slack.

42:02 TW: Or positive reviews on iTunes.

42:04 MH: Or… I wasn’t gonna do that this time, Tim, come on.

[laughter]

42:08 MH: I’ve gotten some new reviews and I thank you all for who has submitted them. And then once we get your data, we will prepare the matrix. Anyways, we’ll go through that, but no, I’m just kidding. We’re not gonna do that, but we would love to hear from you. And we’d love to hear your thoughts on this topic, because we’re sensitive to your analysis, and that’s all I have to say about that. Okay. One thing we do on the show is a last call. And we just go around the horn to talk about something we are particularly interested in right now or excited about. Moe, since you have been our primary host today, what is your last call?

42:53 MK: Okay. So I’ve got one point, which is on the topic, which is that I have written a blog, which I think it will make a little more sense if you have a read. So feel free to check that out.

43:02 MH: Oh, nice.

43:03 MK: And the other thing I would also really recommend is doing a cheat’s version of this, which is that you basically list your hypotheses down and then try and find evidence to disprove them, which is actually really quite simple to do, but still your emphasis is on disproving. And then keeping on topic, I had not read this article before but I’ve decided I am going to point the finger at Tim. He wrote a topic… I mean a blog called, “What will you do with that?” And it’s on the Analytics Demystified blog. And I actually loved it because, I kinda wanna make all my product managers go back and read it, but I just thought, Tim, you wrote it really clearly, and I was like, “Yes! And, yeah, and, yeah.” So if you’re going through that, getting asked for bullshit data requests, it’s worth a revisit.

43:55 TW: I feel like I have to jump in and say that the key take away is I hated when an analyst says, “What are you gonna do with that data?” Is the premise of that blog. I don’t want anybody to listen to this, not go take it, and say… Think that I, think that asking what are you gonna do with that data is a good thing for an analyst to say. I think it’s not.

44:16 MH: Excellent. Well, that covers your last call too, Tim. No, I’m just kidding. Tim, what’s your last call?

44:22 TW: So I wanna do 17 last calls, no, I’m actually only gonna do one, but the last call somewhere Nathan Yau, flowingdata.com dropped out of my rotation, and I’m not sure how that happened. And I wound up back on his site recently because there was a post that he had done that was called Visualizing Outliers and talked about the depth, he talks through, he makes you think about outliers. Is this an outlier that needs to be removed or is it an outlier that needs to be explored? And what are the visualization ramifications, and talks about scales and point of focus and all those other stuff. So it was kind of like a deep dive on the visualization of outliers, which is way down in the weeds or the rabbit hole, but actually found it pretty interesting and thought-provoking. So, Michael, what’s your last call?

45:15 MH: My last call is a blog post as well and about a specific blog. So, I like to read contrarian views and people who disagree with the conventional wisdom. I don’t know why, but I just enjoy it. Maybe that’s something about me, but there’s a guy who writes, he’s in the media or advertising space, and he’s very anti-digital media, and I just find his views very refreshing because I think they kinda comb your hair back the other direction just a little bit. But he had a blog post the beginning of March about how brands become famous, and I just really loved thinking about this concept ’cause he kinda challenged people to think of any consumer brand that had become famous only through online advertising. And it was just really, really, interesting to think about. So anyways, it’s just kind of a neat thought exercise. His whole blog is kinda worth reading. It’s ad contrarian. Bob Hoffman, I think, is his name, and I’ve probably talked about him on here before, ’cause I just… I like this guy a lot. Anyways, so check that out.

46:25 MH: Okay. Great. Thank you all so much for listening. And Moe, thank you so much for walking us through this analysis of competing hypotheses sort of methodology. I really, really like it. It inspired… After I heard your talk, it inspired me to buy the book and start looking at this more in depth. So I’m really glad we got to cover it on the show. And we’d love to hear from you. We already talked about that a little bit. So contact us if you like to. And for my two co-hosts, Moe and Tim, remember, keep analyzing.

[music]

47:04 S1: Thanks for listening. And don’t forget to join the conversation on Facebook, Twitter, or Measure Slack group. We welcome your comments and questions. Visit us on the web at analyticshour.io, Facebook.com/analyticshour, or @analyticshour on Twitter.

47:25 Speaker 5: So smart guys want to fit in, so they made up a term called analytics. Analytics don’t work.

[music]

47:33 MK: So maybe don’t bring it up ’cause I haven’t figured my shit out. Well done!

[music]

47:38 TW: Ah! [chuckle]

47:40 MH: You’re okay there? Just spilled your drink?

47:42 TW: No. I know, I just my… Whole thing. [laughter]

47:51 TW: Oh, the desk is going down. [laughter]

48:00 MK: Damn organized people.

48:02 MH: It was a lot of people. A lot of people were like, “What?”

48:07 MK: Wow!

48:09 MH: None of this can make the… Tim. None. Yeah.

48:12 MH: Okay. Perfect. Okay. Go, and just keep going. [laughter]

48:16 MK: Okay. So… Basically, the level of rigor… Do you see how I’m just plowing through, completely ignoring both of the wise cracks? [laughter]

48:27 TW: We’re like, “Moe, you have to talk about this. You have to talk about this. And now we’re going to just be just cracking wise all through it.

48:35 MK: Kind of like writing a memo.

48:37 MH: Like a one page memo?

48:38 MK: Some would say.

48:39 TW: Some would say.

48:40 MH: Like a one page memo.

48:41 MK: I know. It’s like a one page memo.

48:44 TW: You’re not familiar with testing the pH?

48:46 MK: I’ve got nothing.

48:47 TW: I was trying to see if it was a pH or not. That was literally…

48:48 MH: Like a pH balance. Something is alkali or acidic.

48:57 MK: This is super interesting, but I’ve no idea what you guys are going to do.

49:02 MH: That. The fact that you feel it’s bad, means it’s good. [chuckle]

49:05 MK: Oh no! No.

49:07 MH: No, it’s always been the case. It’s always been true. Every time, except for once, and that was real. [laughter]

49:16 MH: Red flag and rabbit holes.

[music]

Podcast: Download | Embed

Subscribe: RSS