Episode #145: COVID-19 Analysts, Policy, and Black Swans with Gary Angel

[music]

00:04 Speaker 1: Welcome to the Digital Analytics Power Hour. Tim, Michael, Moe, and the occasional guest discussing digital analytics issues of the day. Find them on Facebook at facebook.com/analyticshour and their website analyticshour.io. And now the Digital Analytics Power Hour.

[music]

00:27 Michael Helbling: Hi everyone, welcome to the Digital Analytics Power Hour. This is episode 145. Okay, take a minute, just breathe. Okay, I’m not just repeating stuff I’ve learned from my new Calm subscription and this is not a paid advertisement, it’s just that there’s been a lot going on lately, and as analysts, we get asked to monitor and figure out where things are going with our data, with our businesses and organizations, and what happens when the data makes no sense because, you know, maybe a global pandemic shut everything down or made everything quadruple, our data’s been vastly affected. And how do we analyze it effectively, and what can we learn from current events. Hi Moe, I’m sure you’re experiencing a lot of changing data over these past few months.

01:25 Moe Kiss: I definitely am, but I’m actually using the Headspace app too keep my cool so, you know.

01:31 MH: Oh, very nice. Well, I got a free subscription with my American Express card, so I was only too glad to register and I am using it and I’m getting some benefit from it for sure. Okay, and obviously, Tim Wilson, our quintessential analyst, I’m sure you know exactly how to solve this problem for people.

01:52 Tim Wilson: I just threw up my hands and went home. Oh wait, and then realize that’s what I was supposed to do so it worked well.

02:00 MH: That’s a nice general solution.

02:01 TW: Yeah.

02:01 MH: Yeah. And of course, I’m Michael and I’ve been noticing that something is off in the data lately, but we needed a guest, someone we’ve turned to before to help us make sense of analytics over the years. Gary Angel is the CEO of Digital Mortar. He took a lifetime of experience in digital analytics at companies like Semphonic and E & Y, and applied those principles to the landscape of retail. So who better to be our guest, and also establishes him as the first three time guest, Josh just added the reverb to that, of the podcast. Welcome to the show, Gary.

02:41 Gary Angel: Thanks.

02:41 MH: Welcome back to the show. That’s what I…

02:43 GA: Welcome back and I believe I was the first second time guest as well, so this is a part of a trend, and I look forward to being the first fourth time guest.

02:52 TW: We know how long our roster is.

02:54 MH: Right. [laughter]

02:55 TW: We know exactly how much… What our inventory looks like.

03:00 MH: So obviously, a lot of happened in the digital analytics space, but I think also a lot has happened in the retail space, so as you kind of sit across both of those and look at things, Gary, what are some big things that you’ve noticed over the last few months?

03:12 GA: Boy, stores are really empty. It’s puzzling. I don’t know, I feel like we have a lot of equipment that’s malfunctioning. No, I think a couple of things I’ve noticed, one thing that really struck me is just looking at the broader picture, how many of the lessons that we had to fight hard to learn about analytics, data quality, data governance, and interpreting numbers over the past 20 years are suddenly showing up in the mainstream media in ways that are both enlightening and frustrating because these are problems that we’ve lived with, fought through and learned how to do things, I think, in a fairly straightforward and formal manner, and now we’re seeing a kind of free-for-all around analytics and KPIs and people just throwing numbers out there and everyone’s trying to analyze data and I think that’s really interesting.

04:00 GA: I think from the standpoint of doing our own jobs, yeah, there’s been just dramatic changes in the way people are behaving, and that’s not gonna go away. I think one of the really significant things about this is sometimes you get brief blips in crises. You put out a YouTube video, it goes viral, your traffic spikes up 100 times, but give it a week and you’ll probably be almost right back exactly where you were. But I think all of us are looking at this and thinking that, “You know what? In a month’s time, in two months time and maybe in a year’s time, there’s a pretty good chance that things aren’t gonna be back the way they were.”

04:28 TW: Well, and even I think… I would even go as far to say one or two or three years, if you were looking at Star Trek and saying “That world is so so completely different,” I think this is one of those cases where… I’ve got a pharma client that’s looking at what’s happening with their sales reps, basically pharmaceutical reps going into doctor’s offices and sure, it went to zero and it’s gonna come back but they’re looking at it saying, “Well, doctors are gonna look up and realize that maybe they didn’t really need that interaction… ” They’re like even when everything’s all good people will likely say, “Yeah, I’m gonna operate in a different way.” Like, how many brick and mortar stores, if this all gets completely contracted, squeezes out and then how many of them could be pure play e-commerce and actually are better for that. So it seems like even when we say post-vaccine, post-treatment, this is like this huge disruption where aspects of the business models will have completely shifted and it’s really hard to know exactly what and where and how.

05:41 GA: I agree with that. I think there’s two kinds of things to look at, one is you’re gonna get some brand new trends, some brand new kinds of behaviors. I think if you look at traditional retail right now, the equation about what you look for in a store has fundamentally shifted in a way that really is just brand new. I think about six months ago when I was thinking about what store to go into. Trust was not an issue. Price was an issue, selection was an issue. Trust and safety just weren’t things you thought about. That was not part of the equation. Now I think it definitely is, and I think that’s probably not gonna go away, at least for the foreseeable future. On the other hand, I also think with things like this, you get an acceleration sometimes of trends that were already happening, and I think that’s part of what you’re talking about. I think I saw something where the basic statistic was, we’ve seen 10 years of digital adoption happen over the last three months. But that’s a trend that’s been happening already over the last 10 years. And I think when you get something like this, one aspect of it is a dramatic acceleration in some pre-existing trends. And I think some of those things like shifts to digital, that probably isn’t gonna go away. This probably just will accelerate that.

06:47 GA: And I agree, things like those pharma reps, that’s a great example too, where pharma’s been trying to cut back on reps for years and years and years, right? And there are fewer reps now per doctor than there have ever been. It’s been a long-term trend, but they’ll take advantage of something like this, both because they’re seeing the doctors don’t demand it, and because it was probably part of their overall strategy anyway to cut that. So I agree, I think you’re gonna see some dramatic acceleration in some pre-existing trends, as well as some whole new ones that no one ever anticipated, and they really are entirely just a response to new conditions.

07:18 MK: I wonder… So in Australia, I find online shopping is behind other countries, particularly when I look at something like… Somewhere like the US. And obviously over this period, online shopping has gone completely through the roof. But there were some really interesting experiences. There was a shop who I’m gonna name and shame because I was completely disgusted. Which I love this store, it’s called Kmart. It’s kinda like Target or something. And their website ended up having to have a waiting room. You had to basically wait in line to go on their website because they couldn’t scale for the traffic they were suddenly getting because so many people used to go into the store. And I almost resented the fact that some of these companies haven’t moved fast enough, and by going through this experience, it’s forced them to move faster, which is kind of a little bit frustrating because it’s almost like there’s obviously people in the company who’ve made decisions to not scale or invest on that digital side, and now they get to be weirdly the happy recipient of being forced in that direction, which maybe their leadership team didn’t choose. Does that make sense?

08:29 GA: It does but I gotta say, I think what you’re suggesting is that they saw the perfect convergence of store and digital that we’ve all been talking about. We got the waiting room into the website, so instead of bringing the best of the digital [08:42] __.

[laughter]

08:44 TW: We brought the brick and mortar experience online. [laughter]

08:48 MH: That’s funny.

08:49 MK: Yeah.

08:49 GA: Also, we would put arrows in our website that can only go up and down and these…

[laughter]

08:55 TW: We’ve introduced the equivalent of long lines… Yeah, checkout is kind of a nightmare. You have to actually wait to check out.

09:02 GA: Yeah, right. That’s brilliant. No, no, I agree with you 100%. I think that in some ways, I think it’s easy to throw stones. It’s really hard to plan for something like this, and I understand not necessarily being ready from an infrastructure standpoint. And honestly, I don’t blame like Zoom for having difficulties. How could they not. Would any of us really have been able to handle that kind of growth in our business without having some glitches and some problems? But I agree too. I think a lot of businesses that probably should have had at least reasonably scalable digital capabilities, just had never put that in place. And I think that this caught them out in ways that probably revealed fundamental problems in the way they were thinking about digital. So I kind of carve that up into… Some of that, I think is justifiable. It’s really hard to plan for massive disruptions in the world and in your business. But the flip side of that is that I think a lot of people do an insanely poor job of even planning for minor disruptions and changes in their business.

10:00 TW: Well, unless you can’t… Yeah, there’s the macro even outside of digital. Yeah, there’s a lot of investing in things that you don’t ultimately need, so weighing… It is one of those that it’s like the thinner your margins, and you can imagine if retail’s getting squeezed and they’re saying, “Well, we’re shifting more online, but we’re already running at razor thin margins.” And you’re saying we have to invest either for this catastrophic, but highly unlikely, in the grand scheme of thing. Is the pandemic gonna happen next month? Generally no. Is the pandemic gonna happen in the next 10 years? Generally, yes, or whatever that equation is. So it seems like that’s kind of the decision-making under uncertainty to quote or to use the phrase I learned from Matt Gershoff.

10:46 TW: I had a big retailer as a client years ago in the US that they basically had a Black Friday similar… Somewhat predictable that it was gonna be a spike. The infrastructure team was begging for more resources, but nobody was gonna buy into that until they shat the bed on Black Friday, at which point it became the CEO going in on January 1 and saying, “You have an open check book. That never happens again.” And that was more predictable than what’s happened now. But it also seems like it’s just kind of where are you in the cycle? Companies go through a cycle of, “We have the latest and greatest in great architecture and infrastructure.” And then over time, that’s gonna kinda degrade and when do those resets happen. There’s gotta be some degree of luck of the draw, I assume.

11:43 GA: Yeah, no, I think that’s a great point. I think Black Friday is a really interesting analogy in the sense that every retailer really should have been able to handle spikes of that magnitude. Black Friday’s huge in compar-… That whole Cyber Monday, Black Friday, that is an astonishingly huge time in retail compared to the average day, and even… It was surprising to me when I first really started doing retail, how dramatic even within a week, the swings are. For instance, a Saturday might be as much as 6 to 7 or 10x the volume of a Tuesday or Wednesday. It’s a really dramatic difference, and then Black Friday might be 20 or 50x the highest Saturday you have in other normal times of year. So you should have really had good reason to have quite a bit of scalability baked into your infrastructure just because you exist in a world that’s pretty highly… That has that kind of variation in it on a normal basis. And again, I think that’s something that in a way people should have counted as a plus going into this. They should have really had a lot of experience with that kind of variation, and they probably should have been able to handle it better than a lot of people did. Because it’s not like they’re in a business that’s been steady state every day for the past six years or something like that. They should be used to a significant amount of that kind of variation.

12:57 MK: Is there something to say, though, and maybe my memory is really short-term. I don’t know, maybe it’s a reflection of my generation. I don’t feel like there have been any events this significant in my lifetime, particularly that have such an significant impact on business. And I just wonder, they always talk about that stat, which I’m gonna totally fuck up, but there’s something about, our generation experiences more change in 10 years versus what our grandparents did in their entire life. So it’s not to say we haven’t experienced change, it’s just that the rate of change, it’s like lots of small changes over a short period of time versus this is one huge divot. And so maybe we were just not prepared to deal with it because we haven’t had any of these major global events.

13:50 TW: Well, I think… Yeah, I… I don’t think… There’s not been an event like this, especially with the degree of long-term uncertainty, but I think a lot of the comparisons to the 2008 financial crisis, some of the comparisons to 9/11, it’s like you cram those together and then you have this knowable uncertainty. But that’s actually another one. Where trying to look at the financial system seizing up for things that happened in the past, and now it needs to get… How do you get around that? This is different, I guess. I heard a really good point made that when the financial system seized up, it was the back end in the economy going. Since this has both the health and the… Everybody’s heard this a million times, ’cause it’s got both the economic and the health, yeah, it’s totally unique, ’cause it’s… What do you do with that?

14:49 TW: But it does seem like it’s one where it is somewhat useful to say, “What is this like? Let me draw from… ” And it goes back to just being like, yeah, you need really smart people saying, “How could this be modeled?” using the term as a modeling behavior. I’ve got another client that’s a healthcare client where we were talking about the adoption… We’ve talked about the adoption of telemedicine and how the patient perception is just… Actually, Gary, it’s like your example of what you were considering… What you would have considered when going into a store then versus now. And telemedicine used to be a set of hurdles which were, “Does the convenient overcome the likely technical glitches?” And so even measuring… And this is a client that does a really good job of measuring consumer sentiment through ongoing surveys and studies, but it’s like, even if you plan for the perfect scenario of where your customer’s mentality is today, it’s moving. It is changing, and there are external factors well beyond your control that are shifting that.

16:03 TW: So, yeah, it’s good to know where they are now, it’s good to know where they were two weeks ago and a month ago, but you can’t just draw a straight line and project that that’s where it’s gonna go. You kinda need to do scenario planning, and thinking through, “What if X happens? What if Y happens? What’s likely to happen?” which just feels like it is this whole soft skill of strategic thought that I think analytics has always benefited when that’s happening, but now sometimes it’s all we’ve got.

16:33 MK: Isn’t that the tough thing about being analyst right now, though, is you do use your prior experiences to help guide your approach to analysis, like, “Oh, the last time sales dropped, it was this, so I’m gonna look at that hypothesis.” And in this particular case, you can’t do that, ’cause you’re like…

[vocalization]

16:51 MK: Numbers are going up or numbers are going down. I don’t know how long it’s gonna last. You don’t have any of that previous experience.

17:00 GA: Yeah, I think that that’s a huge challenge. And Moe, I was gonna say, with regard to prior events, I agree that nothing in our lifetime, nothing in my significantly longer lifetime is remotely comparable. I was gonna say, I think the closest analogue is actually World War II.

17:17 TW: Yeah, agreed.

17:18 GA: You’re looking at a global change with massive uncertainty that hit almost every aspect of business and life, right? People were scared. No one knows how it’s gonna come out. Wars come with massive uncertainty. There are massive implications for culture, for health, for business. There were tremendous changes in supply chain, there were tremendous changes in demand, resource availability. That’s all the stuff we’re seeing now. It’s funny to think about, but this is, in a lot of respects, I think the closest analogue is a world war, and that’s sort of shocking to say. And in fact, I think in some respects, it’s conceivable that this might even have bigger longer-term implications on the basic way that business works than even the World War did. So it’s a huge, a huge deal. It’s unprecedented. There’s nothing remotely like it, and the closest analogues are so vast that it’s almost scary to trot them out and say, this is a close analogue.

18:13 TW: But the parallel to… Where I think analysts can use muscles that hopefully we’ve developed and honed is… Not to get as prescriptive as good key performance indicators and figuring out what they are. An analogue to war is saying, when we set a strategy, and we’re gonna try to do… It’s important for us to have our supply chain set up so that any soldier always has eight days worth of supplies within 100 miles, or whatever it is. And these are completely unknown situations, and somebody is saying in this aspect of my universe, this is the strategy I’m gonna pick, these are the metrics I’m gonna use to make sure that I am executing against that, these are the metrics I’m gonna use to tell everybody this is the direction we’re heading. And I’ve had some of these discussions with clients as well, saying what’s not gonna be effective is everybody running around like a chicken with their head cut off.

19:15 TW: So, if we can just figure out right now what’s our plan, what are we gonna do? There’s gonna be guess work. We’re gonna be wrong in some places, and it may need to change ’cause we’re gonna get new information, but let’s figure out right now, what is it we’re gonna do? How do we think we’re gonna do it? How are we gonna measure that? And now I as an analyst can say, “I’m gonna go back and instrument the crap out of that to make sure… ” And I’m gonna push you to say, “Where does it need to be?” And I’m gonna try to automate that, so we’ve got really tight ability to measure, are we doing what we thought we were gonna do? Because then if things are still not working, it’s like, “Okay, then we need to change our plan.” If we’re not even able to execute on that, then it’s like, “Well, it might be a good strategy. We’re just not executing it well.” So it’s the classic good strategy, poorly executed, bad strategy doesn’t matter how well you execute it.

20:10 TW: I think an analyst and the tools that analysts have can actually be enormously valuable to try to say, “Let’s at least try to focus what our new strategy is and… ” Yeah. And I’m not gonna come and beat you up if three weeks from now we need to revisit what our KPIs are ’cause, yeah, stuff is changing, but this could be clarifying for the organization. And I feel like I’m seeing that across my clients. There are the ones that have said, “Here’s where it is, and we’re gonna clearly march in that direction.” And I feel a lot better about those guys than the ones who were like, “Well, everybody just try stuff and we have no idea what’s working or why. ‘Cause we don’t even know… Really know, what we’re trying to do for a response.”

20:52 MK: I feel like I’m on board with the strategy component. I’m cool with that. I’m less sure of the KPI component. So yeah, I work at a business. It’s really weird right now when a global pandemic happens and all of your numbers look great. You almost feel bad going to the business… Well, obviously the business are happy things are going great, but you feel kind of guilty being like, “Everything’s going up, that’s amazing.” But at the same token, teams keep being like, “Hey, we did this and this and this, and look how amazing the results are.”

[laughter]

21:28 MK: And I keep being like, “Yeah, COVID.” Like that is… And like…

21:31 TW: Yeah, there were some really crappy toilet paper salespeople who are looking really good right now… That’s gonna end really… Yeah, I’m sorry.

21:40 MK: But it’s just… It’s one of those things that, yes, it’s great to have a strategy, and obviously I think KPIs are important, but the water is so muddy right now. We can actually look at… The most interesting thing to analyze was the fact that our China data, everything that was happening was two, three, four weeks ahead. The exact same trend, but China was just that time period earlier than other countries, and watching that has been incredible, but then teams keep being like, “I’ve done these amazing things.” I’m like, “They’re very amazing.” I can’t tell you whether or not they did jack shit.

22:20 TW: But that, and I think I’d had a thought on that, that’s the other… To me, I feel like, and I don’t know that I have enough of a broad view to see, but I feel like there’s a little more appetite for testing, which is somewhat for that reason. If you’re saying, “When I do things I’m doing it in a test and control environment, then I can actually get some sort of signal.” And this is not off the charts, everyone’s like, “Yeah, yeah, yeah, yeah, yeah.” But it feels like there is a little bit more of an appetite for saying, “Since we really don’t know we’re not going off of somebody’s opinion that we really need to change this.” People are like, “Okay, we’re all admitting, we have no idea what’s going on. So maybe if we did X. Well, let’s try X and not X, and see how they compare.” So…

23:07 MK: But here is the counter argument, right? ‘Cause we’ve talked a lot about testing during COVID. COVID is this weird anomaly, so is it a good time to be testing anything right now? I still don’t have a strong opinion… Yes, we’re still doing a heap of testing, and then all the countries are performing at different stages, and I don’t know, I just…

23:30 TW: Well, if anything, wouldn’t it be making a case for… Wouldn’t it be making a case for a multi-armed bandit to say, “Yeah, this stuff is so much in flight that we should be, but we need to be building these so that as things shift they’re… ” Yeah, I think it’s a good point. It’s not like, “Hey, we tested X and we tested not X, and X won so now we’ve figured it out.” Well, you know what, that may be more likely than normal to try it again in two months and you could get the opposite results, but…

24:01 GA: I was gonna say one of the things that I think about this situation in general is that the key to it is the high levels of uncertainty, and I think the right question ask is, What are the appropriate behaviors for addressing high levels of uncertainty? And I think one obvious thing is to tighten up all your decision-making cycles. There’s no such thing as a five-year strategic plan right now. It’s absurd. And a test… And when you think about testing cycles, testing for something that you think is gonna run for a year, probably is a bad idea right now. Tests should be small, they should be focused. When you’ve got lots of uncertainty, I think the key thing you have to do from an analytics standpoint is just shorten off the types of things you’re doing. Run smaller and run more aggressively, but there’s a certain kind of test that just doesn’t make sense right now.

24:45 GA: Tests that might have made sense because we think they’re gonna make a difference over the course of two or three years, or they’re big strategic tests that we think are gonna fundamentally change our sales cycle. Stuff like that probably shouldn’t be on the table right now. It should be things that are responsive to the here and now, in particular countries and places. And I just think that’s the world we’re living in, is one where you just can’t do big long-term stuff. Right now you have to be focused on little short-term stuff, and that’s probably a terrible way to be at other times. It’s not like I’m recommending that as a general business plan, I just think right now, that’s almost where you have to be.

25:20 MK: But then jumping onto those trends, so for example, Canva launched Zoom backgrounds, because obviously the world was blowing up, everyone was using Zoom, and so it was completely incredible. The team launched it in a couple of days.

25:36 MK: I totally agree with the decision to not test that. That’s like… There was a trend going on around the world, get it out the door as quick as you can and take advantage of that trend and be part of whatever is changing in the industry, right? And so then it becomes really hard to decide what is the thing that you should grab onto and just run with and what is the thing you should try and stop, slow down and test. Particularly in an environment where people are kind of scrambling and don’t know what’s going on. And you don’t know which one’s like a giant macro trend and which one’s just a tiny thing that you wanna test. So just from a strategic point, I think that decision making gets really tough.

26:14 TW: Well we’ve had the healthcare client and we have not managed to do it for other reasons but this whole shift to telemedicine. Okay, they know that for the near term, telemedicine is their best option. So therefore testing around what messaging, is it the convenience play, is it the safety play, is that I can do an appointment while I also watch my kids? And working through that which I agree could be completely irrelevant in two months or six months. And it seems like that’s the case for if the relationship… If the way the consumer is interacting with you or your customer, whoever they are, if they’re natural way of interacting with you is being forced to change, then you’re guessing on what motivations or what drivers would get them to continue or they would like to interact with.

27:07 TW: So it seems like you can kinda point to that. Okay, given where we are now and then, that’s actually also one where you start to get potentially different geographies really could be shifting in different ways. So your point where China was kind of a leading indicator for Australia for some things, it’s like well you could have the option to say, “If I’m a global company and I’m able to see who is kind of opening up in which ways, that may actually give me a leg up to say, “Well, we tried a few little short term things that we’re ultimately gonna throw away for Belgium but, hey, now I have the deck stacked in my favour when the UK comes to the same spot… In theory”. Still gotta get everybody lined up to agree to that. I’m not saying that’s easy.

28:01 GA: And in some ways I think Moe, your challenge is harder than most peoples. Doesn’t it seem to you that it’s almost always harder to get people to, a, be rigorous and, b, deploy analytics when things are going well than when things are going badly? And I guess I’m not gonna say fortunately for analytics most people are doing badly, not well right now. Heck, it’s a lot easier to pick losers, you know? And if you think about… Going back to that military analogy in a war, when you’re winning the war, generals don’t get replaced and a lot of bad generals stay on top and so does a lot of bad strategy ’cause you’re winning. But when you’re losing, it becomes really obvious what’s not working. And I think for most people right now, I do think those tight, aggressive cycles of testing, people will pay attention because they need some kind of win, right? They’re seeing disasters all over the place and it’s a lot easier to pick out the occasional rays of light.

29:00 GA: I also think, and Tim, just dealing with what you said, I was thinking as you were talking about that, I was thinking that I don’t think there’s ever been a time where voice of customer is probably more valuable than it is now, right? I’m a big believer in behavioral analytics, it’s what I do most of the time but we’ve always thought that there was a lot of value in integrating voice of customer. But voice of customer’s one of those things that often gets stale, right? If you’re running ongoing surveys, I think everyone’s had the experience of after you run it for a few months, the learnings tend to decline because you’re hearing the same things over and over from customers. Well, we’re in a situation right now where it is on a week-by-week basis, I think what customers are gonna tell you is gonna change. So I don’t think there’s ever been a better time to be doing aggressive constant in-market voice of customer and that’s something that I think any company right now should be doing.

29:49 MH: Alright, I wanna ask a different question. And it’s… Well, it’s actually fairly similar but a slightly different area, which is in machine learning and things like that. Because I think kinda to rephrase what you said it makes sense that we would narrow our scope of prediction, if you will, for any analysis we do. But now, for the next, well however long we’re gonna leverage models and things like that, there’s gonna be this big old divot in our data based on this. And so how do… I know Gary, your team and your company has leveraged machine learning in various ways, how are you looking at this from that perspective?

30:26 GA: Well, the way we leverage machine learning isn’t of much value I think from that perspective. Most of what we use machine learning for is data quality and fixing up data that we get from electronics and cameras. So we do use a bunch of sophisticated machine learning but it’s totally un-impacted by this. Our data quality problems are the same or… No different than they’ve always been. So I’m not sure that I have a great insight into that. But yeah, it’s clearly… We’re in a situation where pretty much all your training data is garbage right now and all your forecast. Whatever forecasting systems you had or whatever models you built up, most of those are out the window for most businesses and that’s… What’s the… You can take a great forecasting model and it could’ve been performing just perfectly and it’s out the window now and as stores re-open, there’s no expectation that those models are gonna work.

31:16 GA: So yeah, this is a real challenge. I think people are basically at square one having to rebuild. I think most of the models they’ve made, think about customer behavior, and that’s gonna be a challenge ’cause there’s gonna be very limited data as people go forward. And that is certainly more true on my side, one nice thing about digital is that at least the data comes in pretty quickly, right? It’s relatively easy to start to get enough data coming in where you can start to quickly regenerate models for stores. God, that’s gonna be I think a multi-month, maybe a multi-year process before you can really start to build good forecasting models again.

31:51 MK: Okay, so what do you do as an analyst? You still have to forecast, right? You need to tell the business if you’re gonna hit your 2020 KPIs. What do you do? Do you just like throw out five months of data?

[laughter]

32:05 GA: Oh that’s easy.

[laughter]

32:05 GA: You’re not hitting the 2020 KPIs.

32:08 S1: Except for Canva.

[laughter]

32:09 GA: You missed it by that much.

[laughter]

32:14 GA: No, I’m not sure. I don’t know. I think it’s really hard. I mean, forecasting’s forecasting. We do data-based forecasting, and it’s based on a bunch of… I used to teach this forecasting class, and the thing I always started out with was the idea that all forecasting is based around an implicit assumption of stability. That the key assumption we make whenever we generate any kind of forecast is that everything tomorrow is gonna be the same as it was today. And then we start maybe changing one little thing and seeing how that’s going to be different. That’s just out the window right now. I just don’t think it’s plausible to do good forecasting right now. It feels like it would be magic. There’s just no data for it.

32:55 MK: So Tim, okay, if you had this… You’re an analyst at a company, you need to… I get an email every month that tells us whether or not we’re gonna hit our 2020 KPI and there’s a forecast. Do you just put a disclaimer being like… Because right now it says we have some super aggressive targets, and right now we are completely on track to hit that target, and it’s COVID. No surprise there, and it just assumes that the COVID trend is gonna continue, which the mind baffles. Do you just write a caveat being like, “Hey, shit’s really uncertain. There is no reason that we can actually use this forecast, but here it is anyway.”

33:37 TW: I feel like that’s… It’s not gonna be a surprise to anyone that there’s a higher variability and that’s been kinda super refreshing. Maybe not the best word. But as companies are… Public companies are reporting out their financials which… We used to be able to use that as saying, “Look, why do the public companies have rigor around their forecasting and they’re being held to that?” And I’ve used that as an analogy for setting targets in marketing saying, “It is useful. It’s valuable. The markets rely on it.” And there are companies coming out saying pretty openly, “Well, here’s our target but who the hell knows?” But having said that, I know of a small agency that is… One thing they’re doing is forecasting more frequently and tracking the forecast accuracy. If I’m forecasting for tomorrow, it’s gonna be probably better… I’m gonna be closer for tomorrow than if I’ve forecasted three months ago, and actually is trying to get a handle on how much does that tighten up. How much variability, how… So there’s that aspect of actually learning about your forecasting accuracy, which is again, if you’re doing KPIs and setting targets, one of the reasons is you start to learn that you’re always optimistic if you’re just in marketing. And that’s one of the reasons of doing that as you get better at setting targets, the more you’ve set targets and realized how optimistic you are.

35:15 TW: So I think there’s a little bit of an opportunity there to actually understand how far into the future, knowing that you’re working under that uncertainty. And then I think there’s also that kind of modeling the business, depending on the organization to say, “We really need to understand what our business is.” And it depends on the… I imagine Canva… Not that useful, but I could see companies that haven’t really stopped and said, “How do things relate? Where are the gaps? And now we can build just a good old-fashioned spreadsheet.” Well, Fred Pike kinda demoed at SuperWeek where he was like, “Yeah, make a fancy spreadsheet to provide a tool to help with scenario planning.” Which is not forecasting, but it’s like, “Okay, let’s… ” What if… “What if unemployment goes to X? What if our customers are X or Y?” So there’s still an opportunity to say, “We know how to wire stuff up, and hell even run simulations.” Hadn’t thought of that too. You build and say, “Why not run a million simulations with… These are the variables that are most unknown?” So I think there’s a lot we could do and quantify that uncertainty through simulation.

36:32 GA: I like that idea. I think there is… When we used to do sales forecasting using CRM data, one of the things we did was generally run it hundreds of times over with different… Just randomly generating account closures… Account… Wins and losses based on different probabilities and then seeing what the range of outcomes was. Because in most pipelines, you have a small number of clients that are really large. A good year can happen if you randomly have some success with those people, and then you’ve got a large tail of clients and you can have a bad year if you have a lot of failures in those too. But just running over thousands of simulations of that kind of thing gives you a pretty good sense of what the risks are.

37:15 GA: What’s the low on the forecast and what’s the high on the forecast? Now that’s still with an assumption of a relatively stable environment. I think a retailer, looking at the environment right now, is faced with massive uncertainty where they could be anything from shut down again for the rest of the year, to reopened, and something close to a normal environment. That’s probably a ridiculously optimistic scenario. But within that spectrum, almost any kind of outcome is plausible, and the interesting thing about it is there’s a good chance that nothing they do from a business perspective will have any influence on that whatsoever, which is disheartening when it comes to forecasting. So I definitely think that, yes, there are some techniques you could use. I love the idea of the short forecast and finding the variability, and seeing how close you can come and then working from there. But realistically, if someone came to me and… As a retailer, if someone came to me and said, “Give me a forecast for the rest of the year.” I’d kind of look at them like they were crazy. I’d be like, “No. What do you think I am? I’m not Gandalf. I can’t predict the future here.”

[chuckle]

38:17 GA: You know as much as anybody, and none of us know very much. And there’s a concept that I think in some ways flows out of analytics. It’s actually a philosophy concept called epistemic humility, just understanding that a lot of times we don’t know a lot of things. There’s a lot of unknowns, and we’re living in an environment where there’s known unknowns and unknown unknowns, and both of those are unusually high right now, and I think we should acknowledge that.

38:42 TW: Oh my God. I literally heard epistemic humility used on a podcast within the last 72 hours. Not only was it used, one of the other hosts on the podcast knew what they were talking about…

38:53 MK: Of course he did. [chuckle]

38:53 TW: And rolled with it, and I was like, “Who do I know who know… ” I was like, “You know what, I bet Gary will drop that in casual conversation. Get out! Hey, I was a philosophy major but… No that’s something that I… I think that’s a timely concept right now, right? Because I think you see a lot of people who are absolutely certain they know things that maybe they don’t know, and I think we see a lot of that in public policy. Obviously, we see all of that in the media, obviously. We see that a lot in the world, obviously. But as analysts, I think that’s something we need to fight. And one drawback to putting out a forecast is that when you make a forecast, a lot of people take it to be the word on what’s going to happen. And I think, sometimes part of what’s important in issuing a forecast is letting people know what the uncertainty actually is, so not only a rider but a big range. Being upfront about how unknowing things are is I think the right approach to forecasting.

39:47 MH: Yeah. Although Moe, I do think you are missing a golden opportunity to jump in there and start taking credit for some stuff so…

[laughter]

39:56 MH: She’s always good advice, right? Yeah. I mean, go grab on to some of those winners, I just… Like, “Hey, look what I did, everybody.” Okay. Really great conversation, and not just because we found a way to sneak in epistemic humility with apparently very current on podcast these days. They’re trending.

[chuckle]

40:15 MH: I’ll look for that hashtag to be trending on Twitter later. [chuckle] No, but we do have to start to wrap up a really good conversation. But one thing we love to do is go around the horn and just share something with our listeners that we think might be interesting to them. Gary, you’re our guest, did you have a last call you wanted to share?

40:35 GA: You know, one thing I guess, I was gonna talk about just a little bit is that as grim as things are for retailers right now, and I think it’s incredibly trying times and incredibly difficult for them, but I think a lot of them are missing opportunities. There are chances right now in this economy for companies to fundamentally win customers over. I think about things like curbs I’d pick up, and what an opportunity it is for businesses to win customers that they would never have had before, and to win the loyalty of customers that maybe were waning in some respects. Times of change are times of high risk, but they’re also times of potentially high reward. And one thing, I think, companies should be thinking about is where are the places where we can really focus our efforts and our analytics that are going to give us real wins in this economy? And I think there are opportunities for that, just because so much behavior is changing, so many customers are up for grabs, and so many behaviors are up for grabs that if you can deliver truly winning experiences, I think you’ll get rewarded with tremendous customer loyalty coming out of this. And so, I do believe there’s an upside if people can take advantage of it.

41:46 MH: I like it. Excellent. Okay. Moe, what about you? You have a last call?

41:52 MK: Yeah, so this is a bit of an interesting one. My sister and I were chatting about Jacinda Ardern, who, if people don’t know, she is the leader of New Zealand who has handled COVID completely amazingly. And someone had told me that she wasn’t particularly smart, but she has amazing EQ, which I thought was really interesting, and off the back of that conversation, my sister ended up sharing an article with me. It’s an opinion piece from the New York Times called Women Leaders During Coronavirus by Nicholas Kristof, and it was such a fascinating read because it actually talks about the fact that every country that has a female leader has actually weathered coronavirus pretty well. And obviously, he goes through the points that it’s not necessarily because they’re female, it could be because societies that pick a female leader might be more open to take advice or might have more diverse thinking, listen to experts… But it was a really fascinating article that just kinda got me thinking about that whole interplay, like how important is being the smartest person in the room as a leader versus having EQ and listening to the other smart people in the room during a crisis. So I just thought it was a really incredible read so I’ll share the link to that one.

43:09 MH: Fascinating.

43:10 MH: And there’s a little bit of lag between when we record and when we actually… These release, which is why… I’m pretty excited about Kamala Harris in the US being in the race for the… No, we can cut that out. No. Did not land.

43:27 GA: I mean, you’re out here forecasting stuff so you know.

[laughter]

43:33 GA: You heard it here first, don’t cut that out people. We want that prediction in there.

43:36 TW: No.

43:36 GA: Maybe it’ll happen.

43:39 TW: Actually, I do want… Just Moe, just in case, let’s clarify, just in case that didn’t come out quite right. I think having the woman in the room may very well often be also the smartest person in the room. So just to be clear, that was not positioning it, and either the highest IQ or the… I think there’s some evidence that some of the women world leaders are clearly the smartest person in the room.

44:04 MK: I totally agree with that. It was just the point of intelligence versus EQ and which one can actually… Anyway, that was my own thinking about the topic, not saying that all women are stupid, but thank you for clarifying.

[laughter]

44:18 MH: I think… And Gary, you kinda made this point during the episode, is leadership got put into stark relief during this crisis. And I think as a society, we’ll be able to learn some pretty interesting things. I’m really looking forward to reading that article, Moe. Thank you. Alright, Tim, what about you? What’s your last call?

44:38 TW: I’m gonna call an audible. I had one thing I was gonna do, but I think instead I’d like to mansplain what that New York Times article was really about to Moe.

[laughter]

44:45 MH: Okay. Yeah.

44:48 TW: No. No. I actually am calling it audible and doing something different from what I originally had just because we talked about simulation there towards the end, and I wanna say again, Matt Gershoff has regularly told people to go do simulation to try to understand what’s going on. But Merritt Aho from Source Discovery posted on the CXL blog a while back a post, it was called Tempted to Peek? Sequential Testing May Be the Solution or something like that. I just butchered it. Tempted to Peek? Why Sequential Testing May Help. But what I thought was great ’cause I’d seen him working on it well before it became the blog post. He was just trying to understand the actual ramifications of peeking on a test, and he did that through simulation and what he quickly found just by putting the rigor and the thinking behind it was, well it’s not like peeking is a binary thing, there’s… There’s when you peek, there’s how you peek, there’s how you make judgments from it. And so it’s a pretty in-depth and it gets a bit over my head when it comes to the sequential testing part, but it’s kind of a cool read just as how he approached it from a, “I wanna understand this more. So you know what, why don’t I simulate it? I can simulate the world of testing.” And it’s a good read.

46:09 GA: That’s cool.

46:10 MH: Nice.

46:10 GA: What about you, Michael?

46:12 MH: Well, I’ve got a few maybe, but I’ll try to make ’em quick. So obviously, we’re recording this in the middle of June, and it’ll come out in July, but one thing that’s obviously front of mind or top of mind here in The United States, just some of the activities that are going around in terms of the protests and things like that, are happening ’cause of the death, murder of George Floyd and Breonna Taylor, Ahmaud Arbery and others. And so whenever a new thing happens, it’s not new but I mean like a current event. I’ve been using Twitter as sort of a way to educate myself more. So while I’ve got some familiarity, I’ve been trying to find more ways to do that.

46:56 MH: And so one of the things I do is, I try to find authoritative voices, people who can point me to information that I could go read deeper and dig into the information and things like that. And so I ran across a really interesting research project called “Campaign Zero”. And one of their main guys, is this guy named Samuel Sinyangwe, which I’m probably mispronouncing that, but he’s a data scientist that specializes… He does a ton of research in terms of police funding, police brutality, all these different things. And so I highly recommend him as a Twitter follow, just a wealth of information and I’m learning a lot about, kind of a lot of things I just sort of had passing knowledge of, but not really a lot of in-depth knowledge. So highly valuable as a resource to me so I’m throwing it out there.

47:43 MH: The other thing is and this one is completely off-kilter, but just allow me just a slight diversion and I appreciate it from all of our listeners. So last week I made the journey to get my first haircut in many months, and living in Georgia as I do, that’s legal. And our barbershop that I usually go to did a great job, everyone wore masks, they took temperatures, all that was safe. But as I was talking to my barber hairstylist, her name is Latasha and we were just talking about random things. And she’s told me that she had a YouTube channel. And so if you have a moment and you’re interested in hairstylists and barbers or whatever, go check out Latasha’s channel. And Latasha if you’re listening to this episode, just know that I’m shouting you out, so hoping to helping you build your subscriber count. So anyways, everybody go check out Latasha’s YouTube channel. She’s an awesome person.

48:41 MH: Okay and then lastly, ’cause sometimes we all need a little levity, and many years ago, there was this brilliant writer and comic name Sarah Cooper, who published this 10 Ways to Look Smart in a Meeting, which I have fully internalized as part of my normal state of way of operating including, and my favorite thing is asking the question, “Will it scale to anything?” Because it’s genius. She’s a super smart, funny person, and lately she’s been doing a comedic turn as taking quotes from the President of The United States and just doing a voice over of them so not… She’s just taking raw audio and then acting out and lip-syncing and it is hilarious. It’s so funny.

49:31 TW: She was a Googler, right? She was at Google, I believe it was Google. When she started writing that list, she actually was still there when the list had been published and had people… I’m pretty sure she was a Googler or Facebook or something, Yeah.

49:42 MK: Wait, wait. Where do I find the funnies? I need a good laugh today.

49:45 MH: Sarah Cooper, so she’s on Twitter and so hold on I will find it for you right now because I’m committed to your… So it’s @sarahcpr, all one word, Sarah with an H, CPR, Sarah Cooper. And she is… It’s amazing, it’s just amazing. And it’s also blowing up on the media and everything, so she’s getting a lot of media attention and I’m hoping she gets like a TV show from this or something really amazing. It’s just… Anyways that’s my last call.

50:15 TW: She was also on wait wait. She was on Wait Wait don’t Tell Me! In early June. So that’s where I… She’s funny. Yeah.

50:22 MH: Yeah, outstanding stuff. Okay, You’ve probably been listening and thinking about this and analyzing these things for your business. We would love to hear from you. So don’t hesitate to reach out to us on the Measure Slack or on our Twitter account or in our LinkedIn group. And there’s a lot of things happening around Coronavirus and how data is responding to that. We didn’t even get a chance to talk about everything we had kind of in our minds but it’s a good conversation and we would love to hear how you’re dealing with it as well. So Gary thank you once again for coming on the show. It’s great to have you back for a third time. [laughter]

51:00 GA: My pleasure, it’s been fun guys. Thanks so much for having me.

51:04 MH: Yeah, no. It’s delightful as always, appreciate your perspective. And as you’re listening, I know that I speak for my two co-hosts, Moe Kiss, Tim Wilson. Oh, wait before I do that. I do wanna give a shout out to Josh Crowhurst, our producer. So before I say that anything else, first give a shout out to Josh ’cause he’s awesome. Sorry Josh, didn’t mean to skip over you.

51:29 TW: Got a few editing challenges on this episode, so you know…

51:32 MH: And then… No just leave it just like that, I think it’ll be just fine. [laughter] Anyway I know that I speak as you’re dealing with all this uncertainty, and I speak for my two co-hosts, Moe Kiss and Tim Wilson when I say, keep analysing.

51:49 S1: Thanks for listening and don’t forget to join the conversation on Facebook, Twitter or Measure Slack group. We welcome your comments and questions, visit us on the web at analyticshour.io, Facebook.com/analyticshour or @analyticshour on Twitter.

52:09 S6: So smart guys want to fit in, so they’ve made up a term called analytics. Analytics don’t work.

52:16 S7: Analytics, Oh my God. What the fuck does that even mean?

52:26 MH: We’re just like Australia. Just dumber.

52:29 MK: Please tell me that was recording.

[chuckle]

52:31 MH: Rock, flag and worthless forecast.

[laughter]

52:40 MH: Yeah.

Podcast: Download | Embed

Subscribe: RSS