00:00:05.76 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language.

00:00:15.15 [Michael Helbling]: Hey everybody, welcome. It’s the Analytics Power Hour. This is episode 285. You know, for some reason, I have always been Bayesian when it comes to statistics. I didn’t arrive there on purpose. It was just sort of intuitive. And I also wasn’t paying super close attention to my priors along the way, so who knows really how I ended up this way. On this episode, we’re going to do something incredibly brave and slightly reckless. We’re going to talk about Bayesian statistics without scaring anyone who doesn’t have a PhD. And if you do a PhD and keep a stress ball handy, it might get turbulent. But before we get into it, let me introduce my co-hosts, Moee Kiss. How are you?

00:00:57.38 [Moe Kiss]: I’m good. Thanks for asking.

00:01:00.84 [Michael Helbling]: Nice. I’m hoping you’ll be my buddy on this episode. And Tim Wilson, how you doing? I’m going great. I’ve never come right out and asked you, Tim, but are you or have you ever been a frequentist?

00:01:20.71 [Tim Wilson]: I beats the hell out of me. I live my life for p-values that are less than 0.05.

00:01:28.26 [Michael Helbling]: That sounds like something a frequentist with. Anyways, I don’t know. Okay, I’m Michael Helbleck. We needed a guest, someone who’s worked. We’ve been appreciating for years now in the area of media mix modeling. Michael Kaminsky is the co-CEO of ReCast. He’s also one of the organizers of the locally optimistic Slack community. And he was previously a guest on episode 232, and now he is back once again. Welcome back to the show, Michael. So glad to be back. Thanks for having me. Really excited for this one. So you heard Tim say it. He is a huge fan of p-values and hacking them to get the answers he wants. Why is he wrong? No, I’m just kidding. We’re gonna talk about Bayesian, maybe just start in on sort of a little bit of an explainer into Bayesian statistics.

00:02:19.88 [Michael Kaminsky]: Yeah, so this is a great question. So there’s a lot of different directions that we can take the conversation. I’ll try to do a little bit of an overview of how I think about Bayesian statistics and how this fits into the sort of universe of different types of analytical strategies that people might take. maybe we’ll start with a little bit of history. So people, I think today, hear Bayesian statistics. And if you came up through college and maybe graduate degrees like I did, Bayesian statistics sounds like a new thing, right? You maybe didn’t learn about this in university. When I took statistics classes and econometrics classes in university, never talked about Bayes, never really talked about Bayes theorem. And so when I first started hearing about it, I was like, oh, this is some new thing that people invented. But that’s not really true, right? If we think back about the history of probability and statistics, Bayesian quote-unquote statistics is sort of the original type of statistics, right? It’s a very sort of really simple mathematical approach to thinking about how do we calculate probabilities, right? How do we estimate the probability of something happening in the future? And really, it’s the frequentist approach. largely spearheaded by this guy, R.A. Fisher, around the turn of the 20th century. That was sort of the new statistics at the time. And because this frequency approach was very convenient for a lot of important questions that were relevant at the time, it really gained a whole ton of popularity. But Bayesian statistics has been around for a very, very long time. It’s sort of the original type of statistics, if we want to call it that. But it has been resurging in popularity in recent years for some very specific technical reasons, which we might get into later. So what is the idea behind Bayesian statistics? The way that I like to think about it is that Bayeans tend to think from simulations, where what we want to do is in general as a Bayesian is we want to build a model that describes the world. Here model, you can generally think about it as simulation. I want to build a simulation that I think describes the world. What are the rules of the natural universe of the thing that I am trying to model? And then I want to compare the implications of that simulated world with actual data that we observe. and then try to learn about some parameters of interest from comparing the simulated world that I code up, right, in generally in some software programming language with the data. And so again, a lot of people when they’re thinking about statistics or thinking about A-B tests, they’re thinking about regressions, but there’s all kinds of statistical models that we can imagine that are way far outside of that, right? If we think back to like the COVID pandemic, a lot of people did really interesting work trying to model the spread of disease. And so you might have some biological model where you’re trying to think about, okay, there’s some coefficient of spread and there’s some amount of people being treated and not. And then what we want to do is we want to see, well, how well does this model fit the data? What would the coefficient of spread have to be in order to explain the pattern of data that we see? That is a very natural Bayesian approach to statistics. We’re going to start with some model informed by science, informed by our understanding of the world, and we’re going to compare the implications of that model with data that we actually observe and then use that to infer things about some parameter of interest. I’m going to pause there. Hopefully, that was like a reasonable summary.

00:05:42.23 [Moe Kiss]: Can I just jump in there? Because I am going to talk multiple times probably about the wonderful Cassie Kozyrkov, because I feel like she has this way of explaining things to me like I’m a fifth grader, which is perfect. And one of the things she said in explaining a similar, like in an effort to explain it, she was like, it’s just a best guess. And like, How does that sit with you? She’s like, when it comes to Bayesian stats, it’s about us making a best guess. So it’s like, if that is your best possible guess, why would you use any other guess other than your best guess? Even if like there’s the possibility it could be wrong, it’s still the best guess you’ve got. And there was something about that that kind of sat with me well.

00:06:28.84 [Michael Kaminsky]: So I think I probably get where Cassie is coming from. I don’t think that I would explain it exactly right because I think that that undersells Bayesian statistics a little bit because it’s like, oh, it’s just guesswork. And I think a lot of people have this association of Bayesian statistics specifically with the idea of priors. They’re like, oh, you’re just guessing, you’re just putting your own beliefs out there, and you’re not really doing any analysis. I would shy away from the idea of a best guess. What I would describe is what we want to do as scientists, I’ll just say as scientists in general, is we want to combine all of the information that we have about some phenomenon out in the world, and then combine that information in the most rigorous way possible, and then call that our best guess, right? So again, there’s a scientific process here that I think is really important. And again, I’m talking about like scientists, right? And this includes biologists, this includes people who are estimating populations of fish in the ocean, like there’s all kinds of different people who might take this approach. And what we’re often doing in the process of doing science is trying to collect evidence from out in the world and then combine it into some theory of being able to explain how the world really works. And I think the Bayesian process of sort of building a theoretical model and then comparing that with all of the different evidence that we have and bringing that all together to have some best guests or some understanding of the world, that is a good description of the Bayesian approach to science, really. So I think there is a part where it’s a best guess, but it’s not just a best guess, right? It’s a strategy and really like a really principled approach to combining information that yields the best guess based on all of that information that we have.

00:08:17.39 [Tim Wilson]: So when the contrast between frequentist and Bayesian, and I feel like you started to hit a couple of things, and I will throw in a plug because I thought my first run at trying to understand Bayesian was reading a book that was called The Theory That Would Not Die, how Bayes rule Bayes rule cracked the enigma code, hunted down Russian submarines and emerged tram from two centuries of controversy. And what you said was kind of like the one main thing I took away from it was this is not new. And it’s called Bayes. Like it’s named after Bayes Theorem from Thomas Bayes from however long ago. But you started to say that when what Fisher came up with, that is kind of, I guess, sort of the frequentist world, is it that it’s kind of computationally simpler but relies on, for frequentist statistics to work, the shortcuts or the less computationally intensive aspects of that approach do rely on some underlying assumptions being true that in the past, if you’re going that, when you go that route, you say, well, I don’t have a strong enough reason to think they’re not true. So I’m going to accept them as true and then kind of calculate. I mean, I think it is worth talking about what the frequentist world misses. And part of what you were leaning towards was some technological or computationally, one is more computationally intensive than the other. And I know it didn’t end with a question, but that’s because I’m kind of confused.

00:10:00.72 [Michael Kaminsky]: Let me try to restate some of this, and then I’ll expand on it a little bit. So Ari Fisher developed a lot of these really powerful statistical methods to solve some of the problems that were floating around at the time that he was working again at early 1900s. And a lot of the problems that people were facing statistically were in the field of agriculture. And so it was either like, hey, we have this field full of, I don’t know, watermelons, and we treated each of these different plots differently. And we want to be able to take a sample of these watermelons and then try to determine if plot A versus plot B versus plot C, which one is yielding the bigger melons. That is the type of analysis they were doing. Or, classically, you get a shipment of grain to the Guinness factory. You want to be able to take a sample of that grain and then use that to guess about the overall quality of the full shipment, obviously, without having to analyze every grain of whatever wheat that is coming in that shipment. And so they developed a lot of techniques for analyzing from small samples and then generalizing to a population. And the frequentist sort of body of work often or pretty much always takes that as the philosophical basis and even the mathematical basis of what they’re trying to do is they’re saying, okay, there’s this population that is very big that we can’t measure all of. And so we’re going to take samples from that, and then we’re going to try to generalize from those samples to the population. And The frequentist statistics work really, really well when you are in that scenario. When you are taking random samples from a large population, frequentist statistics work very, very well. For doing, I mean, and again, I use frequentist statistics a lot. I’ve published papers using frequentist statistics. I am not a zealot for Bayes in any sense. These methods work especially well in this context, right? A, B testing. you’re testing feature A versus feature B, you’re going to randomly draw from your sample of visitors to your website. Some are going to get A, some are going to be perfectly aligned with the frequentist approach and the frequentist model. Those statistics, that works really well. There’s a lot of baked-in assumptions there, right? You’re getting a random sample. There is this population. All of those questions tend to line up really nicely. But there’s a lot of other questions that we might ask that don’t really fit naturally into that framework. where we aren’t necessarily drawing samples. Maybe we actually observe all of the data. There is no sample. It’s just like, we have all of the data. I work a lot in marketing analytics. We have all of the data on marketing that we have. There’s no sample. There’s no other universe. Frequent statistics start to get weird, and you often run into this, people like, okay, help me interpret this p-value. If you ran this analysis a thousand times, what would the distribution of results be? You’re like, wait, but I have the code for the analysis, the data doesn’t change because I have the universe, what do you mean run it a thousand times? Like, it doesn’t really make sense. And so there’s lots of questions that we might ask as a business that don’t really naturally fall into this idea of drawing a sample from a population. Maybe we want to predict how much revenue this business is going to do next quarter. Like, what’s the population and what’s the sample there that we would be analyzing? And all of a sudden, it’s like, OK, well, what are we actually trying to do here?

00:13:22.40 [Tim Wilson]: Can I ask, I remember my brain melting a little bit when that whole sample versus population, specifically when it comes to time, when you can only sample the past to the present, if you’re determining your population is all time or the next year, isn’t that like, definitionally, you don’t get a representative sample, or it’s not a, you’re getting a sample that’s from a one set of times, you can’t go and pull some of your sample data from the future, right?

00:13:54.70 [Moe Kiss]: So wait, sorry. The sample bit is definitely something that is turning ground in my mind, too. So have I got it correct that in a frequentist world, you basically would need to be confident that you have a fully representative sample to then make a decision about the action you’re gonna change versus like, because there’s a very high likelihood that you have a shitty sample, right?

00:14:21.21 [Michael Kaminsky]: And so is that part of the- A lot of the Frequentist math is about determining what sample size do you need to believe that it is representative, right? That’s what a lot of like the machinery behind the statistics is actually sort of doing for you. The p-value has baked into it this idea of, well, how big is the sample? There’s an n minus 1 divisor somewhere in the formula, and that’s controlling for, well, how big do we think the population is? How big is the sample size? It’s trying to do some of that machinery for you. But you’re right that that does matter. One of often the core, a core assumption of a lot of frequentist methods is that you have this random sample from the true population. Often, it is the case that we don’t have that. And then again, the sort of frequentist paradigm, again, people who are like arch-frequentists or have PhD statistics are going to quibble with this. But often, I feel like the frequentist paradigm starts to really break down or get very confusing once you get into this world where it’s like, okay, we have a non-random sample, now what do we do with it? And that’s a place where, all of a sudden, it’s like, oh, all of these tools that we have no longer really apply, all of the frequency tools we have no longer really apply, but it is a place where you can start doing Bayesian modeling very well. You need to account for that non-randomness. You need to have some story that you can program into this simulated world about how your sample came to be non-randomly. But you can still make forward progress there very easily. You just simulate what would happen. Here’s what it would look like, here’s the implications of that, and you can go on your merry way. That I think is one of the really powerful tools of the Bayesian approach is that you can start just from what is my model of the world and then work from there. You don’t have to be like, okay, do I have a random sample from a population and then what do I do with that? It opens up the aperture on the types of analyses you can do and the types of questions that you can answer.

00:16:19.03 [Tim Wilson]: So can you maybe come up with, I don’t know, like a baseball analogy to explain? So this is such a good example if you follow baseball and sorry. Yeah.

00:16:32.00 [Michael Kaminsky]: So Tim is taking this up because I sent over some notes via too many notes via email. And this was one of the examples. This was one of the examples that resonated with me very early on in my journey. Actually, I’ll give you all some insight into bibliography. When I first heard about Bayesian statistics, I was incredibly skeptical. I was like, What is this prior stuff? This all sounds like BS. It sounds like you’re just putting your thumb on the scale. I trust my statistical significance. I don’t want to hear about all your priors. And so it went on a long journey to start to really understand and appreciate the Bayesian approach. And this example that someone shared with me, I don’t remember the original source, really resonated with me. And so the story is that imagine that you’re at a baseball game. It’s called Opening Day. So again, apologies to everyone who doesn’t follow baseball. Opening Day is like the first game of the season. And you watch some batter come up to bat. Batters generally bat about four times in one game. And you see this batter get a hit four out of four times. So they are batting on average of 1,000 on this opening game. And then I say to you, I say, okay, well, what do you think, Tam knows baseball? So, Tam, what do you think this batter’s average is going to be at the end of the season, given that we just saw this batter go four for four, batting a thousand with this at opening day?

00:17:53.45 [Michael Helbling]: So first I’m sorry, is this MLB or Savannah Bananas?

00:17:57.59 [Tim Wilson]: Let’s call it, let’s say MLB for now.

00:17:59.15 [Michael Helbling]: Yeah, okay, MLB, okay.

00:18:00.98 [Tim Wilson]: Well, now, so here’s, I don’t even know whether I’m winding up giving a intuitive or a Bayesian or a frequentist answer, but I would say, well, that sample size is so small, I’m gonna say that I think it’s probably 300 or 325. Okay, 300, how’d you get to 300? Because 300 would be a good solid respectable from this early, Viewing, it was way better than that, but it’s only four at bats, and it’s a long season, and there will be some regression to… Well, yeah, but why 300? Where did 300 come from? Why does that not jump to your mind? To me, not being a super, super close follower of baseball, that’s a number that I know is kind of normal. Maybe 250 or 300 is standard. It’s the average. Yeah.

00:18:47.06 [Michael Kaminsky]: So just to talk through this logic, the idea here is we just saw this batter do really well. Four hits in a game is really, really good. So we think this batter is probably a good batter. But we’ve only seen one game, and we know, based on our other knowledge of baseball, that Batting 300 for a season or for a career is really, really good. That’s Hall of Fame. So 250 in general, almost every batter is going to end the season between 200 and 350 as an average overall. This is 0.2 to 0.35. Again, apologies to those who don’t follow baseball. So what Tim did here is Tim used his Bayesian reasoning to get this idea. We have a little bit of data, and we want to combine that with our prior knowledge of how baseball players normally perform. We know that pretty much everybody’s gonna land between 200 and 250 in the season. This player seems pretty good, so maybe at the top end of that range, 250, 300, maybe a little bit better, but that’s probably where he’s gonna end up. That reasoning is a Bayesian approach. We have prior information about in general where people land, we observe a little bit of data, we combine that together in our mind, and we come up with some range that we think is reasonable based on that combination of evidence that we have. The Bayesian approach to statistics is very intuitive for humans. Humans do Bayesian reasoning constantly. You have prior expectations about what’s going to happen, and then you observe evidence, and then you update that in your mind. And your mind is constantly doing this all of the time. It’s not necessarily applying Bayes theorem exactly, but it’s doing this process of combining evidence together to come to some conclusion. And then the apparatus of Bayesian statistics that we have gives us a very formal, very precise way of doing that reasonably efficiently.

00:20:34.46 [Moe Kiss]: Sorry, go ahead. I’ve always sort of boiled things down to a TLDR, aren’t I? It’s updating your beliefs, right? That is the most simple way I try and rationalize this.

00:20:50.68 [Michael Kaminsky]: That’s exactly right.

00:20:52.13 [Moe Kiss]: coming to the business, which I feel like I’m always trying to do. So you’re working with your stakeholders. You feel like you’ve done a really good job of being like, this is the way that I have come up, like the assumptions that I have made. These are the beliefs that I have. And then over time, they adjust. What does that process look like with your business stakeholders? Because I presume, so we talked about this a little bit earlier about the sample, but the quality of the assumptions you make also are based on your knowledge of that particular area. I know shit about baseball. So listening to that, I was like, I’d probably make, my assumptions would be pretty shit versus someone who, I don’t know, works in data in baseball. There’s would be much better. The same would be true if you’re working with a business stakeholder on a problem. They might have business expertise that could help make better estimates to start. How involved or not involved do you think the business stakeholders should be if you’re using a Bayesian approach to solving a problem?

00:22:00.12 [Michael Kaminsky]: I think ideally very involved. Again, I talk about how do we do science? If you’re a scientist, you should start from all of the theory. You’re going to go read all of the papers. You’re going to go learn deeply about how other people have studied this particular phenomenon. And that is going to give you the best starting point for learning something new about the world. And in business, we want to do the same thing. As analysts, as scientists, we want to start by talking to the domain experts. We want to learn about how do you think about how this system works? What are your expectations? How do you know that? And then once they share that, you can use that information to, again, sometimes you can’t approve on it. You’re like, you clearly know what you’re talking about. I can’t help you. You’re like, good luck. But sometimes we can take that information and combine it with other data in smart ways to generate new insights. But I would say, As a scientist, again, we want to be talking to the people who actually know how the system works. That’s what’s going to allow us to build better structural models, this idea of a simulation, and it’s going to allow us to better incorporate their knowledge explicitly and directly into this model. If this person says, look, there’s no way that this effect that you’re talking about is larger than X, Well, great. I can put a bound on my model and that really helps us to get more precision on the other estimates if we know that it can’t be x. It might save us a bunch of time. Again, this Bayesian process, even for quote-unquote frequentists, often happens very naturally. Many of you all might have been in this situation where you’ve been tasked with doing some analysis, maybe you’re running a regression to try to get some answer, and you run your regression and you get a number and you look at the number and you’re like, that can’t possibly be true. It doesn’t make any sense. You throw it out. And then you keep tweaking the model, you tweak the model, you tweak the model, you throw a bunch out. Finally, you get to a result that’s like, okay, yeah, this sort of makes sense. That is a base in process, right? Like you have priors about what you expect the results are going to be. You’re throwing out a huge number of potential analyses, but you’re just doing it based on your judgment. You’re actually sort of doing this in like a sort of quote unquote bad way, right? A frequentist looks at that process and is gonna say like, ooh, no, that p-value you have, doesn’t mean what you think it means, because there’s actually like a ton of other comparisons that you just did. And so that process, which again, all analysts do, because we look at the results and we think about them, we say like, does this match our expectations about what these results should look like? You throw out other results that don’t, and you’re only left with the ones that are. The Bayesian system is just a more formalized, honestly more rigorous approach to that same process.

00:24:37.87 [Michael Helbling]: It seems like to be a good poker player, you need to be a Bayesian.

00:24:42.55 [Michael Kaminsky]: Poker players tend to be good patients. They tend to think this way in terms of, because again, you think about like a poker hand, right? Or again, what is the popular type of poker that people play right now? Texas Hold’em. Texas Hold’em, right, Texas Hold’em. You have some information that you have about what cards you have. You see, you know, cards come out one at a time. And based on as the cards come out, you have to sort of update your belief about, is my hand a winner or not? Or is my opponent’s hand a winner or not? It’s again, it’s this process of updating our beliefs about what is likely to happen in future states on the world as new information comes in. And that, again, is a very Bayesian approach to just thinking about what’s gonna happen in the world.

00:25:24.18 [Michael Helbling]: But there’s a big difference in priors between a professional and a, you know, that’s exactly right.

00:25:29.58 [Michael Kaminsky]: Professionals will have a lot, they’ll have a much better model and a lot more confidence about, they’ll just know the numbers a lot better. It’s actually like, that’s actually a less, maybe not as good of an example on priors because they’re sort of like a mathematically correct answer in the case of poker.

00:25:44.85 [Michael Helbling]: There’s a bunch of probabilities they’re doing as well.

00:25:47.09 [Michael Kaminsky]: Yeah. But like the sports analogy is maybe a good one or like a really good scout or like a recruiter for some sport. And again, this applies to soccer or football or whatever, Aussie rugby or something. Yeah. That like those scouts might have much better. expectations on how well this player is going to perform because they’ve seen so many others, they’ve studied it so much, they’re going to have a much better prior on the future career trajectory of some player than I will because I don’t know as much about that sport and I’m not a trained scout. And so experts can have a lot better guesses about future outcomes and a lot stronger beliefs because of all of that expertise that they have.

00:26:26.40 [Tim Wilson]: So when you say So I think there’s the priors in taking and updating your beliefs, but you also keep saying simulations. And I keep thinking, okay, if I’m gonna go back to the baseball example, if I have knowledge that, look, between 200 and 350, that’s gonna be where batters wind up being. I’m gonna run, I’m gonna build a model where I have batters and take four at bats and see how many of those, as I range them with their overall batting average between 250 and see how often I come up with four hits in a row. Is there literally simulations like you’re picking? That is literally how I would do this.

00:27:14.74 [Michael Kaminsky]: What you just described is literally how I would solve this problem. I am not very good at traditional equations-based math. There probably is a shortcut to coming up with this answer that someone who is really good at math and probability could write down. I would just simulate it. I do exactly what you said. I have a population of hitters that would have a range of overall batting averages, and then I’d run the simulation and see how often do we observe a person getting four hits in one game and then see how that relates to the range of batting averages they end up with. It is an analytical way to generate an answer to this question. That is, in some ways, very computationally intensive. Running that simulation, it’s going to be a pain. It’s going to take a few hours. It’s going to get coded up. It’s going to take a few hours to run the computer. But it will give you the right answer every single time. And you don’t really need to know any statistics to get the right answer. You just simulate. And that is, go ahead, Moee. No, oh God, I was just gonna say, that is what I like about Bayesian statistics is that you don’t need to memorize all of the different rules, right? You don’t need to memorize, oh, what are the assumptions of the linear regression and oh, am I gonna run this check and that check and this check to make sure it’s just like, we’re just gonna simulate. We’re just gonna simulate from what we believe based on the structure of the problem that we’re analyzing. We’re gonna compare it to the data that we observed and we’re gonna count. And so it’s a simulation and it’s a counting problem. And again, this goes back to why is this becoming more popular now? It’s becoming more popular now because we have more powerful computers that can do more bigger simulations faster. If you wanted to answer one of these fairly complex problems 20 years ago, you just couldn’t do enough simulations. It would take way too long. The computers weren’t powerful enough. The algorithms for doing these simulations efficiently weren’t powerful enough. Now, they’re getting a lot more powerful in terms of computers and in terms of the algorithms that we can do to explore different parameter spaces, and that allows us to answer these problems. Again, where we don’t have to learn any rules, we just simulate and then count. We put way more effort on the computer and way less effort on the theory and the assumptions that go into these shortcuts that can give you really nice answers in the frequentist paradigm, but aren’t often as flexible as just the straight simulation approach.

00:29:26.55 [Moe Kiss]: I want to talk about the Enigma machine because it’s something that I’m mildly fascinated about. Personally, I’m going to be very open that I’m making more of an effort to learn more about stats. This is something that I want to do over the next period of time. So I can attempt to explain it, and we’ll probably fuck it up. Or if someone else feels more confident in trying to explain it, I am also happy with that. But this is me live learning. So you guys let me know whether we’re going to go with option A or option B. I say take it away, Mao.

00:30:01.62 [Michael Helbling]: Yeah, go for it, Mao. I’m ready. Because I’m pretty sure I couldn’t, so.

00:30:05.89 [Moe Kiss]: Enigma Machine used in World War II by the Germans, they thought it was unbreakable because there was basically so many different variations. Of course, very famously, Bletchley Park, they broke the code. There were some great men involved, also many, many, many amazing women, which is why I know lots about it. Like I said, I’m going to do my best to try and explain it. My understanding is that the machine A could never equal A in the machine, right? So like, you know, A can’t equal A, B can’t equal B, et cetera. So you have like a series of beliefs, rules that like is your starting point. And then they had I don’t remember exactly the cribs, but basically they would go in and they would make a guess using what was like a common word from the message. So like deer or like kernel or like I’m trying to think or if it was a rain, a weather report, it might be like rain. And so they know what the output is they’re looking for. So they would try and make like a best guess of, I think that if I tried to put be here and we’re trying to spell the letter rain, then the next letter, which is A, should be this. And it was basically them testing out a series of these different things. And if they got one letter, then they would go down that, I think they called them cribs. And that was basically how they ended up breaking the code. But they would have to do that again every single day because they change the codes every day. This is one of those things that they constantly come back to. This is an example of Bayesian a Bayesian approach.

00:31:44.68 [Michael Kaminsky]: So it sounds Bayesian to me, right? Because basically, so let me again, try to just re-describe in my words, what you’re describing. I am not an expert on this, but like the intuition behind this makes sense is we’re going to. So we have some model of how like I’m we’re English, right? We have some model of how German works of the frequency of different words, the frequency of different letter pairings, certain common words, certain common patterns that we’re going to look for. And then what we’re going to do is we’re going to throw a ton of computation at the problem. And we’re just going to say, look, we’re going to run all of the different combinations as fast as we can. And we’re going to see, based on the expected distribution of certain, like again, letter combinations, under a bunch of different assumptions, these are the parameters in the model, right? How often do these expected combinations come up under these different parameters? And that lets us back into the idea of, OK, well, the patterns workouts is that this is probably the cipher. Again, I don’t even actually really know exactly how you would say, oh, this is Bayes’ theorem applied here, but this idea of we’re just going to look for patterns and we’re going to run all of the compute and count the things up and that’s going to give us some indicator of what’s most likely. That again, it’s a very Bayesian feel to the type of analysis that we’re talking about.

00:33:02.67 [Moe Kiss]: Okay, so everyone has homework, but my understanding is they did start with people doing this and then the enigma machine ended up taking over, yeah.

00:33:11.97 [Tim Wilson]: I mean, that’s in the title of the book. I don’t remember the details of the one that I was that somehow Bayes ruled to help crack the enigma code, but you gave them, I don’t remember how they said that that worked.

00:33:24.71 [Michael Kaminsky]: There’s probably something to it where it’s like, because like the total number of combinations is too big. And so you have to smartly guess what is the next combination that you’re going to check. Like what is most likely based on the information on like the things that we have done so far? And so like my guess is that thinking about what is the next sort of combination of cipher wheels? Are we going to try that is most likely given what the information that we have. And then based on the results of that, we’ll have some new best guess about what the next best guess to try is. And then we’re going to do that. And that allows you to not have to run all whatever 7 trillion versions there are because you want to have the smart like you can’t run all 7 trillion. And so you say like, okay, we have to do the ones where we have the best guess first. And the base theorem would give you a principled way to think about what’s the next best guess based on everything that we’ve observed so far.

00:34:15.62 [Moe Kiss]: Okay. So now to go down another complete rabbit hole because that’s what I like to do is like, what are all the questions that I have on this topic? Let’s talk a little bit more about bias, because I feel like, is that where everyone gets into, like, tips about this, where, like, the Bayesians think frequentists have, like, more bias issues when it comes to selection, and then the frequentists think the bias is to do with the beliefs that the Bayesians are making? Like, is that the crux of the bias debate, or is there, I feel like you’re going to explain it in a much smarter way than I probably have?

00:34:52.05 [Michael Kaminsky]: I’ll try to jump in with at least what I have seen. And this is also part of my personal journey. I talked about I had this similar belief about Bayjens when I was first learning about it, where I was like, oh, you can just use priors to make the model say whatever you want. And I was like, that feels wrong. That’s not science. You’re just using your priors to make the model say whatever you want. And what I learned is that Bayjens, the response to that is they say yes, that is true. I can use my priors to make the model say anything. just the same way that you can run the regression a thousand times and just selected the version that you want to use. The problem is that there is no way that we can design an analysis procedure that will prevent an analyst from shaping the results. The analyst has the power to shape the results always. There is no procedure that eliminates the possibility of analyst bias. If we think that we are working with an analyst who is going to try to trick us, There’s basically nothing we can do. The p-value is not going to prevent us from the analyst tricking us. Bayesian approach, not going to prevent us from the analyst tricking us. There is no way to get around that. The analyst can always bias the results. And again, those of you who are data scientists who have worked in academia or an industry, you probably feel this in your bones. You have run analysis different ways, and you just know that I can always run an analysis and make the data say anything I want. It’s always possible. I’m very good at it. I can do it. So that, again, there is no procedure that will prevent that. And so if you’re in a world where you’re like, we have to figure out the procedure that’s going to prevent that, throw it out, never going to happen. Once we’re in a world where we’re like, okay, we’re interested in finding the truth together, then we want to start thinking about, well, what’s the best way to do that? It’s totally correct that, yes, an analyst could bias the results with the priors, but a part of analyzing a Bayesian analysis is also looking at the priors. The priors shouldn’t ever be hidden. When I am reading the results of a scientific paper, whether it’s using Bayesian analysis or frequentist approach, what I’m going to be looking for is, well, what are the assumptions that’s being made? Exactly what is the structure of the model being used here? How is that combining with the data that they have and generating these results? The priors should always be a part of that. If you read an academic paper where people are taking a Bayesian approach, they’re always going to share their priors. That is just part of understanding a Bayesian analysis. What I tell people is that an analysis is an argument. It’s not truth. An argument is up to the reader or the person hearing it to make judgments. Is it a good argument or is it a bad argument? If I see someone doing an analysis and they’re taking a patient approach and I don’t agree with their priors, I’m like, those priors don’t make any sense. It’s going to be a very not very compelling argument. I’m going to say that I’m not going to buy your results. The results aren’t going to be convincing to me because I don’t think your priors make any sense. But good analysts or good Bayesians, they’re going to try to justify their priors. They’re going to say, look, these are the assumptions that I’m making. Here’s what it’s based on. It’s based on these other experiments and these other papers and this theory. Combine that with the data that we have. It’s going to be limited in some way. And this is the implication on the other side once you go through that process. And maybe that’s very compelling. Maybe it’s not, depending on exactly what assumptions are made. But it’s the totality taken together that makes it a compelling argument or not. Not does it have two or three asterisks in table number two.

00:38:23.75 [Moe Kiss]: What I don’t get is can’t frequentists just solve this by also being more open about potential misinterpretations or reasons that I don’t know, they might have run the same test 20 times, but only taken one result. The fact that they took the results early. Any of the ways that you’re typically p-hacking, who did they just have the same thing where they’re like, these are the explicit ways we could have made an error in this analysis, and then wouldn’t that just be like, great, now both methods are both great, and we use one depending on which one’s better for the problem we’re trying to solve.

00:39:04.11 [Michael Kaminsky]: I think the really good scientists do that, and that’s what we see. The good scientists do exactly that. And it’s great. I love that. Again, I am not a zealot on one side or the other. I think the good scientists do that. They admit where there is risk, where certain assumptions are potentially biasing the results. Really good scientists, again, both frequentist and Bayesian analyses, they’ll run sensitivities. And they’ll say, if this assumption that I made was different, if the priors were different, how much does that affect the results? If this assumption is different, how much does that affect the results? That’s the sort of stuff that you really want to see to understand. And that’s what makes for a very compelling argument. The problem comes when people think that a p-value of less than 0.05 means that something is truth. And that’s, again, I think we all, there’s growing awareness that that’s not true. People listening to this podcast probably know that that’s not true. So there’s growing awareness of that. But if we’re taking sort of like an honest approach to science, we need to recognize that all of these different assumptions that we make are impacting the final results. We need to be transparent about what those are. We need to, in the interest of, again, getting towards truth, the scientific process, explore the implications of those different assumptions. And if we do that, we’re going to end up closer to the truth. And again, now all of a sudden, there’s not such a difference between this Bayesian approach and the Frequentist approach, right? If we’re doing this all very transparently, we’re going to end up probably in a very similar world.

00:40:36.02 [Tim Wilson]: I think that’s, as you were saying that, I remember Matt Gershoff saying that at some point. I want to say Chelsea Parlatel-Riedi said it as well. The people who are genuinely knowledgeable, if you have enough data and you put a similar amount of rigor on it and you have that you’ll, you’re going to wind up coming up with the same answer. You don’t hear people saying, well, the Bayesian says that the truth is X and the rigorous Franckletist says that the truth is Y. One, neither one of them is declaring a truth. And the other is they should probably be getting at results that even if what they’re saying is a little bit different, they, Neither one would contradict the other if they’re both done rigorously and well, and maybe I’ll add on to that. Frequencies have p-values and confidence intervals, but Bayesian have credible intervals. Does that tie in that if you’ve only got a credible interval, if you’ve done this Bayesian approach, you’ve got a confidence interval, you’ve done this other approach, They’re not the same thing, but they’re both kind of pointing to the level of uncertainty. And if you’re doing a rigorous job, you’re going to be indicating an appropriate breadth of uncertainty in whatever conclusion you’ve arrived at. Yeah, so I think that that’s great.

00:42:04.53 [Michael Kaminsky]: It’s mostly fair. The things that I would add on to it is that almost always you can get to the same frequentist results with a Bayesian approach. There’s a few exceptions, but almost always you can sort of back in. And if you give me some frequentist analysis that you’ve done, I can sort of write those same assumptions down using priors and model structures and get effectively the same results. The wrinkle is that you can’t always go the other way. There are sort of questions that you can answer with a Bayesian approach that are just really not possible with a frequentist approach because of the limitations of the specific assumptions that are being made and the specific mechanics there. Again, because the simulation based approach is effectively infinitely flexible, right? If you can simulate it, you can generate a statistical analysis with it. And that means that there are more questions that we can answer with a Bayesian approach that can’t be answered with a frequentist approach.

00:42:57.75 [Moe Kiss]: I really want to be a jerk and ask an example.

00:43:00.48 [Michael Kaminsky]: Great question. As an example of a type of analysis that I think is very difficult to do in a frequentist analysis, this comes from Richard McElrath. He’s an amazing statistics educator. A lot of my knowledge and even some of the explanations that I’m sharing on this podcast come from him. Everybody should check out his books, Statistical Rethinking, and then his online YouTube course. So he gave this example, which I think is really interesting. It’s technical and sort of in the weeds, so apologies for that. But it’s sort of the best example that I have coming to my mind. And this is an example of trying to analyze multi-level models. And so we could imagine, you see this a lot in like, education econometrics, where you have a model, where you have like the school level effects, and then you have classroom level effects, and then you have individual student level effects. And you’re trying to estimate, okay, if we teach phonics versus if we don’t teach phonics, like, how does that happen? And there’s often we wanna estimate like all of those different effects, student level, classroom level, school level. And in frequentist approaches, it can be often very difficult to estimate all of those simultaneously. Frequentists invented this machinery to estimate these multi-level models called Munlock machines. They’re weird, and again, it requires a bunch of assumptions under the hood, and it gives okay results. It works. But the Bayesian approach, we just simulate the whole thing. We simulate the students, we simulate the classrooms, we simulate the schools, and we can estimate all of those parameters jointly no problem. zero issue at all. This is in chapter 12 of statistical rethinking on multi-level models. Richard McElweth goes into the details of exactly how this works, and we can just show there’s no real straightforward way to estimate all of these different parameters simultaneously with a quote unquote, frequentist approach. We simulate the whole thing, use a lot more computing power to get there. It takes 30 minutes to run as opposed to 30 seconds, but we get the answer that we want on the other side. That’s one example, and there are a bunch of others of just questions that aren’t even really well formulated in a quote-unquote frequentist analysis. A frequentist would be like, that question doesn’t make sense to me, but in a Bayesian approach, we can just write it down. We can say, okay, look, we’re just going to have this model and we’re going to simulate and we’re going to get a result on the other side. Another good example that Moee flagged while we were on break is very small data. So another thing that Professor Richard McKellar likes to say is that the minimum sample size of a Bayesian analysis is zero samples. You can do a totally reasonable Bayesian analysis with zero samples with no data at all. We’re just going to simulate and we’re going to see what happens. If I have these beliefs, this is what the implications of those beliefs are. That is a totally reasonable Bayesian analysis. It’s no data at all. We’re just going to simulate from our beliefs from the papers that we’ve read. If this is true, if this is true, if this is true, if this is the structural model of how the world works, this is what the implications are. That’s a very reasonable thing to do as a Bayesian. If you have one data point, you can see how well that aligns with that. If you have four data points, you can see how well those four data points align with that analysis, and you can move from there. But the Bayesian approach allows you to really, or actually encourages, which is the thing that I really like. It encourages you. to think from the physical system or the biological system or what a Bayesian would call the data generating process. How does the world work such that it would generate these data and then work from there, as opposed to starting from a dataset and being like, how can I run a regression that will get a statistically significant p-value, which I think is the wrong way to do science. You want to start from the science, what do we know about the world, compare the theory to the data, as opposed to being like, oh, I found this dataset, how can I get a statistically significant p-value out of it?

00:46:50.27 [Tim Wilson]: But that does seem like that is a heavy lift when working with the business counterparts. I mean, it’s the battle I feel like we’ve been fighting for 20 years and that we have taught the world of business that the data is this objective quantitative thing and you just have to pick the right model as opposed to saying, no, no, no, you should be thinking about your beliefs, your expectations, your knowledge, what you’ve seen, what you’ve theorized. You should do all of that first and then see how well the data matches up with that and then adjusted, right? I mean, Moe, it kind of gets back, I think, to your question earlier when you were bringing up, like, working with a business user, all of a sudden, you’re going to them saying, I need to tap into your knowledge. And they are saying, why? Like, my knowledge, I’m just a marketer. Why can’t you just have the day to give me the answer? Which I think, Michael, is where you were just landing up because that’s absolute lunacy if you actually step back and think about it because you’re leaving information on the cutting room floor.

00:48:09.91 [Michael Kaminsky]: I hear both sides. I think I talk to a lot of business people who complain about their data science team and they’re like, why does the data science team never ask me about this section of the business that I know everything about? They come in with some presentation that has a bunch of results that clearly don’t make any sense. They should have come and talked to me. I think you get both sides. I do think that there have been some executives that have been trained into the cult of statistical significance, unfortunately, and they want to turn off their brain and just allow the data to speak to them. But I actually think that if you look at especially really successful executives in practice, they don’t do that. They actually are really carefully weighing the different pieces of evidence that they have. If they see a fourth-year analyst come in with some presentation and some analysis, they’re like, that doesn’t match my expectations at all. They’re going to poke a bunch of holes in it very quickly. And that again is like the sort of Bayesian process at work here, being like, this analysis doesn’t match all of my beliefs. Like what could be going wrong with it? And sometimes analysts get frustrated by that and they’re like, oh, you know, this executive doesn’t listen to the data. But actually in my experience is that like nine times out of 10, the executives are right. And it’s the analysts that like didn’t understand something or did the analysis wrong or missed some crucial assumptions. And I think that that’s actually like a reasonable process.

00:49:33.40 [Moe Kiss]: Can I walk through a real situation that is bubbling in my mind? Okay. So you have a new, a product team. They want to build a new feature. We have all been like auto trained.

00:49:48.29 [Tim Wilson]: Feel free to use specific names if you want.

00:49:50.12 [Moe Kiss]: We’re not recording this, Moe. Go ahead. The best way to do that is always through an AB test because that’s the gold standard and we want to have as much confidence as possible in the new feature yada yada yada. What I’m picking up from this conversation is that you could potentially, even if you haven’t built or yet rolled out the feature, you could use a Bayesian approach to have a better understanding to how your users might respond to it. That might help you then decide whether you should do feature A or feature B or feature C. I feel like working with product folk, they still come back to the like, we need an experiment, it has to be done through an experiment. And the thing that I’m kind of reconciling or like rolling around in my head right now is you could get to a decision a lot faster with a different approach. That might be good enough, but then is the like, let’s say we don’t have any data because the feature hasn’t been built. All your expectations about that feature are going to be based on previous features and how they performed. Are you not just going to potentially then end up with a very average assumption of how it might perform, which is potentially not reflective of a new feature?

00:51:12.80 [Michael Kaminsky]: Okay, so I think this is a really interesting topic and there’s a bunch of different directions that we can go. And I would say that like everything you’re describing sounds very reasonable. And what I would say is that you should design your decision-making approach based on what you’re trying to achieve as a business. And this again is another place where I think Bayjens do really well. Bayjens care a lot about decision-making, right? How do we make good decisions and how do we weigh the trade-offs of different decisions? And this again is one of their obsessed with like uncertainty. And this is one of the reasons is because you can take the full distribution of your estimated parameters and then plug that through some decision-making process and then evaluate on the other side. Like how much would we regret making a wrong decision, right? How much profit is going to generate on average, and the framework allows you to do that in a really nice and convenient way. That’s one nice thing. If I was giving advice to you or to this product team, I would say, set up your decision-making process to optimize for whatever it is the thing that you care about. You could imagine a product team saying, look, all that we care about is not making a huge mistake. We’re going to design tests that are only going to rule out huge mistakes. We’re probably going to release a bunch of stuff that isn’t actually better than the old stuff. That’s fine. We’re fine with that. We’re going to design our decision-making procedure. And again, we can set up our Bayesian framework to do this so that we run a lot of very small tests. And we’re going to power these tests just to be able to rule out the very bad errors. And everything else is just going to go through. you could have a different decision-making procedure that goes the opposite directions. We only care about winners. Everything that isn’t a huge winner, we’re going to throw out. You can flexibly design your statistical analysis of these different experiments that you’re going to run to optimize for whatever it is the thing that you care about. And that’s all, again, I think totally reasonable. We don’t have to have like the perfect A-B test 50-50 split statistically significant. We can do a bunch of other stuff that isn’t bad and still have a very reasonable business outcome. But we need to use our like business judgment to decide what that is. And then we can make the statistics sort of do whatever we want. Another good example of where Bayesian statistics can be a nice add-on is we know in AB tests, there’s this problem of the winner’s curse, which is to say that businesses that run lots of AB tests will get a bunch of winners that actually aren’t very good. There’s a bunch of reasons why this happens, but it has to do with Eventually, if you run enough tests and you only select the statistically significant ones, a bunch of those aren’t going to be actual true winners. Again, because of false positive rates, this should be fairly intuitive. You can apply a Bayesian prior to your test results to account for that. We’re going to actually make sure that tests get even further. There’s even more signal than what would be implied by a normal traditional one-off AB test to try to account for the winner’s curse. That’s, again, very reasonable thing to do. You can make that as strong or as weak as you want, but it allows you to take your information from previous tests, from your own business judgment, from the cost of releasing something bad, and then pull that together into one single decision. The Bayesian approach gives you a nice framework for being able to do that.

00:54:29.48 [Michael Helbling]: All right. We do have to start to wrap up. Who could have predicted that that would happen? Uh, this is good. Really good. I’m, I’m, my mind is going a lot faster than it was when we started. And so I’m thinking about a lot of different applications. So this is awesome. Thank you, Michael. And now it’s time for a quick break with our friend, Michael Kominsky. Think of that. From ReCast, they are the media mix modeling and GLIF platform, helping teams forecast accurately and make better decisions. I can’t wait to see what you have to share with us. This time was, which is a marketing science bite-sized lesson to help us all measure smarter. Take it away, Michael. And this is pre-recorded, Michael.

00:55:20.27 [Michael Kaminsky]: Causality is a complex topic with deep philosophical roots. Determining or proving causality is notoriously difficult, and in the context of analytics, there’s lots of confusion about what we mean by causality at all and how to determine if we found it. Because of the correlation versus causation problem, simply being able to predict what will happen in some system is not enough to demonstrate causality. That’s because if the system is stable, you can use correlations to make very good predictions. So predictions alone are not good indications of a good causal model. To demonstrate that our model is causal, we have to be able to forecast accurately when there’s an intervention. This is the core of the philosophy of causality in the sciences. In the chemistry lab, if we can consistently predict what will happen when mixing different compounds together, we likely have a good understanding of the underlying causal mechanics. And in business, we should think the same way. We can demonstrate that we understand causality when we can intervene on some system and still predict what will happen. By combining research and analysis with taking action, we can validate our understanding of causality and actually make our business better at the same time. Remember, you can’t demonstrate causality from the armchair. You have to make a falsifiable prediction and then actually go out in the world and take action.

00:56:26.86 [Michael Helbling]: All right, thanks, Michael. And for those who haven’t heard, our friends at ReCast just launched their new incrementality testing platform, GeoLift, by ReCast. It’s a simple, powerful way for marketing and data teams to measure the true impact of their advertising spend. And even better, you can use it completely free for six months. Just visit getrecast.com slash geolift. That’s getrecast.com slash geolift to start your trial today. All right. Well, one of the things we love to do on the show is do a last call, something that we shared might be of interest to our audience. Michael, you’re our guest. Do you have a last call you’d like to share? Sure.

00:57:08.35 [Michael Kaminsky]: I’m reading this book currently called The Maniac by Benjamin Labotute. It’s very bizarre, but maybe of interest to people who listen to this podcast. It’s sort of a biography slash you know, fictionalized biography of Johnny von Neumann, who was one of the original inventors of the computer, spent a lot of time not necessarily working on cracking the enema code, but working with the scientists that were doing that, developed a lot of amazing science and mathematics, and then also sort of a meditation on AI and sort of the trajectory of the world as computing power explodes. So very sort of bizarre literary fiction biography thing, but if that sounds good to you, I highly recommend checking it out. out.

00:57:53.73 [Michael Helbling]: Very cool. All right, Moe, what about you? What’s your last call?

00:57:59.44 [Moe Kiss]: I have two, but they are both very show related, so that feels appropriate. The first one is Cassie Kozyrkov has an article, Statistics for People in a Hari. It leans very heavily on the frequent aside. But the favorite thing in my life at the moment is that she also does an audio version where she reads out the article she’s written, which I adore because she is a very witty uh human and so getting to hear her read it you can definitely appreciate the wit a bit more and but also just on the enigma code if you want to read something for fun it is not a science book it is i think they call it historical fiction so it’s based largely on a lot of historical events but it’s totally fiction, with maybe some characters inspired by people in history. One that I really enjoyed was called The Rose Code, which was about the enigma and the women who worked at Bletchley Park, the author is Kate Quinn. And she does these amazing historical fiction books that are just actually enjoyable to read, but you get to know, I don’t know, a little bit about a topic or a time in history that you wouldn’t have known about otherwise. And so those are my two for today.

00:59:06.85 [Tim Wilson]: Over to you, Tim. I’m going to go with two as well. These are both a couple of posts from people who have been on this show in the past now that I realize that both the little bit cranky, both kind of spitting into the wind or screaming into the void. Yeah, which resonated with me. So one, Juliana Jackson, her sub-stack, she had a piece called how Silicon Valley’s hype machine continuously creates solutions nobody is asking for and has a lot of good history of past places and I think just makes some really spot-on points. And then even in Juliana, I would say kind of raging against the machine is kind of on brand. The other is a medium post by Eric Sandisham, who I think of as being a little bit kind of calmer and less likely to scream into the void. But he wrote a piece called Another Bullshit Job, the AI Transformation Officer exclamation point, which is also a pretty nice little takedown on that front. Couple of good reasons.

01:00:18.79 [Michael Helbling]: I would go as far as say, if you have transformation in your title, I’m nervous. I’m nervous for you.

01:00:27.49 [Tim Wilson]: Well, Michael is the Chief Podcast Transformation Officer. What’s your last call? Thanks so much for asking, Tim.

01:00:34.85 [Michael Helbling]: Wait, should I be nervous? Actually, it’s a good question. Should I be nervous? A lot of people are nervous about AI and whether it’s going to take our jobs or whatever. And so to that end, Cedric Chin over at Common Cog wrote something which I really like, and I’m passing out to everybody I run into who expresses those concerns. It’s called a letter to a young person worrying about AI. And it’s an excellent read as well as being Bayesian in its approach. I realize now after the facts, which is cool tie into the show. Anyways, I highly recommend it because I think we’re all having that conversation about where’s AI going? What’s it going to be like in five years? Are we all going to work on robot repair and stuff like that in three, four months from now? Whatever the case may be. So that’s That’s a really good one. All right. As you’ve been listening, you’ve probably been thinking, oh, wow, this is a topic that I hadn’t ever thought about before or I want to learn more about. We’d love to hear from you. And the best way to do that is to reach out to us and you can do that on the measure slack chat group or on LinkedIn or by email at contact at analyticshour.io and Obviously, we’d love it if you left us a review or a rating on your favorite platform where you listen to podcasts. That’s something we always appreciate hearing from you as well. And if you’re a huge fan of the show and you want to show it to the rest of the world, we’ve got stickers. And so you can request that on analyticshour.io. Just go to our website and we will send you some in the mail. And we ship internationally. for no apparent reason, but we do. It will find Tim, Tim will get on a plane and he’ll bring them to you through rain, snow, sleet and hail. No, but anyways. And yeah, so feel free to reach out and ask for stickers. Okay. Michael Kaminski, thank you so much. This has been outstanding. Appreciate it. Thanks for having me. This is a blast. It’s been such a pleasure. So thanks again for coming back on the show. We really appreciate it. Keep up the great work. We read all the stuff you put out and then we steal it and we bring it into the show as last calls and things like that. So big help to us as well out there. And I think I speak for both of my co-hosts because I have a prior about that. Moe and Tim, when I say, No matter what your process, frequentist or Bayesian, keep analyzing.

01:03:15.16 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at @analyticshour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Grohurst. Those smart guys wanted to fit in, so they made up a term called analytics.

01:03:37.51 [Charles Barkley]: Analytics don’t work. Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition.

01:03:53.11 [Moe Kiss]: I walk to work. I walk to kids. I mean, I can’t listen when I have kids. Or I’m like prepping dinner.

01:03:59.24 [Tim Wilson]: Interesting. You have a Bluetooth speaker in the shower? Or are you not?

01:04:06.77 [Moe Kiss]: I have one in the bathroom. I was listening to ABC News Radio in the morning and I’m like, I should use this time more productively. I don’t get that much value from the news. Yeah, that is incredibly optimized.

01:04:20.94 [Michael Helbling]: Yeah. Why she’s a co-host on the analyst power hour. That’s right. I see how you recruited her.

01:04:33.53 [Moe Kiss]: I mean, there are many ways I let everybody down constantly.

01:04:38.08 [Michael Helbling]: So, you know. Meanwhile, I’m in the kitchen. Do you believe in life after love?

01:04:46.58 [Moe Kiss]: Well, last night I did make dinner to the dulcet tones of a song called Chicken Banana. I highly encourage you to go listen because it is possibly the first song. I don’t think I’m going to have time for that.

01:04:58.72 [Michael Helbling]: Could you maybe just do it quick? I mean, go ahead and knock it out. Go ahead. You can make it your last call if you want.

01:05:05.66 [Moe Kiss]: I mean, I should get the video of Harry, but it basically is like, uh, dance music. I don’t know why the kids like it, but it goes, chicken, banana, chicken, banana, to like high tech music.

01:05:18.88 [Michael Helbling]: That’s why right there.

01:05:20.44 [Moe Kiss]: Yeah. It’s, it’s a thing. Yeah.

01:05:23.29 [Michael Helbling]: Something about the word chicken delights children.

01:05:27.13 [Michael Kaminsky]: Really? Wait, so actually, are we doing, is this video, do we do video or is this audio?

01:05:33.63 [Tim Wilson]: Yeah, so we do the recording. You’re getting ahead of yourself.

01:05:36.38 [Michael Helbling]: Yeah. Oh, he, you know, we might have changed the checklist to the last one.

01:05:40.40 [Moe Kiss]: Well, he’s very sweaty.

01:05:41.54 [Michael Helbling]: Yeah, I just actually want to confirm. We won’t release the entire episode.

01:05:44.51 [Moe Kiss]: Do I have a massive lag?

01:05:46.79 [Michael Helbling]: But we typically will. do shorts, so there’ll be snippets that we’ll take out and those we’ll have video, but the whole show we do not do video.

01:05:56.93 [Michael Kaminsky]: All right, I’m gonna keep the shirt on for now, but I might beg off.

01:05:59.65 [Michael Helbling]: Yeah, that’s fine. It’s fine. Whatever you have to, and we won’t… I won’t be fully shirtless. I have an under shirt on, but I get it. That’s fine. All right, so there’s that.

01:06:10.47 [Michael Kaminsky]: I try to use the collared shirt for public reasons.

01:06:12.73 [Moe Kiss]: Oh, that is not where I thought this was going.

01:06:23.00 [Tim Wilson]: Rock Flag and Bayesian Enigmas!