[music]

0:00:05.8 Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language.

0:00:14.2 Michael Helbling: Hi, everybody. Welcome. It’s the Analytics Power Hour, and this is episode 243. Back in 1919, the DuPont Company had a secret room where they would put up all of their charts and graphs on big boards, and the managers and leaders of the company had wheels on their chairs, and they would move around the room as they discussed the data and the decision to be made from it. The room and its use were so important they didn’t openly talk about it until 1950, and that room was finally replaced by a booklet in the 1970s. And I only mention that ’cause sometimes it’s easy to think that all the ideas about being data-driven and using data to drive business decisions are a product of recent times. But maybe we have a lot to learn from people who came before us if we would just pay attention. Speaking of people I like to learn from, let me introduce my two co-hosts, Julie Hoyer. Manager of Analytics at Further, welcome.

0:01:12.9 Julie Hoyer: Hi, fancy seeing you here.

0:01:15.4 MH: Yeah, I’m glad to be back on a show with you. And of course, I had to do titles this time because, Tim Wilson, co-founder and head of solutions at Facts and Feelings, nice of you to finally get a job again.

[laughter]

0:01:31.3 Tim Wilson: Well, it was time. It was time.

0:01:34.6 MH: It was time. No, that’s so exciting, and this obviously launched back in March, but we’re excited to see the launch of Facts and Feelings and finally see that happen, and I’m Michael Helbling, I’m the managing partner here at Stacked Analytics. And I’m really excited about our guest today. Cedric Chen is the founder of Commoncog. It’s a publication dedicated to understanding business expertise and how to accelerate it. He’s also held engineering and marketing leadership roles at two other companies. And today he’s our guest. Welcome to the show, Cedric.

0:02:02.6 Cedric Chin: Thank you for having me on.

0:02:03.9 MH: It’s honestly really exciting to have you on. There is a series of events that happened a couple months ago because you wrote an article about becoming data-driven from first principles, which I picked up and read in an evening. And then it just started showing up in every place I look at content. Everyone was talking about it all of a sudden.

0:02:28.7 CC: Yay.

0:02:29.7 MH: And I pushed it over to… Yeah, exactly. That’s what… That was… And so I posted it in our podcast Slack group and I was like, “We need to talk to this person, we’ve got to do this show with him,” so this is awesome to see this finally come together, so thank you. But, yeah, what are you doing with this… With… What are you doing? Like why are you writing about this stuff?

[laughter]

0:02:55.1 CC: Well, I… Yeah.

0:02:55.5 TW: That is that is the mark of like a polished interviewer right there.

0:03:00.0 MH: Yeah, what… What…

0:03:02.1 CC: I listened to a number of your podcast episodes before coming on, and what I love is the chemistry between all of you, so it’s amazing to sort of see this live. I was wondering how it would play off because is it editing? No, no, no. It’s all real. For everybody who’s listening, this is all real.

0:03:17.4 MH: It’s all real.

0:03:19.8 CC: Yeah. I guess I should sort of start it off with, so the marketing role and leadership role that I held, I actually just ran content marketing, but it was for a business intelligence visualization company, right? And during that period, it was like a two-year, slightly over two years that I spent at that company. And at the end of it, I ran like a repositioning exercise that doubled their annual recurring revenue. It’s a bootstrap company, it’s pretty small. But when I left, I remember sort of thinking to myself, I’ve dealt with all the these customers, maybe not directly, but I’ve listened to a heck of a number of… A lot of sales calls, because your goal… Your job in marketing is to really understand your customer, right? And a lot of the customers didn’t seem to be particularly data-driven. And I’ve always wanted to know, what does it mean to be data-driven, right? Like, I could not find, in preparation for that content marketing job, any good information about how to actually be data-driven in business operations. And I think one of the most important or like weird sort of things that came out of that was, I think actually the vast majority of companies, even the vast majority of the people who buy that software, they’re just not data-driven, right? So a couple of months after I left, Colin Bryar and… Reached out.

0:04:35.4 CC: So Colin Bryar is one of the early Amazon executives, and he had just written a book called Working Backwards, which explains the mechanisms of how Amazon does what it does. And chapter six in that book is a book about metrics, or rather about how Amazon uses metrics and in particular, there’s this practice that’s described in the book called the Weekly Business Review, right? And he was like, okay, you seem to get what the book is trying to say because I had written like a bunch of summaries of the book. We are running a consulting company now, and we’re helping large companies put to practice many of the Amazon practices described in the book. And the thing that a lot of people struggle with is the chapter on metrics. So we want to explicate, because it turns out that whatever is described in chapter six of the book about the weekly business review, it’s not enough to actually put that to practice. We want to do more work to explicate, like, how do you put the weekly business review to practice? Would you come on and do this consulting project with us, right? Like, we’ll pay you, and we’ll also build software, and you help to check this, the software, write tests for the software, basically help with using the software.

0:05:36.2 CC: And to sort of set a context for why the WBR is so tricky, every Wednesday morning, Amazon leadership sits down and they go through 400 to 500 metrics in one hour. And I was… The question I had to ask was like, “Really? Exactly one hour?” He said, “Yeah, yeah, exactly one hour, except for the holiday season where it’s 90 minutes.” And they’re always on the dot, right? 400 to 500. I was like, “How?” So yeah, sign me up. I want to learn how to be data-driven, I want to learn how to do this from you. And hopefully I can go down a path where I finally understand how to become data-driven, because I’ve read a whole bunch of books and none of them actually explain like linear analytics, my bugbear is… I tried reading linear analytics and they do not actually explain how to become data-driven, which why? So long story short. Near the end of that project, Colin Bryar was… He kept saying things that I didn’t understand during the time that I was working with him, like, for example, the WBR is a process control tool. If you have a problem with X function, like hiring or whatever, you should just subject it to process control, which means that you need to put it into the WBR. And at the end of the time working with him he just sort of said like, “Oh, you should go read Donald Wheeler.” There’s this book called Understanding Variation.

0:06:46.3 CC: And so I read it, it took me maybe like two months to read it, but it doesn’t take that long, it’s literally just a 150-page book, you can finish it in an afternoon. And I… It blew my mind, I realized that a lot of the ideas, in the WBR are not new. They came from this field, this really old field called Statistical Process Control that primarily lives on, I think, in programs now known as Total Quality Management or Six Sigma, they came into Amazon in the early 2000s, and then sort of spread and turn into the WBR. So I realized that there was this whole body of work that had been created roughly around the time of World War II and shortly after World War II that explains to you how to actually become data-driven. And it’s explained in a really simple way because it was written for production managers and factory workers and factory foreman, right? You don’t want to be sophisticated and talk about extremely complicated data concepts. You just want to like sort of take them by the hand and tell them how to use data in a way that improves the operations, right? So that’s the context in which I came to. And then I spend… One big thing about me is that I don’t feel comfortable writing about things that I haven’t tested. So I took… I then took like about a year to put it to practice, and now I feel confident enough to write about the ideas that I’ve been discovering.

0:08:07.9 CC: And we’ll get… We’ll probably talk more about like how I became even more dissatisfied with the current state of analytics or data content on the internet as I was going down this rabbit hole and putting these ideas into practice and finding out that it worked, it really worked.

0:08:21.9 MH: So can you give, I mean, I’m sure there are, some of our listeners are sort of familiar with the [0:08:27.2] ____, I mean anybody who’s Six Sigma knows SPC and hopefully kind of inside out. But can you… Many are not, many have come up through other areas, so can you give kind of the quick overview of kinda what the core sort of statistical process control mindset is and kind of how that works and what’s at its core?

0:08:47.1 CC: So the core of SPC is understanding of variation, and it sounds a bit dumb, right? Like, what does understanding of variation actually mean? But one of the core ideas that I found incredibly friendly or like amazing about the whole SPC body of work is that like the biggest problem that most people have, most operational people have with using data is variation, right? Charts go up and down, they wiggle wildly, they are very sort of confusing to look at, right? And sort of one of the feelings that I wanted to call and early on in the piece was sort of just saying something that I think is quite true and that most people have, which is, have you ever looked at a business metric? And then it’s… And you think to yourself like, so what, like, is it good or bad? Because it’s constantly wiggling, right? And the SPC pioneers, in particular, this statistician called W. Edwards Deming, he had this observation that the main obstacle to becoming data-driven is that you can’t deal with variation.

0:09:46.0 CC: Most human beings can’t look at the chart and go like, “Okay, things are going well or things are going badly.” So what SPC is, is a small set of tools to help deal with variation, right? And the sort of technique that I call out in the piece, but actually fairly late in the piece, is this technique called, XmR charts, which are a form of process behavior chart, I mean, there’s many names over the years, these charts are like 90 years old at this point. In the past they’ve been called control charts or Shewhart charts. But basically what they are is that they allow you to characterize variation and to quickly detect, there’s a small set of rules, allow you to quickly detect if the variation you’re looking at is special or if it’s routine. And I think the core benefit of XmR charts is that I think most people, when you sort of show them a chart, right? You can tell them like, variation is sting, and they go like, oh, yeah, yeah, variation is sting. But then when you present to them a chart, right? They somehow forget that variation is sting, right? People forget that numbers are expected to wiggle around a line and it’s not just a clean straight line, right? And then they become confused. But if you show them a chart with two lines above and below the wiggling, right? And then tell them like, Hey, this is normal. You’re expected to see some number between these two lines, because that’s what routine variation means.

0:11:02.7 CC: That’s what variation means. When you weigh yourself on a weighing scale to check your weight, your weight is not going to stay steady in one point, it’s gonna wiggle based on like, fluctuations in a gajillion small tiny factors that you don’t know, right? And so similarly, any business metric you look at will wiggle between two lines. And once you give them the two lines, they sort of internalize like, oh, every metric that I’m looking at should wiggle naturally. And that’s just a fact of life, right? And what the XmR chart does is it unlocks the sort of, the way that I’ve been thinking about it is that it unlocks something that we are all naturally equipped to do, which is trial and error, right? If you think about how we learn things, right? Like for example, if we go climb a tree, and then we fall off the tree, and then we’re like, oh, okay, I learned not to do that. Hopefully, the tree’s not very high. So you don’t die when you fall off. But then you climb a tree, you fall off, you’re like, okay, I’m not gonna do that again. Right? So you’ve learned from very clear feedback that, hey, I need to do something different if I want to go and climb that tree, right? Humans are very good at trial and error because we learn through trial and error.

0:12:00.4 CC: But if you don’t have a way to separate signal from noise to be able to tell like, Hey, I can just ignore this routine variation, or, Hey, there’s something going on here that’s not normal, I need to investigate, because there’s some pattern in the data, there’s like the change, the wiggle in the data is larger than historical trends would indicate, then you’re not actually able to do good trial and error because sometimes you’ll be investigating when there’s nothing to investigate, and sometimes you won’t investigate when there’s actually something that’s worth investigating, right? And so all the XmR chart does is it re-enables humans to do this thing that we are born to do, which is trial and error. And I can go into like a lot of the problems of the XmR chart there. It is not a magical tool, but at the minimum it has a amazing track record of helping operational people, not data people, sort of wrap their heads around like, oh, okay, actually data’s not something that I can be scared of that I should be scared of, like there is a way to approach data that I can use to do trial and error, which everybody is familiar with, right?

[music]

0:12:57.0 MH: It’s time to step away from the show for a quick word about Piwik PRO. Tim, tell us about it.

0:13:03.5 TW: Well, Piwik PRO has really exploded in popularity and keeps adding new functionality.

0:13:09.6 MH: They sure have. They’ve got an easy-to-use interface, a full set of features with capabilities like custom reports, enhanced e-commerce tracking, and a customer data platform.

0:13:20.5 TW: We love running Piwik pro’s free plan on the podcast website, but they also have a paid plan that adds scale and some additional features.

0:13:28.0 MH: Yeah, head over to piwik.pro and check them out for yourself. You can get started with their free plan. That’s piwik.pro. And now let’s get back to the show.

0:13:40.4 TW: I used to give a talk about kind of hitting on the same variability where it kind of point out that when we see the most recent data point, we all of a sudden say it went up or it went down. And then I say, but if we go two more weeks out and then we point to where that data point is, it just looks like part of the natural variation, there’s just this human nature to say it went up. Why? Or it went down. Why? And if we just fast-forwarded, we’d say, oh, well, then it just looks like kind of, part of the natural variation, but we keep asking that question. I also have to throw in, ’cause I… I mean I literally have completely dismissed anyone who has Deming’s and God we trust all others bring data. There’s somebody who has it as like their LinkedIn banner and I just… I want to be like, wow, like that’s, you’re dead to me. And people will occasionally use it in presentations. And it’s interesting because you use that quote in the piece and basically my takeaway went from that was, yeah, people are wielding that very… And that’s been my beef, you made it a little more clear that when people say that, it’s kind of like they make this silly shortcut that like, oh, well, that means if I bring data, then you will trust me, and it’s like, well no wait, you just…

0:14:54.9 TW: He wasn’t saying that if you show up armed with a chart, then whatever you say is the gospel and that in God we trust because it’s God and trust and truth is in there, I think it also sets up people treating that the data is the truth. And, to me when you talk, when we say variability, that’s kind of analogous to uncertainty. Like the… It’s always going to have variability and uncertainty in it, and that one quote from Deming, like absolutely acknowledge like all the stuff that he did, but that’s the one that analysts and marketers like pull up and love to like wave around like they are… And declare themselves data-driven because they have that one Deming quote. So sorry, you triggered me, I think it’s just a, it’s just a reaction that when Deming comes up, I’m like, oh, gotta get in my rant there.

0:15:49.1 MH: But I feel like it’s a product of the digital age too, because in digital we have told ourselves that we can measure anything, and as a result of that, we can learn the truth, right? And I think in your article you kind of talk about this difference between gaining knowledge versus gaining truth, and I really like that juxtaposition. It’s a super powerful way of positioning to help people understand like what we’re actually after here when we’re doing any kind of analysis, it’s actually to learn things about the system and the reason why variation may be occurring. And so anyways, I don’t mean to steal thunder for later on, but I love that a lot.

0:16:32.6 CC: Deming is, I’m sighing because I’ve been struggling with this piece on Deming for a while now, and I sort of flipped back and forth between, I’m, I was supposed to finish the piece on the Amazon WBR by the time for this podcast, but that was also difficult to write. But Deming is particularly difficult to write because the blast radius of the man is huge, right? If you realize all the things he’s affected, it is huge. The problem is that his ideas are incredibly difficult to… They’re very controversial, right? From a simple single thing, which I think I’ve tried to communicate, hopefully successfully in the article, right? Understanding variation is important. Deming has built an entire coherent philosophy of business and operational rigor on just understanding a variation. And maybe we can’t get into it in the podcast, but like to sort of give you a preview of some of the things that he says as a result of variation, right?

0:17:24.3 CC: If you understand that there is routine variation, which comes from just… Not just from systems, but also from people acting in systems, right? The performance of your sales organization is a complex interplay of systems and technology and processes and people and the training you give to people, right? Which means that… I think the sort of more acceptable sort of implication of that, which I put in the essay is that you can’t go at a sales person and yell at them for say, oh, 12%… Your performance is 12% below what it was last, last month. You can’t say that because you need to actually take a look and characterize overall performance to sort of say, is that routine variation or not? Right? Because if it is routine variation and you’re blaming the sales person for bad performance, right? The sales person becomes demotivated because it’s not actually his fault. It’s a complex interplay of the environment and the system and the training he’s received and his own performance, right? Instead, what understanding variation, and again, this is quite… This is not very controversial, is that the implication once you internalize that routine variation is the thing is that you realize, oh, I need to change it systemically, which means I need to fundamentally think about the way that I’m training my salespeople, or fundamentally think about the way that marketing qualified leads are being sent to the sales team, right?

0:18:32.8 CC: Because I can’t just change or blame one person for their performance. I need to shift the entire system. That is the kind of thinking that looking at XmR charts every week will and have a long history, a track record of enabling. So that’s the uncontroversial bit, the controversial bit which I never talked about is that, and therefore you cannot rank people. And therefore if you do performance management of people, you need to understand that there’s only three buckets that people can hold into.

0:19:00.1 CC: Stack ranking is bullshit, right? Because when you test people, if you test today and I test tomorrow and I test, maybe I give you an exam or I test your performance like three months from now, it could be routine variation, right? So that’s one bucket. And if you’re inside routine variation, there should not be any ranking because you are just within the expected performance.

0:19:17.7 MH: Within the range. Yeah.

0:19:19.1 CC: The other two buckets is below routine variation and above routine variation, right? And if above routine variation, yes, you should be rewarded and we should study you to sort of see what you’re doing differently so that we can then communicate what you’re doing differently to the rest of the org, right? And of course, below routine variation, then we can punish you. But as a result of this, anytime you see somebody talking about stack ranking, right? You’d be like, this does not map how reality works. And you only are able to do this if you truly internalize variation. Right? And that’s just like one of the radical ideas, right? He goes on, he’s like, oh, since understanding a variation and then understanding all the root cause, I’m probably getting ahead of myself a bit here.

0:19:57.0 CC: You shouldn’t have OKRs because the way to hit OKRs, the way to hit OKRs is to not focus on the OKR, but to focus on the process that produces the OKRs, right? So you need to characterize performance of the process and then figure out controllable, well, Amazon calls it controllable info metrics, but Deming was just talking about root cause is figuring out what are the actual control factors for that process such that you can modify process behavior such that you can then eventually hit OKRs, but you shouldn’t incentivize people based on whether they can hit OKRs or not. You should incentivize them based on actual things that they can control. That you know trial and error affects the outcome that you want. So these are all incredible.

0:20:38.8 MH: But Amazon does stack rank…

0:20:42.4 CC: Yes.

0:20:42.9 MH: Doesn’t Amazon, I mean, Amazon fairly famously stack ranks people.

0:20:44.1 CC: Yes.

0:20:45.0 MH: Right. Okay.

0:20:45.5 CC: This is the other difficult thing about talking about Deming, right? If you read all just the Deming consultants, they will go like, they will sound a bit like communists basically. True Deming has never been tried.

0:20:58.5 MH: I mean, if you want to find problematic people, go look up Ronald Fisher and the stuff that he generated, and you talk about problematic for somebody who brought some principles, but those White guys back in the middle of the 20th century, you don’t know what you’re getting into.

0:21:15.9 TW: It was an exciting time. I’m with Deming on OKRs though. Geez. I like that, actually.

0:21:24.0 JH: On the OKR part though, you talked about the OKRs in the sense of, you’re talking about the optimization worldview compared to the process worldview. Could you kind of describe the difference between those two?

0:21:36.8 CC: Right. How do I set this up? No, no, no.

0:21:41.1 MH: Julie being the one person who’s working in an OKR environment currently, so.

0:21:45.6 CC: We have wade into incredibly controversial topics, and I don’t think we should go down talking more about OKRs because the argument that Deming makes is very…

0:21:53.5 MH: That was a pass.

0:21:55.2 CC: Complex.

[laughter]

0:21:57.6 CC: Well, and I guess to sort of address, sorry, before I talk about Julie’s question to sort address Tim’s observation, you are absolutely right. Amazon uses OKRs. In fact, many companies that use Demings ideas use OKRs, and I don’t know whether this is a plus or a minus, but the man’s impact is so large that you can just use parts of his philosophy and get great results without using all of his philosophy. That seems to be the case. If you trace down multiple businesses that use them and you realize that some of them do pick and choose, some of them do go like, okay, we use the whole statistical process control metrics, how to actually be data-driven bits of it, and it’s so powerful that they get incredible business success and then they don’t really care about the psychology bits, which Deming added later on in his career.

0:22:44.8 CC: Anyway, back to Julie’s observation. I’m so glad you brought that up because this is actually the thing that was the most mind blowing to me. I think that for the vast majority of people who want to become data-driven, right? Our default worldview is to think, okay, we need to optimize something. This is usually expressed in a funnel. Like when you learn marketing, you quickly learn the concept of a marketing funnel or a sales funnel, and then you sort of immediately sort of say, oh, if we want to hit our goal for the quarter, whether it’s like marketing goals or it’s sales goals, let’s go take a look at the funnel, I mean, and let’s go take a look at our existing conversion from each stage of the funnel and then ask ourselves how are we going to increase the conversion rate, the optimization rate? And it’s a very pervasive worldview. If you talk to people about data, at some point, somebody’s going to say, oh, why do you want to be data-driven?

0:23:39.0 CC: It’s just about making small, tiny changes, right? Small improvements to the performance of your business. Or it also shows up when you sort of hear people talking about, oh, okay, when we want to improve our business, let’s look for some metric and then let’s increase it by 10%. This is all the optimization worldview. And the problem with it is that it doesn’t give you a mechanism. So Deming’s big thing is, if you want somebody to do something, you need to tell them the goal. You need to give them a mechanism for accomplishing it, and then you need to tell them, give them a test so that they know when they’ve arrived, when they’ve accomplished it. It’s very common sensical, and the optimization worldview does not immediately afford you these three things, the whole Deming sort of approach to data gives you a different worldview, which is the worldview that Amazon uses, that Amazon has as I think in-built in it in most of its culture, at least according to Colin Bryar, and I believe that it still is true, although Amazon is so large at this point, you can probably find pockets where it’s no longer true.

0:24:44.7 CC: But the core worldview that Deming taught and that Amazon’s sort of imbued in its mechanisms is that, look, here’s a process. Everything in your business is a process. Your business itself is a process and a process may compose of multiple other processes. Your job when you’re given a stream of data thrown off by that process is twofold. You have to find the root causes that cause this stream of data to be what… To change. You need to figure out what controllable input metric, sorry, I’m using the word controllable input metric because that is the Amazon in-term, but let me use a more simple term. You need to find the root causes that cause this metric to fluctuate or to go up or to go down. And the way you do that is we are gonna give you just one data tool, which is the XmR chart, which is the ability to differentiate between exceptional variation and routine variation, signal versus noise. And every time you observe exceptional variation, you go investigate and you most likely find a root cause that you can then exploit. Maybe it’s a bad root cause, in which case you sort of change your process, your business process, so that it no longer becomes a factor or it’s a good root cause, in which case you want to go and start an experimental program to see how you can get more of it, right?

0:25:53.4 CC: So that’s observation. But then there’s also the experimental bit of it, which is that your job is to improve this process. So therefore this tool that separates routine from exceptional variation finally tells you if the changes that you’re making, the experiments that you’re running, the trial and error process that you’re running on the process is actually working or not. And this is the thing that was sort of saying, unlocks the ability to do trial and error, right? Because finally you’re able to look at a stream of numbers that goes up and down and go like, okay, the change has actually worked or the change has not worked. And the way this typically works is when you do an XmR chart, you do have to wait for a minimum of six extra data points, new data points after you’ve made the change to sort of see, hey, has it worked or not, right?

0:26:27.9 CC: And six is just the bare minimum. The limit lines in the XmR chart the lines above and below the wiggling data that you have, it’s six to eight. The limits begin to gel. It begins to solidify between 10 and 15, and it actually hardens between, there’s marginal benefit to get more than 15. So 15 to 20 is funny. You’re pretty sure it’s not going to change. And these are our estimates, right? But it’s good enough for you running a business process to be able to do trial and error because now you’re not sort of confused by like, hey, did the change that we made, did it actually work? So the core of the process control worldview, why it was so mind blowing to me was for the first time somebody had articulated, here’s how you become data-driven. All data is, it’s just a way for you to chase down the root causes of what causes your process to increase or decrease.

0:27:16.6 TW: And then you can then exploit that. And if you pursue that, you pursue that for every process in your company. You realize that every process in your company affects every other process in your company. And over time, what should emerge is a causal model of your entire business, at which point you can start running control… You can start doing continuous improvement because you can improve every aspect of your business, not just manufacturing, where it’s sort of been stuck in for decades, but any aspect of your business, whether it’s hiring or recruiting or sorry, recruiting, hiring, same thing, but engineering and marketing and sales. This single insight sort of just, oh, okay. It’s actually quite obvious and quite easy. And data is not just the realm of sophisticated professionals. Anybody can be data driven.

0:28:00.1 JH: ‘Cause I was really interested in part of your article when you were talking about, I don’t know if you tied it, I can’t remember if you tied it directly to this classic optimization worldview or not, but you were saying that people that don’t understand variation and are just looking to improve the number either end up, it was distorting the data or distorting, oh gosh, it was like distorting the goal. Was that it? You had the two things they can distort.

0:28:27.0 TW: Yeah, distorting the system and distorting the data.

0:28:29.7 JH: That’s what it was. The system and the data. And it’s interesting because I feel like that’s what I run into a lot, is people I work with that have worked with data forever that I would expect would talk about and appreciate variation more. Again, you said they kind of head nod that it exists, but then they go into application and I feel like we end up telling the story from the data for your client, your stakeholder, whatever, and they end up falling into that where they’re distorting the data to tell the story that sounds good. And it’s as if variation has gone out the window. And I guess for this being such a widespread conversation and technique, I’m just shocked how much I still see that every day.

0:29:18.8 TW: Well, I mean, ’cause part of me that, I mean, so SPC is very old, but there’s also, when it comes to a lot of the battles that I feel like Julie, you and I have fought, I mean, when you’re looking at Bayesian structured time series, it’s not, oh, just fancier and plain old SPC would be fine. It’s got the exact same concept of how can you account for more factors going in? And then you wind up with an interval which is representing the… Anytime there’s a confidence interval or a prediction interval or the error term and a regression, all of those are the concept of variability or noise. I mean, Nate Silver’s book from years ago, The Signal and the Noise, he was trying to understand it, but we’re working with people who kind of wanna have a very short-term view often and say, oh, this went up. I wanna celebrate it. And you’re like, if you celebrate that, then you’re gonna have the same on the back end when the noise drops you down. And there is times where it seems like there’s no, it’s just like the memory of a fruit fly, it’s just like… Or a goldfish, is a goldfish that have no memory. They’re like, this is great. Great. It’s like, no, no, don’t say great. Oh, the world has ended. Ah.

0:30:39.6 MH: Yeah.

0:30:39.6 JH: Yeah. You won’t get a wiggling chart and they won’t call out the dip in 0.3, but they’ll call out the rise in 0.2. I’m like, guys, what are you doing here? You can’t do that.

[laughter]

0:30:51.7 MH: Yeah, there’s costs to calling out. If a number goes down, and to the sales example, it’s down by 12%. People send people off on wild goose chases. Why are we down? And teams are off analyzing the data, looking for root causes where none exist. It’s the simple variation of the data. I call that data chaos. You create data chaos by not understanding that.

0:31:16.3 TW: So that’s the question. When I hear… The part of we have is understood process, and therefore then we’re using, basically it’s anomaly detection. You kind of lay out the different rules for saying, when is this outside of the norm? It goes more than X. It goes beyond the process control limits or more than X consecutive points that are above or below the challenge. Where I start to lock up is that Adobe Analytics, VERY, very famously, and the people who use Adobe Analytics, put anomaly detection in, and there was a big movement saying, this is now in essence, it was, oh, this is statistical process control applied. Now it’s gonna tell us when there’s an issue.

0:32:00.0 TW: The problem is that fundamentally, and you even called this out in the article, if something exceeds your process control limits, that’s expected for that to happen periodically. And the more metrics that you have, you say 400, say, wow, even if I set my interval like 99%, that means every week they’re gonna have four on average metrics that show up as being signal, but they’re not. You have such high volume that you’re still gonna have things pop up in the extremes that are still noise. And what you do is, so you can still wind up with that data chaos, right? If you look at too many metrics through a statistical process control lens, you kind of… That’s the statistics works out that way that you’re still gonna have things popping up that cross that check the anomaly, the outside of the limits criteria, even though they are still noise. ‘Cause you’ve ratcheted up the volume so much.

0:33:03.6 CC: Oh. That…

0:33:05.6 TW: Maybe.

[laughter]

0:33:06.2 CC: No, no, no. That doesn’t seem to be so, okay, so first things first. Amazon still uses XmR charts in specific parts of the company, but it’s not widespread. Like Colin was quite clear to me on this because like, not every part of Amazon is Six Sigma. What do you go train? Right? Primarily the logistics and the warehouse and the e-commerce parts are very rigorous. AWS from what I hear, is a bit more sloppy because like everything is up and through the roof there. Every experiment is success because they’re growing so big, like they’re growing so quickly.

[laughter]

0:33:37.6 MH: You don’t need to analyze when you’re winning.

0:33:39.2 CC: Yes. You don’t need to analyze when you’re winning. That’s a good one.

[laughter]

0:33:46.4 JH: Michael’s got all the quotes. I love it.

0:33:48.2 CC: Yeah.

0:33:48.8 MH: Yeah.

0:33:49.7 CC: But to sort of describe a bit like what happens in Amazon, the WBR is generated every Sunday night. So on Monday morning, the metrics owners come in, right? And they look at it, right? And you are expected… So even if you don’t use XmR charts, you are expected to understand what is routine and exceptional variation. And from what I’ve heard, because at this point enough X Amazonians have reached out to me from various parts of the company’s life. This is no longer as true as it was during Colin’s day.

0:34:17.7 CC: But every metrics owner is expected to have a deep seated feel of exceptional versus routine variation for their metric. On Tuesday, the departmental WBR happens, which is just with your department. And if there’s an exception, you need to go investigate and figure out what the root cause is, right? And XmR charts are, they are a way to bootstrap intuition, but the intuition is the thing, right? The intuition or whether it is exceptional or routine variation is the thing. It’s not actually the XmR chart says the magic, the XmR chart just forces people to understand, like especially normies who are not very good at data like me, right?

0:34:51.0 TW: But part of that intuition comes from the other thing you’ve talked about that is like through being forced to kind of think through what is the process, right?

0:34:58.0 CC: Yes.

0:35:00.2 TW: Presumably the, identifying the process and refining the process is the starting point is thought. And that thinking supports intuition, which then has a positive feedback loop. Like I loved that part. Okay.

0:35:12.3 CC: 100%. 100%. Like I should add, by the way, that I am not a data professional. I’m a business person, who is trying to get more data driven. And so lots of things that Colin said during the time that I was working with him was very confusing for me. Like for example, one of the things that he said was like, “You should never draw conclusions from the data. You should always start from a qualitative understanding of the business. And then you verify with data.” This is the classic like scientific understanding sort of thing, right? Which is that you don’t want to overfit or I guess that’s the machine learning term. The more scientific thing is that you don’t want to P-hack or like, you don’t want to conclude from the data based on like an existing sample set that we thought actually experimentally verifying in the future, right?

0:35:50.3 CC: That, hey, this causal relationship is a thing, right? So you have to understand like early Amazon like this kind of thinking was widespread in the organization, right? So you come in on Tuesday, you sit down with your departmental head, who also by the way, has a sense of routine versus exceptional variation because this is expected. You don’t get to sit in the WBA for understanding routine or exceptional variation because only exceptional variation is discussed, right? That’s how you get true 500 metrics in an hour. So then you come in and then when it’s your turn, you either say nothing to see here we are on track, right? Or routine variation, in which case everybody looks at the metric for one second and moves on to the next metric. ‘Cause it’s still important to look at it for one second because this is how you develop a sense of a like finger feel for what the data looks like when it’s normal and then you go move on until somebody says, “Okay, this is exceptional variation and either it was caused by this and we are doing these three things to try to see if we can solve it.”

0:36:41.0 CC: Or like, maybe it was a strike, maybe it was like, there was a crazy like hurricane in this state last week. In which case there’s nothing you can do or you will say, we don’t know and we’re still gonna find out. And sometimes you don’t know and that’s okay, right? And because the WBR law will be like, we don’t know yet, but in the future there may be new pieces of information or new weird patterns in the data that will give us new hypotheses to test. And that’s okay because it’s a complex adaptive, like it’s a very complex system, right? So XmR charts are to bootstrap intuition. But the really important thing, and this is the thing that I really struggle to express in the essay, right? I was trying to like tell you like, “Hey, there’s this magical tool that can help you bootstrap intuition or routine and exceptional variation.” But the important thing isn’t the tool because the tool has limits, which I did not get into the essay.

0:37:28.4 CC: There are real problems with the XmR chart, but I’m confident enough to recommend that because you can tell within seconds of plotting one, whether it works for your data or not. So I don’t actually have to talk about distributions probability. There are certain distributions that XmR charts don’t work for, but that’s like, it’s common sense. You just plot it, you’re like, “Ah, this is useless.” So either you throw it away. But the thing I wanted to communicate was it’s actually understanding of routine and exceptional variation that is important because if you have this org wide understanding of routine versus exceptional variation, you’re not gonna celebrate individual data points. You are only gonna celebrate when the overall pattern of the process has improved.

0:38:01.4 CC: And this also opens you up to trial and error, which then allows you to sort of start looking at your entire business as a causal system traffic on one end. And this is one thing that I should point out, like the WBR is organized in a way that is causal traffic on one end, then it flows through in financial metrics out the other end. So and the order stays the same week in, week out. Sure. Some metrics are removed because it turns out it’s no longer predictive. Some metrics are added back in because it turns out that there’s new metrics they need to track, new controllable input metrics that they’ve learned, they’ve discovered. But overall, you have the same causal model of the business that’s constantly being tested because constantly you’re asking, “Hey, we’ve been trying to drive up this controllable input metric for like five weeks now, or like five months now. And the output metric is no longer budge. Maybe there’s no more causal link. Maybe we need to find a new controllable input metric.”

0:38:46.0 CC: Or, “Hey, this is very weird, this does not feel right. We are driving the controllable input metrics as high as we’ve driven it for the last year, but the output metric growth has stalled out.” I think there’s something going on with the relationship you need to go investigate. So like the S team will say this and then like then the actual operational team will have to go investigate and figure out what’s going on.

0:39:09.1 TW: Is there a piece you said earlier, and it was in the piece as well, that the like a knock of like, well this is just like little incremental gains and that being kind of a mechanism for dismissing it, which seems hopefully people, that’s an easy one to get past. But does it also have the counter effect that if you do have an idea, if you think there’s a new input or there’s a way to really goose an input, you are gonna be a little bit held to, well, it better be, your intuition must tell you that it’s a big enough of a goose of that input or something causally related to the output that you’ll be able to detect it. ‘Cause you’re gonna have to move it far enough, similar to if you’re doing an AB test, we’re gonna test a change.

0:39:54.9 TW: If we test a tiny little change, then our ability to detect a difference is gonna be pretty low. And this seems kind of, it’s similar as well that if you’re saying I am going to try to affect, I guess I’m not affecting the process, I’m affecting a piece in the process, which I expect to have this effect, it’s gonna push me away from doing little bitty micro things, ’cause I’ll be like, I could do that, but it’s gonna get lost in the noise. Like the noise is having much more impact. The variation is gonna have, is gonna dwarf whatever change to the signal that I may have.

0:40:34.4 CC: That’s correct.

0:40:35.8 TW: So it seems like it would help you on that backend from too small of work.

0:40:40.2 CC: Yeah. One of the things that we very quickly internalize, and I sort of talk about this through the sequence. There’s a part in the essay that goes through a sequence of events that should happen when you start using XmR charts. And you can short circuit all that by putting the WBL to practice, which is what we did. But we went through a spiritually, like a similar sort of journey. And one of the big things that we learned is that most things don’t work. And I think that’s true more broadly of data, right? You think, you come in gung ho going like, “I know how my business works. We’re gonna change this and it’s gonna move metric.” No, it doesn’t. And everybody goes through this adjustment process where they’re like, “Oh, we don’t actually understand how our business works. All the things that we think would work, don’t work. And then the things that we don’t expect to work actually do work.” Huh? Maybe our causal model of the business in our heads is wrong. Maybe we should go do more experiments to figure out like what actually changes the numbers.” Right?

0:41:32.0 CC: To speak to your point about AB test, the problem, one problem with SPC is that it is comparatively insensitive, especially if you are talking about two, what do you call it? Two body experimental designs or statistics where you have to sample and you can have a control group and then you change something, that is more sensitive and it’s actually more able to tease out effects. SPC is relatively insensitive because it optimizes the XmR chart optimizes for, they’re okay with false negatives. They’re not okay with, they try to have give you less false positives because in experience if an operator discovers a false positive and is sent on a wild goose chase investigating some data point, it turns out to be nothing. Right?

0:42:12.7 CC: They’re gonna lose trust in the tool very quickly. And so the tool is tuned for operational focal are not very sophisticated in terms of data or stats to trust the tool. As opposed to if you use something like Facebook’s profit, which is supposed to do anomaly detection, but actually it has horrible performance in the beginning and over time gets better as it sees more and more data, especially seasonal data. That thing actually doesn’t work well with operational teams who are not sophisticated data people because they would just get the first false positive. They’re just gonna distrust the tool and they’re never gonna use it again. Right.

0:42:42.9 TW: Well, I mean, where I’m starting with XmR, starting with basic SPC and as you said, like it is a simpler, it still is gonna take a little education. I mean, I wonder if one way to think about it is that’s like your starting point and then there’s a whole world of more sophisticated techniques. I mean, I, to this day, like a seminal point in my analytical career was understanding like time series decomposition. ‘Cause when you talk about in your piece shows the, a Google analytics chart that kind of just goes up and down and digital analysts are used to seeing kind of the humps where it dips on the weekend goes up, dips on the weekend. So there’s a recurring pattern. And if you just did, I think if you just do a straight up, I don’t know if an XmR chart is gonna wind up showing more variability, but you can extract that seasonality out and then get a much, much tighter variation because you’re controlling for the day of the week.

0:43:43.2 TW: Or if you go to profit or you go with the causal impact, basing structural time series, like to me those are all like, they’re ways of trying to turn up the sensitivity on what is fundamentally the idea of statistical process control. How can you make the observed variation as small as possible by accounting for other things? But it does start to get pretty complicated as to what’s going on and also can cause more damage in the hands of someone who’s not really equipped to drive it. But, right. Julie, ’cause you’ve had causal impact, you’re a causal impact guru at this point. You better than I am.

[laughter]

0:44:28.9 MH: Well, the other thing about this, and I think it might speak to some of its challenges, is that one of the assumptions this SPC stuff makes is that the business person is interested in seeking mastery of the systems, processes and people that are actually at play. And that’s not always the case.

0:44:48.8 TW: Yep.

0:44:49.6 MH: For better or worse, you have plenty of people who’ve somehow ended up in leadership positions who are just simply executing a prior playbook with no sense of why they’re doing it. Really. It’s sort of like cargo cultish in a way. And I feel like that’s where this really falls flat. ‘Cause they have no idea what the variation is or what might be in messing with their numbers. So it sort of is balancing act of like you actually have to want to do better to make this thing work for you a little bit.

0:45:20.2 JH: Can I throw out like a scenario and ask you how you think SPC could maybe help here or not? I’m just, I’m trying to wrap my head around it. So say there’s like, it’s a marketing group and they have, their company has like a big event that goes on and they’re planning to have a lot of small audiences that they target based on what specific sessions they go and attend, like at an event, a conference. And they’re pretty much saying like, “We’re in ahead of this conference happening, we’re planning to have a lot of this like marketing go out to them and we’re gonna target them based on the stuff they say they’re interested in and what they attend.” And then we think down funnel, it’s going to make them have more cross sell upsell. Like that to me is, it’s all big and nebulous, but I’ve heard that story a lot before with different clients and like sure intuitively you’re like yeah, market to them and help them go further in their journey.

0:46:19.0 JH: But to me it’s all just the same storyline and there’s no specifics and it doesn’t feel like anything is really defined in what they’re trying to even achieve. But there is that underlying assumption that them targeting these small defined groups is going to move the needle on something the business cares about. So if that is the group of people you’re working with, how would you suggest or how would you see bringing in these like XmR charts or this SPC way of thinking, like how could it help them and how would they implement it?

0:46:54.7 CC: Ooh.

0:46:57.2 TW: Just just a little simple question.

0:46:58.3 JH: Yeah, sorry. I’m just gonna throw you in the deep end. Like I, this has been on my brain a lot.

0:47:00.6 TW: Yeah.

0:47:01.6 JH: And so I read this article and I was like, “Hmm.”

0:47:04.6 TW: How do I solve 90% of my interactions with clients? If you can do this one.

0:47:08.8 JH: Yes.

[laughter]

0:47:10.6 CC: So I’m gonna break this reply into two parts. I’m going to describe it in an ideal case from the perspective of a business owner who can do whatever they want in their own business, which is actually the.

0:47:22.0 JH: Perfect.

0:47:22.1 CC: The main weakness of my entire essay and my entire approach. Because I assume that leadership is bought in and will put everything to practice immediately, which is not the case, as you well know. And then I’ll answer from the perspective of what happens if you’re in an organizational setup where this is not that great, which is something that I still think about a lot. It’s a huge problem. So the first branch of the answer is if you own this business and you’re running the market, you are the CEO and you can tell the marketing team to do whatever. How this will fit into the whole process control worldview is you subject your entire marketing and sales pipeline to process control. To basically something like the WPR. So you already know what normal looks like. Given all the activities you’re doing, you know what normal looks like. You have a fingertip feel because you’ve been doing this week in, week out for years. And then you say, okay, we think that targeting small amounts of people, sorry, small segments, you do some kind of like segmentation. At least in-person event, is it? Is it, am I correct? Yeah, so in-person event, you have some kind of segmentation, you have a hypothesis that you believe that targeting these segmented people result in some kind of outcome.

0:48:42.4 CC: So then you define your output metrics. You tell them like, okay, you’re gonna run this thing, this program. It’s maybe gonna take a couple of months. I want you to write a six-pager, this is the Amazon way, write a six-pager or a PR FAQ first, detailing the output metrics you expect to impact. And the controllable input metrics that you expect to push, which maybe it could be a number of segmented people that you run through this process at the top of the funnel. And then based on your understanding of this process and your understanding of the business and your judgment, you tell us how long it might take to show up in the output metrics, which is based on qualitative judgment. You might say that, oh, based on like what we understand, it could take like maybe eight months to show up in the output metrics. And okay, so now we’re gonna look at our, again, the WBR week in and week out with the understanding that we’re gonna wait eight months to see if the output metric shows up. And maybe we’re gonna, based on your judgment, wait a bit longer or a bit, if it shows up earlier, that’s great.

0:49:32.7 CC: And so we know it works, so go do more of it. Otherwise, we’re just going to take a look at the output metrics and see if there is a change in the eight month point that you described that sure could be attributable to lots of other things, but are most likely based on your judgment. And there’s a lot of trust here that you understand the process that you’re running because you’ve been doing the WBR week in and week out for years. That, hey, I think there’s a real effect here.

0:49:57.0 CC: So that’s the ideal case example. And the real world example of this is Amazon Prime. When they launched Amazon Prime, Colin told me that they tracked three metrics, which was number of items per order, number of items per year, and number of orders per year. And they waited two years for the numbers to show, for the change in the output metric to show up. Because it was, and it’s a very long story, it’s, and you can actually, it’s actually public knowledge at this point because it’s a remarkable oral history article on vox.com. But the reason they were willing to wait two years was because it was existential. Because they saw that their growth was plateauing, and this was still the early days of the internet, and this was unacceptable to the entire premise of the company.

0:50:37.6 CC: So they were willing to make that huge sort of bet, which could have bankrupted the company. This is also quite well known. And they were willing to wait for two years before it showed up in the output metrics. And you are never truly sure if it’s causal. You only have varying degrees of certainty. And that’s the other thing that leapt out at me, which is that everybody in the S-Team seemed to have this incredibly high, this deep sense of epistemology. Like, what do we actually know? How can we tell? We kind of think it’s true, and so we will act as if it’s true, but it’s judgment-based. Like, we know this because we understand the customer deeply, we understand our business deeply, and we think that there’s a causal link here and here. So there’s a lot of that going on. It’s not just looking at numbers and doing what the numbers… The numbers are an assistant to your qualitative understanding of the business.

0:51:23.0 CC: Okay, anyway, so that’s like the ideal case. The non-ideal case, if you’re running a marketing team and you’re under a CEO who doesn’t understand routine variation, and maybe, or it’s not very data-driven in other ways, you are going to have a lot of trouble. So you can run a WBR internally in your marketing department, and you can inside your marketing department have a good sense of what actually impacts numbers. But then the question becomes like, can you get the political air cover to wait the length of time for it to show up? And that might not be possible depending on the culture of the company. One of the things that is either depressing or amazing about this topic, mostly depressing actually, is that [laughter] Deming refused.

0:52:17.7 TW: You are a data analyst.

0:52:20.7 CC: Yeah, see, there you go.

0:52:20.8 TW: You’re a data person.

0:52:22.1 MH: Come on in.

0:52:22.2 CC: No. I am a business operator where I have full control over my business, but I am fully aware that the data people who are reading my stuff may not apply what they’re doing in their companies. And the reason is like even Deming knew this. Deming refused to talk to middle management. There was a very famous story where a whole bunch of Ford people went to visit Deming at his home and he threw all of them out. He said, come back when you have somebody from the C-suite. And the reason, again, you will realize that everything that Deming does stems from his understanding of variation. And one of the big things is that if you have exceptional variation, you can find the root cause and you can remove it. Any worker can do that. But if your process is in a state of routine variation, it’s predictable, it’s stable.

0:53:05.1 CC: Then the only person who can change the process is management, is leadership. Because only leadership has the ability to buy new technology, rethink the process, work across departmental boundaries, to change the business process, to improve it for the better. And if leadership doesn’t give this blessing, there’s nothing you can do to improve the process behavior, if it is in statistical control, if it is stable. So he basically from this single sort of understanding, just sort of like he was very hard on management. It’s like your business sucks because of you, not because of your workers. Your workers are doing the best they can, embedded in a complex system that is not all in their control.

0:53:40.4 CC: You, only you as leaders have the ability to buy new technology and redesign processes and reconfigure your business. So part of the reason why it’s so effective in Amazon and Coke industries and IEMs is because the leaders were bought in and the leaders were data driven and the leaders had bought into the whole Deming sort of process control worldview. Amazon Prime was a multi-team, multi-departmental effort. Anyway, that’s the depressing, that’s my depressing answer.

[laughter]

0:54:10.1 TW: Well, but I mean, I’ll flip it that I think that there is a, I think there are analysts who haven’t internalized this. Like it’s really not never gonna change if the analyst isn’t embracing kind of the variation and the need to understand that there are lots of ways to sort of, even just looking at metrics that way, like, I don’t know, I feel like there’s incremental progress that can be made by visualizing the way that you’re returning the data, who’s, you have the metric, you can make an XmR chart, like you can put control limits on it and slowly kind of chip away and kind of change things. But you kind of have to understand it yourself first and then kind of work up. Even you may eventually hit a wall and you’ll just become a cranky early to mid 50s fella after.

0:55:01.8 CC: Actually, I just realized, oh Tim.

[laughter]

0:55:05.6 TW: Oh sorry, was that, was that still, was my continued thought, did that come out through my mouth.

0:55:12.1 CC: There is actually one thing that I do want to say and this is actually positive. Unlike all the negativity that we just had. [laughter] No, I remember listening to your, like multiple episodes of your podcast and thinking the one thing that I can offer as a non-data professional is, so XmR charts are quite dumb. They’re really simple. You can detect misuse in a few seconds of just looking at it. And that is part of its beauty. We have a tool that is simple, that is dumb, that is built on some fundamentally, it’s sophisticated statistics, but done in a way that anybody with no data background can use. So here’s a tool with a, and it has a track record, a 90 year track record of teaching variation to operators on the factory floor. Most of them, had at most a high school education had that. And as I was listening to your I was like thinking to myself, there is no way as an operator that I would be able to match your level of sophistication around data.

0:56:13.0 CC: And there is no way in an operational context, given the incentives of businesses, that I would pause what I’m doing to listen to you. And so on the other hand, I have been teaching XmR charts to various founder friends, and operator friends, people in marketing departments, people in engineering departments even, how to use XmR charts, so that they can finally understand data. And I come from a computer science background. I’m a software engineer by training. We are not data-driven. We are like some of the least data-driven people on the planet. And when you sort explain routine variations, like you can see the light bulb. If they get it, you can see the light bulb going on, and then they start trying to apply it to various things in a company, because they realize, oh, I have a tool that I can improve.

0:56:53.5 CC: So I think the one thing that I do want to say to your audience, many of whom I think are analytics professionals, is that you have a tool that you can teach any operator to use that is really easy to teach. That is really easy to detect misuse of, which makes your job easier. That has already been successfully used in, even in companies that are not ideal, that don’t have this sort of bayonet from the top. But what you have right now, is you have a tool that you can spread understanding of variation, which then increases their ability to appreciate what you do, the sophisticated things that you do because, and we have seen this, I have seen this in my own practice, in my own company. I have also seen this in the people that I’ve taught it to. So I taught this to, if you may allow me to tell a story.

0:57:42.0 CC: I taught this to someone who runs marketing in a hospital. And this is a traditionally run, not data-driven hospital based in Singapore. And she wanted to figure out like what are the levers that I can pull to increase the amount of inbound calls to the various clinics in the hospital. And so she subjected the call rate to statistical process control to XmR chart. And then she changed the Google maps listings of all the clinics. And the next, very next week she saw exceptional variation on like a whole bunch of these charts. Right? And the problem with that, of course, was that now a different part of the system became a bottleneck, which is the conversion from the call center to the clinic. Right? And that’s not under her control, because she doesn’t have power, she’s not leadership. But she could go to the operational team and say, Hey, look, this has resulted in changes, and not all of them understood it, but enough people with power were sufficiently interested to sort of go, okay, like, let’s talk about this a bit more, because then maybe she can go get the political power to try to get, change the way the process is run for routing of calls, right?

0:58:45.0 CC: We’ll see whether that works out. But the point is, she now feels more in control. She no longer looks at a chart and goes like, oh, what should I do? She feels like she’s in control. And if you are a data person within that company who had given her this tool, you have just built trust and maybe you can start introducing more sophisticated things. And if you imagine like most of the companies sort of going through this process, right? They will be more receptive to the more sophisticated things that you do that, oh, I don’t really understand it, but if you say that it must be true then. Okay, I’ll trust you. Because you introduced the XmR chart thing and it worked.

[laughter]

0:59:15.8 MH: You gotta meet them where they are and then go from there.

0:59:17.8 CC: Yeah.

0:59:19.2 TW: Yeah, yeah.

0:59:19.6 JH: Yeah. Absolutely.

0:59:19.6 MH: Alright. Well, that’s how all the listeners, we can all create an SPC Paradise for analytics. No, I’m just kidding.

[laughter]

0:59:28.5 MH: Alright. We have to start to wrap up, because we are now exceeding our variation on time. So no, this is such a good conversation and honestly, whatever, for whatever reason Cedric, like your writing and what you’re writing about just really, I think, is important and actually ticks a lot of boxes for me personally ’cause I think like you, like I’m not the most sophisticated analyst, but there’s a system and change that you wanna be able to like impact and have impact on. And like you said, this is a system, not perfect, but a system to kind of start that process. And so anyway, thank you so much for coming on. This is such a good, yeah, I love it. Love it.

1:00:14.7 CC: Thank you.

1:00:15.6 MH: Alright, one thing we love to do is go around the horn and share our last call. Something we think might be of interest to our listeners. Cedric, you’re our guest. Do you have a last call you’d like to share?

1:00:26.4 CC: Yes. So we built a tool, a free tool, that we’re gonna open source probably by the time this podcast comes out called Xmrit, xmrit.com. Because after writing this essay and then while teaching this to multiple people, I realized that it’s not easy to create XmR charts using Google Sheets or Excel templates. And in particular there are certain tools… So there are certain features that Xmrit has that should make it super, super easy to use XmR charts, one of which is dividers, right? Putting dividers is to say, a change has occurred, what is the process behavior after this change? We also give you the ability to lock limits. So you want to often characterize a stable process and exclude certain auto variation points, especially when you already know what caused those special variation points, right? So lock limits is not actually easy to do in a Google spreadsheet, at least it’s not easy to do in a, what you call it, if you’d, if you’re creating XmR charts every week. So Xmrit should make it a lot easier. And because we know that most people do not trust online tools, Xmrit is written in a way that the data never leaves the browser. But if you still don’t trust it, we’re gonna open source it so that you can then run a copy locally and hopefully it will be easier to get started with this tool and to give it to operators in your companies.

1:01:43.3 MH: Awesome.

1:01:44.3 TW: Ooh, can I go next ’cause I gotta follow on. I gotta a tie in.

1:01:47.1 MH: Okay. Yeah, go ahead Tim.

1:01:48.0 TW: Gotta tie in.

1:01:50.3 MH: Just barge right in there with just your last call.

[overlapping conversation]

1:01:55.7 TW: There’s some variation in the way we manage who goes next, but…

1:02:00.3 MH: No, it’s fine.

1:02:00.4 TW: I mean, it, it ties in, in that, my main last call is a LinkedIn newsletter called, Dinesh’s Data to Decisions. Which basically is a guy who does little videos of, it’s all Excel stuff that is, and it’s visualizations, it’s specifically charts. Like I actually wanna reach out to him and say, Hey, here’s a challenge. ‘Cause having been somebody who spent a chunk of my early career becoming kind of an Excel visualization jockey and trying to find ways to efficiently do stuff like that, the stuff that he does, like they look really good and their charts where I, it’s kind of like a game you can play, of say, how do I think he’s gonna do this?

1:02:41.0 TW: And then usually he has a more clever way that he’s actually done it. So a lot of the stuff, it’s like, it is very much Excel. Like it’s not, I mean Google Sheets is definitely not there. I’m assuming elsewhere in the Microsoft stack, some of the stuff would would apply. But that’s just kind of a tie on, if you’re struggling with, if you want, if you’re using Excel, and I’m not gonna say still using Excel, lots of people are using Excel. There are lots of things it can do. And his is one of those newsletters that there used to be a guy named Jon Peltier, I don’t even know if he still does it, who was a similar guy who would do this stuff. But Dinesh’s are like little videos on YouTube.

1:03:19.6 TW: But I also have to kind of throw in it’s conference season is coming up. So the DataConnect Conference in Columbus that I gotta plug every year I’ll be there, I’ll be volunteering, I’m gonna be a virtual MC. But we actually have a promo code. So if you’re wanting to go to a cool data conference in Columbus July 11th and 12th, if you use the promo code Power hour, you can get a 15% off on that registration. But before that, we have in June the, I guess the, well not the three of us, in place of Julie, it’ll be Val, but… No, it’ll be you too. All four of us.

1:03:57.6 MH: Yeah Julie too.

1:03:57.6 JH: Yeah. Don’t cut me out of this Tim, I’ll be there.

1:04:00.1 TW: I was thinking through the promo of it. Oh my God. [laughter] So the three of us plus Val Kroll will be at in Phoenix, June, 6th and 7th, though the workshops the 4th and 5th, but we will be recording an episode there and the, we’ve said this on past episodes, but there, remind you that there is a discount code for that, which is APH20. So I will be at both of those and hopefully we’ll see some listeners there.

1:04:28.8 MH: Awesome. Alright, well Julie, now that we’ve established you’ll be there. What’s your last call?

1:04:34.5 TW: This guy.

1:04:36.3 JH: Yeah.

1:04:36.6 TW: Leave it in Josh. I don’t know whether I’m going or coming.

[laughter]

1:04:41.7 JH: Well my last call is a world happiness report for 2024. That is done by…

1:04:49.5 TW: Not good for the US.

1:04:51.1 JH: Oh yeah. Me and Tim read the same newsletter. Can you tell? He knew exactly what I was going with. But it’s put out by Gallup. And one I’ve never seen this, so it was fun to like go through and look. And Finland is still the top ranked one. I think they have been for the last couple of years they said, which actually surprised me. And then the US has dropped, yeah, the 23rd slot, which I guess they said is the lowest since like 2012, which that’s a bummer. And they had some cool like summary points. One that they were saying that people under 30 are actually the ones I think driving this lower ranking and they’re not feeling, they’re feeling worse about their lives like year over year.

1:05:31.0 TW: Julie, cheer up you’re on a podcast, and you’re going to Phoenix, stop dragging us down.

1:05:35.5 JH: I was happy until you cut me out of Phoenix. [laughter] But it was crazy that they said, if you were born after…

1:05:40.1 TW: What am I have gonna to do? Put a Russian, an invading country on the next border right next to you to bring you up. Okay, cut it up.

1:05:45.6 MH: Alright Tim, Tim.

[laughter]

1:05:51.6 JH: But they were saying yeah, if you’re born after 1980 that happiness falls with each year of age. But conversely, if you’re born before in 1965, your life evaluation rises with age. And I just thought that was so crazy. And so there’s a bunch of other obviously like fun stuff in there, and they have a heat map on the, the link that we’ll add to the show notes. And I will say I wish that their color variation was better to see it, but they have the world map with like the different scores, and it’s pretty cool.

1:06:20.5 CC: Yeah, I’m not that, correlation is causation, but Finland has a lot of saunas, and so if you’re under 30, maybe try that out, see if that affects your mood.

1:06:31.4 TW: Finland has Simo Häyhä, like that’s it.

1:06:34.2 CC: Well that’s true.

1:06:34.3 TW: End of story. He’s been there as long as they’ve been topping the charts. So…

1:06:37.7 CC: That’s right. Simo.

1:06:38.8 TW: That’s it.

1:06:39.6 MH: Way to go. Holding a whole country up.

1:06:41.7 JH: They should do XmR charts on their happiness maybe.

1:06:45.9 TW: Yeah.

1:06:46.6 CC: That would make me happier.

1:06:48.3 TW: That is true.

1:06:50.1 MH: Alright. I have a last call too. And actually it’s funny, Cedric, I was also gonna mention Xmrit ’cause I thought that was a really great little tool. But I will instead talk a little bit about an article, Tim, that you wrote about, Are we dangerously obsessed with data collection? It’s something that you’ve been talking about lately, and I think it’s just a really good thing to think about. And it sort of ties into this a little bit in terms of like where we focus and how we think about data capture and data usage, things like that. So it’s a good article, we’ll put it in the show notes. And then I’ll also just give a shout out to commoncog.com, which is Cedric’s your publication that you do most of your writing on. And the articles there are just really good. I think I already said that, but I’m a real big fan.

1:07:35.6 TW: Thank you.

1:07:35.7 MH: And so yeah, no thank you. I am so bad at writing, but I’ve done enough of it to know how much work you’re putting in to write some of these articles and so it’s a lot. And so we appreciate it. So anyways, but yeah, keep it up. At least you’ve got a few people who are wanna read it and so.

1:07:56.3 TW: Thank you. No, I think, actually what’s been really cool is like there’s more and more people are like, Hey, have you seen this? Yeah, I saw, it’s so cool. And so like, it’s like, oh, did we just become best friends?

1:08:05.9 TW: Well, he put. He put a, I think a few of the charts in our like show prep, we’re like, oh yeah, that stuff’s getting some traction when your stuff, if your stuff…

1:08:13.0 MH: I actually went back and I found the first examples of control charts I ever did, was in 2008. Wow. At Lanzent. And I was like, I actually found the Excel that I did it, in and I was so proud of myself for being able to put them together. And I was like, why am I not doing this more now than I was then? But then you wanna bring it back. Bring it back around.

1:08:33.7 TW: So funny. See my, I’m glad I can’t find mine. The first time I tried to put control, I had no idea what I was doing and I just like kind of came up with something. So like I think it served the purpose of trying to illustrate that there’s variation, but I had no idea how to, on a line that was trending, how to put control charts. Yeah.

1:08:52.6 MH: Okay. I still don’t know how to do that. [laughter]

1:08:55.8 CC: I do not cover that for a reason. It’s very complex.

[laughter]

1:09:00.2 MH: Yeah. So that’s a whole other episode we need to cover is, how do you do this for companies that are growing? And how do you handle things that need or want real time?

1:09:13.9 CC: Oh, real time, ah, yeah.

1:09:14.1 TW: Yeah. Time series decompositions, got a print component, and a…

1:09:15.9 MH: Oh, okay. There you go.

1:09:19.4 CC: Funny thing on that note, I think a lot of people actually after I wrote the essay, sort of reached out and said I had experience with control charts. I’ve used control charts, but I never thought you could use it in this way. And the essay is actually written in a way to sort of highlight this fact, right? Because I think people have been exposed to control charts, but then they don’t realize that you could enable this style of thinking. So if you notice in the essay, the control charts only show up very late in the essay. Right? I go through this entire process to sort of talk about like, hey there is this way of looking at processes that are causal, and control charts are just like one way of getting there, but there are multiple ways, right? So I think the essay did its job. Like it makes people look at control charts and go like, oh, why don’t we do more of it? Which is yay.

1:10:06.1 MH: Yeah. [laughter]

1:10:07.6 MH: Anyways. Alright. And no show would be complete without a huge shoutout to our producer Josh Crowhurst, who does such an amazing job making this show happen. Josh, we appreciate it so much. And this. Fascinating discussion. Thank you Cedric so much for coming on. Really appreciate it. Thank you.

1:10:27.1 CC: You’re welcome. This was great.

1:10:27.7 MH: All right. I know I speak for both of my co-hosts, Tim and Julie, when I say, no matter if you’re seeing crazy statistical anomalies in your process control charts or not, remember, keep analyzing.

[music]

1:10:45.6 Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @AnalyticsHour, on the web at analyticshour.io. Our LinkedIn group and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst.

1:11:03.4 Charles Barkley: So, smart guys want to fit in. So, they made up a term called analytics. Analytics don’t work.

1:11:10.7 Kamala Harris: I love Venn diagrams. It’s just something about those three circles and the analysis about where there is the intersection, right?

1:11:19.8 CC: Oh, we start recording again.

1:11:22.1 TW: Oh, we’re recording again.

1:11:25.1 CC: Okay. Okay. Okay. I’m gonna trigger…

1:11:25.6 TW: Hey folks. Hi everyone. Episode 244.

[laughter]

1:11:32.6 TW: Got another hour?

1:11:32.6 CC: Let me get this off my chest, right? The bulk of PI tool vendors out there seem to think that they’re selling into, and I’m going to use the word data illiterate, sorry. A data literate, data mature audience, right? That’s not the case. It’s like they’re selling pencils into an island of skilled schoolchildren who has never gone through school. They don’t know how to write, they don’t know how to read, and then they’re like, oh, look, better pencil, better pencil, sharper pencil, better pencil, big pencil box, four pencils. And then like, everybody say, oh yeah, the reason I don’t write enough is because I don’t have the best pencil in the world. And then they get a pencil and they’re like, no, actually I don’t know how to write. It drives me nuts that everybody is talking about data products and…

1:12:10.2 TW: No, they say… I don’t, I can’t use this. And they say, you know what you need is you need a pen. So then they give them a pen.

1:12:17.0 CC: Oh yes.

1:12:17.8 TW: And then they’re like, I still can’t write.

1:12:17.9 CC: No, no. And you need an AI to help you.

1:12:20.1 TW: And then you need a pencil case.

1:12:24.0 JH: The robot pen.

1:12:25.4 TW: To put it in. And then you need a backpack. And then you throw it all in a lake.

1:12:28.7 CC: Oh, yes 100%. Yes. Yes. And then I will sell you the best lake. And there’s multiple lakes now, and you can have a warehouse that you put on top of the lake, and it’s a data lakehouse, right? But that’s not the point. The problem that all these vendors are is that they’re selling into an island where nobody has gone to school, and nobody wants to set up a school. I think I overprepared, because of your questions Tim. Tim your questions were really sophisticated. And I was like, okay, I better know the stats down pat. So I spent two hours yesterday going through the statistical of XmR charts, and yeah.

1:13:08.0 MH: Well, in the show prep when you’re like, and then we started using a Taguchi array, and I was like, we’re not gonna get to level…

[overlapping conversation]

1:13:16.0 TW: Oh, I was like, oh, full factorial versus Taguchi. I’m like, yeah, I don’t think with that’s, plus that would trigger some people who are our listeners like…

1:13:22.6 MH: 2006 called and my A/B testing vendor once. No, that was, yeah, that was how all the vendors sold A/B testing back then.

1:13:31.4 CC: Really? They said, Taguchi arrays.

1:13:33.1 TW: We use full factorial or Taguchi And everybody was like, Ooh, tell me more, what’s your…

[overlapping conversation]

1:13:37.6 CC: Oh my God. It’s so old. The technique is so old.

1:13:41.7 MH: Oh yeah. We have a really great friend of the show named Matt Gershoff, this drives him crazy.

1:13:49.0 TW: My finger was hovering over the stop record and I’m so glad that I was like something is coming. Hey, Josh, slip that sucker into the output.

1:13:57.1 MH: I’m not going to say anything about it, it just, he’s rightfully disappointed.

1:14:04.3 TW: Rock flag and variation.

[laughter]
[music]

Podcast: Download | Embed

Subscribe: RSS