#046: Measuring Podcasts at NPR with Steve Mulder

The following is a straight-up machine translation. It has not been human-reviewed or human-corrected. However, we did replace the original transcription, produced in 2017, with an updated one produced using OpenAI’s WhisperX in 2025, which, trust us, is much, much better than the original. Still, we apologize on behalf of the machines for any text that winds up being incorrect, nonsensical, or offensive. We have asked the machine to do better, but it simply responds with, “I’m sorry, Dave. I’m afraid I can’t do that.”

00:00:04.00 [Announcer]: Welcome to the Digital Analytics Power Hour. Tim, Michael, and the occasional guest discussing digital analytics issues of the day. Find them on Facebook at facebook.com forward slash analytics hour. And no, the Digital Analytics Power Hour.

00:00:23.68 [Michael Helbling]: Hi everyone, welcome to the Digital Analytics Power Hour. This is Episode 46. You know, when we started this podcasting journey more than a year and a half ago, one of the things we attempted to do was develop a measurement strategy and key performance indicators, or KPIs, am I right, for the show. And just to review, our goals were to publish 26 episodes, get at least 300 downloads per show, and have a growing trend of listeners, and do a podcast about making the podcasts. Well, we hit all but the last one, and we aim to rectify that right here and now. This might initially strike you as a bit indulgent and definitely meta, but if you’re nerds like us, I think you’re gonna enjoy it too. Tim, a senior partner at Analytics Demystified, and my co-host.

00:01:17.37 [Tim Wilson]: Hello, sir. WCBE and WOSU would be my public radio stations that I listen to. Thank you very much. That doesn’t make sense in context at this juncture. It doesn’t matter.

00:01:28.56 [Michael Helbling]: But I know what we’re talking about. So I’m making a joke. Anybody who’s read the title telling people on this podcast the context matter in any ways, it doesn’t matter. I’m Michael Helbling. I’m the Analytics Practice Lead at Search Discovery. So naturally, we wanted to draw on the experience of someone in the big leagues of podcasting. And so I am excited to introduce our guest, Steve Mulder. Steve is the Senior Director of Audience Insights at NPR. Prior to that, He was the VP of Experience Strategy at Isobar, and a long, long time ago, managed the user experience at Lycos. A little search engine, probably you might have heard of. He also has written a book about user experience called The User is Always Right, a practical guide to creating and using personas for the web. But we are delighted to have him on the show to talk about podcasts. Welcome, Steve.

00:02:23.63 [Steve Mulder]: Hey, thank you very much. Great to be here.

00:02:25.89 [Michael Helbling]: I think you have a perfect NPR voice, so this seems like it’s working really well.

00:02:32.15 [Steve Mulder]: You are very, very kind. I think I’m surrounded by people that I work with who would never say such a thing.

00:02:38.40 [Michael Helbling]: Okay, well, I also have no idea. Anyway, welcome to the podcast. We’re excited we’re here. I think as we kick things off, we want to talk about both podcast measurement as a practice, but also kind of how NPR is doing it. And maybe a great way for us to start is just for folks to understand sort of what it is you do at NPR and kind of how that intersects with all of the different podcasts and audio and measurement that NPR is doing.

00:03:07.98 [Steve Mulder]: Yes, so our team, the audience insights team is really the hub of audience intelligence with an NPR and across the entire public radio system of NPR stations. So we run basically digital analytics, broadcast analytics, custom research, third party research. We’re really trying to bring an understanding of audience and how to act on that understanding across the entire organization, right? Whether it’s programming decisions, digital decisions, business decisions, you name it, across the board. And of course, part of that umbrella is podcasting, the wonderful world of podcast measurement.

00:03:46.72 [Tim Wilson]: Is that entire team based in Boston where you are or is it scattered across different offices?

00:03:52.67 [Steve Mulder]: It’s scattered across the galaxy. No, it’s primarily in DC with a few folks in Boston as well. Ah, okay.

00:03:59.28 [Tim Wilson]: So what is the podcast ramp up from NPR? So I think I’ve, I make no secret of the fact that I listened to multiple NPR podcasts and it was excited when like fresh air kind of jumped onto the became a podcast because it seemed like it was a laggard for a while. I think that’s W-H-Y-Y out of Philadelphia. Is that right? Yeah, that’s right. That’s right. Yeah we’ll edit out all my all the fanboy stuff maybe maybe not. Oh no we’ll keep it in there keep it in there yeah i mean there i’m sure we will get the analytics there is to me it seems like npr is such a natural because you guys already. Me already do audio content and kind of journalism based audio content and so podcast seem like kind of a natural shift, but like how much can you talk about the last the evolution over the last three or four years of somewhere a light bulb going on. and kind of where did measurement fit because it seems like when you’re running through affiliate stations basically and everybody is a member, hopefully is a member and there are funds being raised and you have that local tie and I would think that measuring media as radio is kind of radically different from measuring audio media as the on-demand radio of podcasts. There’s like either six questions or no questions in that.

00:05:20.62 [Steve Mulder]: Yeah, the weird thing about what we’re doing right now is measurement across all these platforms that NPR takes advantage of is so different. Traditional broadcast radio comes with long-time existing standards, first through Arbitron, now through Nielsen. That’s kind of an established science, like measuring traditional radio. But then you get into other forms of listening, whether it’s a live audio stream, which pretty much all the NPR stations have now, or on-demand audio such as podcasts. It’s all vastly different, which is what really complicates our life every single day. You’re totally right.

00:06:00.00 [Tim Wilson]: So let me ask, is the traditional longstanding, I’ve always had a theory about TV, measuring TV audiences, that it’s actually fundamentally crap, but it’s just longstanding, totally accepted crap. And is live broadcast radio, everybody accepts it and just kind of takes the numbers at face value and doesn’t question the validity. It seems like it would be very similar challenges as measuring TV effectively.

00:06:27.88 [Steve Mulder]: Yeah, it depends who you talk to, how much credibility there is in those numbers, but it speaks to something that I really believe in, which is that whenever we talk about a standard and measurement of anything, we’re talking about a standard is a good enough Mutually agreed upon the thing we’re all gonna point out and say yeah that’s a good yardstick that’s good enough let’s just use that so you know while any kind of measurement is never gonna be perfect as we all know we live and breathe this stuff. You gotta start somewhere so you started with TV you start with radio and you know there’s a lot of technologies and radio that are getting better and better and measuring broadcast signals. But they’re not perfect. But it’s something the entire industry has pointed to and said, you know what, that thing that Nielsen is doing and they’re coming out ratings every month, every year, that’s good enough. Let’s just move on with our lives and rely on that as a standard. And that’s what we need in every listening platform, in every text platform. We have it on the web. We need it in the world of podcasts. That’s something that’s been evolving. I know we’re going to talk about. But that’s absolutely true in the broadcast radio world as well. It’s just, you know, Nielsen’s decided this is the way to measure it and we’re all sort of on board with that and rely on it as pretty good. They’ve gotten really good at a lot of things. It’s going to be perfect. Nope, never going to be perfect.

00:07:45.88 [Tim Wilson]: So it’s like bond rating agencies, right? I mean, those are totally reliable and never, no, sorry. You never possibly go, nothing ever goes wrong. I had not even, it did not occur to me the live streaming, which occasionally I like dip into when I’m like surprised, I’m like, oh, hell, so no, look, I can click on, you know, radio by tune in or I’m not sure which one I click on. I’m like, just to check and say, oh, son of a gun, I can’t listen to my local. So do you have a sense? Do you guys have, when you talk about audience that, You’ve got X percent of your audience is only traditional broadcast radio, which I do that because it’s my wake up in the morning on the radio. And then X percent is live streaming and X percent is more on demand. And these are the ones who are across all channels. Is that some of the stuff that you try to wrap your heads around?

00:08:36.90 [Steve Mulder]: Yes. We constantly try to wrap our heads around that. And we do with mixed success. You know, the hardest thing about my job right now, one of the hardest things is getting that true sense of audience because while I have these amazing data silos, it turns out to be really hard to measure someone like you who might be on the radio and on podcasts. And to me in my systems, you look like two different people, right? You’re a podcast listener over here, you’re a broadcast listener over here, and I can’t necessarily combine that and see you as one person. So I end up with exaggerated audience reach numbers in a way that I can’t de-dupe right now in a really easy way. And there are folks working on this, but it’s absolutely a key problem for a lot of organizations, media organizations like us that are struggling with how do we get a much truer sense of audience reach when we’ve got all these platforms out there.

00:09:34.15 [Michael Helbling]: You know, for somebody who claims to be such a huge fan, Tim, it’s pretty bogus for you to do that.

00:09:40.66 [Tim Wilson]: I’m telling Steve right now, so just write it down. I’m giving you the what you need.

00:09:48.87 [Michael Helbling]: Just know that 30% or so of the traffic you’re getting from Northern Columbus, probably Tim Wilson.

00:09:58.63 [Steve Mulder]: I’ll write that down. I’ll put that right into my analytics tools. Yeah, that’s perfect.

00:10:03.38 [Tim Wilson]: Well, I mean, the crazy thing is it runs into the same thing. It’s more challenging than web. My literally, my daughter, my youngest, that’s kind of a common thing she wants to do is when I pick her up from gymnastics, she’s like, can we listen to a podcast? And I’d say half of the time it’s an NPR podcast and you’re back to the same audience issue of now you’ve got a household listening to it. Now she’s probably a couple of years away from being a a donor, but she’s the future. I don’t know where that was going.

00:10:32.28 [Steve Mulder]: No, but this is one of the key ways where we think about podcast measurement as an evolving thing. Like all measurement is right, but it’s kind of a couple steps behind where we are with web measurement when you think about it. So I like to think about this sort of as an evolution. So you’ve got on the web, we began with hit counters. Oh, we remember them fondly, the hit counters of our youth. So you start with hit counters, you start talking about hits, then we started talking about page views. Finally, we started talking about audience, the people, who matter, talking about people, who would have thought, right? And you get this evolution of what you can measure, this evolution of what’s important to measure, how we’re gonna use that information. Similarly, in the world of podcasts, where you’ve got, initially it was all we had was that file was requested, that particular episode, but we don’t know what happened after that. Then you got to talking about downloads, how many downloads did a podcast get? And now finally we’re getting around to talking about what’s important, which is audience, talking about the people that are downloading these things or listening to them via streaming or whatever. And we can start treating that data in a much more evolved way than just talking about hits and page views and downloads in a very raw sense. So the way we think about it even, I think, even in the last three years has really evolved that way.

00:11:50.18 [Michael Helbling]: So not to take us too far off course, but there’s sort of a subject that I wonder if you could shed some light on, which is just sort of what have you observed in sort of the growth of podcasting generally? You can speak from the perspective of NPR naturally, but I don’t know how long podcasts have been around. They’ve been around a long time, but they haven’t been They’ve grown in popularity over the last, say, a few, five, seven years maybe. And I don’t know, have you seen anything in that growth or what have you observed?

00:12:23.02 [Steve Mulder]: Yeah, I think there’s a few things going on. One is you’ve got an entire society that We are now drawn to self-curated ways of consuming media. Think about our consumption of video and TV and movies. It’s become much more prolific across all types of channels and platforms, but it’s also much more about selection, like choosing what we want. And similarly now, in audio, in spoken word, you’re seeing that same evolution where people are realizing there is so much amazing content out there, this podcast included, of course. There’s so much amazing content out there that I have so much in my fingertips, we have merely to expose people to all that rich offerings and let them choose. And part of why you see podcasts just exploding is that level of choice, right? And the kind of just amazing diversity of content and voices and perspectives that are out there right now, much of what you just can’t get on traditional broadcast airwaves. So I think that’s, I think that’s a big part of it. And you’re also seeing organizations, sponsors catching on to just how much value there is in this industry. So here’s the stat that we like to talk about recently at NPR, which is compared to three years ago, NPR’s revenue from podcasts is now 10 times what it was. 10 times in three years. Wow. And that’s not just because we’re delivering 10 times as many shows, right? That’s a lot of our stalwart Programs like Ted radio hour and and right all the ones we’ve been talking about fresh air as well as new programs like embedded and hidden brain and invisibility so it really it really all adds up to be pretty profound growth in terms of the number of people listening and the number of sponsors who are interested in taking advantage of it.

00:14:10.01 [Tim Wilson]: Two questions. So one observation is the other thing is I think the time shifting that I think that as people got, everyone got DVRs and everybody got conditioned to TV on demand. Cause I think if it was plotted that there was kind of an early heyday of podcasts and then it dipped. I remember being made fun of when I was listening to a podcast cause they’re like, people still listen to podcasts. And that was a few years ago, and now they’ve kind of exploded and people point to cereal. But I think part of it is also the time-shifting nature that I can listen to it literally on demand when I want to listen to it. But when you say the revenue growth, I haven’t listened to a pledge drive in probably three years because I’ve shifted. So I know I’m aware when they’re on, because the radio comes on in the morning. But for the most part, I’m kicking over to a podcast. So is the revenue because they’re not called advertisers, they’re called what, underwriters or whatever the NPR? Is it because you’ve got those hidden brain is supported by or is it listeners when there are occasional, I’m trying to think whether NPR or whether it’s more just PRX that kind of occasionally says Go here like I remember trying to go and donate to Planet Moeney and it was it was hard like I wanted to go give money to support Planet Moeney and I think I don’t know if I wound up going given to WNYC or whatever it was because it so where does that revenue where when you say the revenue growth I guess I’m asking what’s the model that that happens under?

00:15:38.41 [Steve Mulder]: Yeah, yeah, really what I’m talking about that it’s it’s it’s classic underwriting and sponsorship So the hidden brain is brought to you by that’s really where the direct revenue comes You know in addition to that obviously NPR gets gets a lot of funding from the member stations with sales, paying for programming. So indirectly, you could say that there’s some revenue coming from that direction as well and institutional giving and that kind of thing. But what I’m talking about really is just looking at underwriting and sponsorship itself. It’s growing tremendously. And I know a lot of podcasts are finding that to be the case. And you guys clearly need to get on it.

00:16:12.24 [Michael Helbling]: Underwriting train. We’ve seen our revenue growth also triple in the last year and a half.

00:16:19.27 [Tim Wilson]: It’s actually increased infinitely because we yeah, that’s right zero times.

00:16:24.39 [Steve Mulder]: Congratulations. That’s awesome.

00:16:27.38 [Tim Wilson]: Yeah roof But is that, do you have, because you’ve got the specific nature of the programs, is part of that, are you providing evidence to the underwriters to say, wow, if people are interested in neuroscience and psychology, you know, these are people that you want to target. So we can kind of guarantee, and we can also give you some sort of data about the scale of how many people are listening to Hidden Brain.

00:16:56.96 [Steve Mulder]: Yeah, so the advantages that we have are in a lot of ways similar to the advantages on the air as well with sponsors. So we have a lot of evidence decades going back now about how listener perception of people who advertise on public radio, who sponsor public radio, the perception of those brands or likelihood to consider those brands is significantly higher than on commercial radio.

00:17:20.12 [Tim Wilson]: The downside is they’re all hippy liberals who are lifted or just being brainwashed by the left-wing media.

00:17:25.43 [Steve Mulder]: I don’t know which podcast you’re listening to.

00:17:30.32 [Tim Wilson]: Okay, so I think I cut you off. But you were saying that you’ve got credibility as a desirable target. I feel like even for for profit podcasts, I feel like with my clients, there’s kind of a mist that It is kind of pretty easy and my perception is relatively cheap to become an advertiser on a podcast and you have a pretty good sense of the audience. But is there also a defining the audience? So you’re an audience guy. And other podcasts and i could have sworn i heard this on an npr one and i went back and tried to find it and i googled for it and couldn’t find it like the old fill out a listener survey tell us about yourself are you involved with those.

00:18:14.55 [Steve Mulder]: Yeah so that that’s part of what we do for the various shows the various podcasts is constantly doing customer research or listener audience surveys really. to understand how people are reacting to the shows, to understand, of course, their demographics. We test things like how much do people remember the sponsors they’re hearing in the broadcasts as well, right? So we can go back to the sponsor and say like, hey, this is how many people unaided were remembering your particular underwriting spot and the kind of reach that you’re getting. So we gather that kind of data through surveys for sponsorship purposes, for content purposes, right, to understand what’s working, for just overall strategy purposes so we know, you know, where are we connecting with our audience in a very real way. So, you know, we’ll do cross show surveys where we do a bunch of, we get a bunch of, we’re doing one of those right now that’s out in the market, and we’ll do like deep dyes. We did a survey on, you know, just invisibility as audience, for example, just recently, when that team really wants to immerse themselves in how the listeners are feeling about the shows that they just finished listening to.

00:19:21.50 [Tim Wilson]: And can people tell the two original hosts apart?

00:19:25.03 [Steve Mulder]: Yes, some people claim to, but we think that’s a myth.

00:19:30.48 [Tim Wilson]: This actually just sparked. God, you’re going to kill me, help link. So I know, so I’m now remembering, I’d forgotten the episode that Planet Moeney did where they talked about, they talked about Dancer Royker. They don’t think they mentioned Optimizely, but they talked about AB testing and AB testing through the NPR one app and actually pushing different It was different bumpers are different titles i can’t remember what it was so are you guys also a b because of you have the npr one app are you regularly a b testing are you involved in that is that another element of it yeah it is so let’s let me lay out like the the two fundamentally different types of podcast measurement we’re doing right now so.

00:20:08.01 [Steve Mulder]: You know, for most of the NPR podcast listing that’s happening out there in the wide world, it’s happening via players, all manner of audio players that we don’t control, right, that are way outside of the NPR influence and so forth. It’s only a fairly small percentage of listing that’s happening within environments like NPR1, the app, or the website or what have you right so where we control the platform of course we can control and do a lot more measurement and do a lot more with data and testing as you said so yes for example when the politics podcast was getting started last late last fall. One of the first things they did was they put a couple of sample episodes on NPR one as a test bed before anybody else heard it as a way to start gauging interest. Did people stick through it? Did they skip? You know, you can learn a lot by just looking at a sample audience, right? In a platform like that and unannounced, right? Just it starts appearing in the flow of NPR one audio and seeing what works. And they just, they learned a tremendous amount about what that show, the promise of that show was and the level of interest as well. And then we back that up. So very often our research, our group does the research behind concept testing of shows. So we’ll get together either as a focus group or just a bunch of individuals and we’ll have them listen to part of a new show and give us their opinion, right? Existing listeners potential new listeners and so we’ll use all this data in making decisions about you know which shows get brought to fruition. If a focus needs to change or things need to pivot. You know we can learn a lot early before an official launch so yeah npr one is a great platform for that kind of testing because we can roll things out. We’ve got a ready audience and we can see all the detailed metrics much more than we can get through. you know, the vast majority of NPR podcast listening that’s happening out there.

00:21:59.27 [Tim Wilson]: So is there a weekly podcast performance measurement dashboard or something? Like, is there internally, is it, are you guys saying, yeah, that one started off strong, but it’s kind of slipped? Is there, I guess, even backing up from that, when it comes to, do you have consistent KPIs for all podcasts basically that people are aligned to. And if so, I guess, what are they and how good or bad do you feel they are?

00:22:30.70 [Steve Mulder]: Yeah, yeah, yeah. Well, let me lay out the way we really think about it is that there are different ways of measuring podcasts for different purposes right now. And in fact, that that’s a good thing. So let me be back up. You think about the websites that we’ve grown to love over the years. When you think about the web, we really have at least three different measures of the web for different purposes. We have tools like good old Google Analytics, our number of record, our internal number of record with the most comprehensive data, where we can do the deepest kind of analysis. That’s one source. We have another source for doing industry comparisons. We have the comm scores of the world where we can do True apples to apples industry wide comparisons that we can’t really get your g a obviously and then finally think about sponsorship and underwriting and the kind of metrics that they need on the web and you got dfp rides not like we’re using google analytics to provide stats for our sponsors for for for banner ads and so forth so. This is exactly where the world of podcasts is growing up into right now. So same kind of format. You’ve got the number of record, our internal number of record. For that, I’ll come back to that in a minute, but for that we are doing raw server log analysis and using Splunk, which was only made for this purpose, but we’re using it and it actually works out really well. Using Splunk to parse raw log files and generate reports and dashboards as we see fit. That’s our number of record internally and our number of record externally as well for like the number of official users and downloads for our shows. For industry comparisons, you’ve got a couple of sources. iTunes was the only one for a while, but honestly that there top 20 is a bit dicey. The iTunes ranking is People who have tried to analyze this kind of throw up their hands because it’s not really a top 20. It’s more like, here’s the top movers and shakers as Apple sees them right now, but not necessarily a reliable top list. But PodTrack recently in the last few months has come out with their ranking now of the top podcast producers in the country. and it’s becoming sort of the de facto, I think, reliable metric of that. And then finally, you’ve got- And where’s advertising and sponsors?

00:24:39.62 [Tim Wilson]: Where’s PODTrack’s data coming from? Is that a panel?

00:24:43.68 [Steve Mulder]: Is that- They think of PODTrack as sort of like a feed burner used to be. So essentially, an organization works with PODTrack and submits every file request through PODTrack down to the CDN. So PODTrack’s seeing all the file requests. and can therefore look at all the raw data almost as well as you could if you’re looking at your own raw log files. So that’s how they do it. So you have to actually sign on for their service and then redirect through them. But once you do, you’re in business and it’s free. And then finally, for advertisers and underwriters, sponsors, similarly, we’re using ad server numbers for podcasts as well. So with NPR, our podcasts are on Triton Digital Media hosted there. We use them for our ad server. and they provide the sort of third party metrics for underwriters in terms of billing purposes. So you’ve got, just like with web, you’ve got three different ways of measuring, all equally valid, different advantages, and all are really important to our business. So you asked about if we have dashboards so it’s those internal numbers i talked about where we’re we’re actually parsing raw log files using this tool called splunk to generate reports and dashboards that’s really the the intelligence that internally we’re looking at day in day out week in week out.

00:26:01.20 [Tim Wilson]: But it does seem like there’s, in the for-profit world, where you’ve got the Panoply network, which is, I think, kind of a media-driven, but more for-profit media-driven. There’s Gimlet Media, which essentially spun out from NPR, because most of the founders came from NPR. I feel like you’ve got probably more clarity of thought around some of this measurement stuff than maybe some of them do. Because it seems like the for-profit world, they’re almost exclusively playing to the advertisers, whereas it seems like with the public media focus, you might have a little heavier stake in the audience, maybe, in content quality. I don’t know. I’m extrapolating or riffing.

00:26:44.91 [Steve Mulder]: Yeah it’s a good question and I can tell you that obviously serving our audience creating amazing content is way up there in the priority list. But obviously it doesn’t happen without sponsorship right so you gotta have all of this it’s a it’s at least a three legged stool going on there and so the focus is just really high on on all of the above.

00:27:02.46 [Michael Helbling]: You were talking about PodTrack, which is interesting because that’s giving you that external view that’s being provided in terms of competitive intelligence, but you’ve also done quite a bit of work inside of NPR to do standardization. Can you talk about that and how does that track with or align with what you’re seeing happening with tools like PodTrack? Are there conflicts emerging? Is there good unity in terms of that? Is it because there’s just so little other data that everybody has to basically follow the same path?

00:27:37.87 [Steve Mulder]: Yeah, I’ll sometimes talk about this as podcast measurement being kind of still the Wild West. Insofar as there’s been for a decade now a lot of really smart sheriffs all around the country doing great work independently on measurement and getting a lot better at it, But what we haven’t had is a true law of the land. We haven’t had a true industry-wide standard like we were talking about earlier where everyone can point to that thing and say, yep, that’s the standard. That’s the yardstick we’re going to use when we talk about defining what a download is, what a user is when we talk about podcasts. It turns out there’s a lot of little things in the world of podcasts that can really result in vastly different numbers. So questions for you guys. So here’s the scenario. So imagine that you’ve got one person and you know they’re one person because they have an IP address and a user agent combination, right? because you don’t have cookies in the world of podcasts. All you have is IP and user agent. So you’ve got an IP and user agent combo that downloads one of your episodes three times, the same episode three times in a single day. Should that count as one download or three? And who decides? So here’s another scenario, let’s say that that person downloads a third of this particular episode we’re making right now, this is very meta. But if they download a third of this episode and you don’t know if they play it or not, all they know is that I downloaded it. Does that count as a download? Would you count that in your metrics? All these decisions come to the fore really quickly. Which is why we needed a yardstick like so what we did was last year NPR got together with about a dozen other public radio organizations. So some of them were stations some of them were big public radio producers like PRX and APM and so forth and all sorts of other alphabet soup and we got together and we started talking about this problem. realizing that we need to create a yardstick of our own. We need to create a common understanding guidelines, if you will, about how we as Public Radio are thinking about this space, are thinking about how we count, what we count, why a standard is important. So we worked on these Public Radio podcast measurement guidelines, and they came out in February, so earlier this year now, and it’s been great. We’ve been seeing vendors start to use them. It’s been influencing larger conversations at the IAB, so, The Interactive Advertising Bureau has a working group working on this issue as well on podcast measurement and technologies.

00:30:03.50 [Tim Wilson]: Please tell me they’re not going to be the ones to wind up as the final.

00:30:09.37 [Steve Mulder]: Well, we’re at that table and having the same conversation with a much larger group across the industry than just public radio, you can imagine at that table. But it’s really important because we want to make sure that When we’re talking not only internally, but also with potential sponsors, with the press in terms of coverage that we get, we want to use the same language, all of us, and we want to use as much as we can the same yardstick. So then when we’re talking about how many listeners a podcast has or how many downloads an episode has, we can be confident that you know we’re reliably incredibly talking about it in the same way we can compare apples to apples so yeah big believer in creating industry consensus around this ideally eventually a real standard like we have on broadcast television. et cetera. And the web, you could even argue, we don’t even may not have a standard per se, but at least we’ve all agreed on, yes, there’s this thing called a page view. There’s this thing called a user, and here’s generally how we measure it. We trust our tools as the de facto standards creation in that regard. We don’t have any podcasts right now, but we really need it.

00:31:16.66 [Tim Wilson]: And just to be clear that when when Steve, you’re saying there are standards because you’ve shared the link beforehand there, there’s literally a Google Doc standard, which I assuming we can just post in our show notes for anyone who’s curious as to what those are.

00:31:30.58 [Steve Mulder]: Yeah. Yeah. I would describe it. Feel free to post it. It’s bit.ly slash podcast guidelines. Feel free to post it. And it is just guidelines. I mean, you know, NPR and the world of public radio, we are not a standards body, nor do we want to be one when we grow up. We’re not in a place to audit police, deal with compliance, and create a real standard. So they are merely guidelines, but we hope useful ones.

00:31:54.04 [Tim Wilson]: There’s a W3C joke in there somewhere.

00:31:59.34 [Michael Helbling]: This conversation is one that’s very applicable across any analytics pursuit. That’s really fascinating. Certainly, the overarching governance from an industry perspective is one that, to your point, NBR is not looking to take that on, but just for yourselves to have that is such a great step. It’s pretty cool. I read through it and it was just fascinating because it started off pretty normal and then all of a sudden it was like in 206 return codes are like this. And I was like, and we’re back because it got real technical. Yeah, actually.

00:32:35.44 [Tim Wilson]: 206 plus bite range, zero minus or zero.

00:32:38.78 [Michael Helbling]: And I was like, all right. So this has got some very specific things to it. And I think that’s what you were kind of alluding to maybe before around sort of, hey, for all these 10 years, people are doing a great job just doing it a different way. and not a lot of consistency on a user or show-by-show basis for measurement. So creating a standard like that is really good. So Tim, we’re going to have to go back. Let’s say now, let’s subscribe to the public radio podcast measurement guidelines and let’s try to refactor our numbers. We don’t really talk about our numbers, so we won’t.

00:33:24.71 [Tim Wilson]: Well, but I guess there’s a question in that because we, you know, we use a third party host, which they’re multi sound like you guys, you said, where do the hosts kind of the cloud based hosts fit in this? They’re not on the list, which I think is fantastic that they’re not contributing to what the definitions are. But it sounds like you were saying that potentially providers saying where we’re running the stats. It’s funny looking at different platforms hosting and I go if you upgrade you’ll get more advanced analytics and I’m like I know what’s basically in the log file like you can’t give me much. more but there is a part of me that thinks when do they start saying we are following the public radio guidelines whether that is our default is we follow those guidelines or whether it’s an option for you to view you know what does it really look like if you’re feeling good about yourself you know slide this switch the other way and you’ll you’ll realize that you and your extended family or half of your audience.

00:34:23.59 [Steve Mulder]: But no, that’s exactly right. And that’s starting to happen. It’s been really exciting to see that start to happen both on the side of vendors and hosting companies as well as measurement companies starting to take on this issue. A lot of them have seat to the table at this IAB group that I mentioned. So the conversation is happening. It’s really fruitful. I think everyone recognizes that we can’t just be out there measuring based on every yardstick we want, we got to be talking about it in the same way if we want this industry to evolve to its full potential.

00:34:54.64 [Tim Wilson]: And are the, I mean, Splunk is, that’s a pretty serious tool. I’ve never worked with it, but familiar with it. You got your log files, you’re cranking them through Splunk. You’re doing some real, some real, real stuff there. As you guys were talking through some of this, what’s the cutoff where they’re contributors to this document who were like, uh, never really thought about it. We just counted every request. I mean, are there some that they’re like, their brains are hurting and they’re realizing that they’ve been really, really simplistic. Or is it kind of by definition, the ones who are involved or ones that have, you know, can kind of get their, get their hands dirty with a log file?

00:35:32.17 [Steve Mulder]: Yeah, it’s a it’s a mix so you know you’ve got some of the larger organizations that have their own capability for this rolling their own numbers using tools like Splunk or elastic search or what have you. Based on log files and then you’ve got you know a lot of orgs part of that group as well state you know smaller stations for example who are. much more reliant on third parties, whether it’s pod track, right? Or whether it’s their own hosting provider to give them the metrics. But now they can go back to these hosting providers, for example, and make sure that they’re getting the kinds of metrics that they want. So they can ask, hey, how are you guys calculating downloads right now? And are you doing this X, Y, and Z? And making sure you’re getting audience metrics, talking about measuring users and not just measuring downloads. How are you filtering out bots? How are you filtering out downloads that we kind of know aren’t real listening? How are you dealing with the same user downloading the same file three times in a particular day? So it can be much more intelligent, right? And we’re all trying to up our game in this department of just being more intelligent about the metrics that we do get, whether we’re generating them ourselves or whether we’re getting them from any kind of third party. And that’s true across everything, right? Not just the world of podcasts, which obviously I care way too much about.

00:36:46.23 [Tim Wilson]: But you can’t care too much about that.

00:36:48.96 [Michael Helbling]: No, because it takes me back to early, early in your career when you would do something to say, hey, technically, our data quality is really poor because we’re tracking to your point all these bots. So let’s start filtering them out. And people would be like, well, wait a second. That’ll make our numbers go down.

00:37:10.51 [Steve Mulder]: Right.

00:37:11.90 [Michael Helbling]: And so it’s just interesting. It’s sort of like, right? So yeah, I get that. These baby steps. So maybe the real big closer of a question is, are we smarter this time around with podcast analytics than we were with digital analytics 15 years ago?

00:37:28.03 [Tim Wilson]: Oh, we got that curve faster. Right. Will we at least go through that mature maturation process more quickly? Yeah.

00:37:35.35 [Steve Mulder]: I mean, it does seem like it happens. It does happen faster and faster, right? I mean, going way back, thinking about radio and TV measurement, it took decades to really get better and better at that. And, you know, we’re never going to hit perfect. Like the, the asymptote never approaches the, I’m forgetting, you know, my geometry just got lost in my brain, but we’re never going to be tracking with you. That’s right. Right. We’re never going to get perfect, but we’re going to get close. And I, yeah, I think we, um, we’re getting faster at getting better, you know, when it started with the web and, uh, it’s caring for us to app measurement and podcast measurement and, you know, you name it.

00:38:10.40 [Tim Wilson]: I wonder whether light bulbs, right now I feel like podcast advertising is likely comparatively incredibly cheap that even without great measurement with all of the gaps, I still feel like just because the weird mix of organizations that are advertising and the lack of kind of really big mainstream advertisers where they could just own seven podcasts for a year for the cost of what they spent in two days on banner ads or 30 seconds on TV. I wonder if podcast adoption keeps going up and there will be more and more brands clicking to that We may run into kind of a messier world where it still seems like there’s a limited the cost to make podcasts like you can get a high margin. We’re not making a business out of this and you guys are public media so maybe the three of us are the worst to. to talk about it, but it seems like from a business model, it’s gonna get better for the podcast, it’s gonna get more competitive. Right now, I feel like brands are missing an opportunity. If they have a key audience that is a demographic that is likely to be enabled to listen to podcasts, and they could spend a little bit of time figuring out where those podcasts are, and get some decent, conservative or aggressive, some range of what the reach is, and it’s a captive audience, And to your point, there are some podcasts where they probably have credibility, not necessarily NPR ones, but other ones saying these guys are putting out great content and I’ve got a better boost than if I’m just running on CBS. There’s potential there. That’s tangent number 72.

00:39:53.14 [Steve Mulder]: Well, yeah. And I mean, the potential is. It has yet to really reach its max. I’m sure right so you look at Edison data Edison research data now saying that 57 million Americans are listening to podcasts monthly 57 million that’s like 21% of the population of adults or 12 plus adults. So that’s that’s still you can look at either way right you can look at that as wow that’s gotten a lot bigger than it used to be 10 years before that. But you could also look at it as that’s got a long ways to go to get huge, truly huge in a way that the really large companies are going to think about it as a primary channel for sponsorship and not just kind of like a minor secondary thing that they do. Yeah.

00:40:34.53 [Tim Wilson]: Remember when paid search was kicking on with that where it was kind of like this little one person’s little part time job for a pretty large company. They’re just throwing some money at AdWords. So yeah, those were the days.

00:40:46.43 [Michael Helbling]: Well, this has been really awesome, but unfortunately, we are going to have to start to wrap up. I think this conversation will be interesting to more than just the three of us.

00:40:55.96 [Steve Mulder]: I hope so.

00:40:59.14 [Tim Wilson]: I’m pretty sure we know our audience has a proficiency to be podcast listeners.

00:41:03.68 [Michael Helbling]: This is not the racket that’s keeping them listening to podcasts. We’re talking to our own type of people for sure with this show. But yeah, as you’ve been listening, if you do have thoughts or questions or things like that, please definitely reach out on our Facebook page or the Measure Slack. Well, one of the things we love to do or we like to do at the tail end of the show is our last call. We go around the horn and talk about something that we find interesting or useful in our job career or life. And so, Steve, why don’t we kick it off with you?

00:41:39.55 [Steve Mulder]: Yeah, sure. So I came across this article last week in media shift. The title of it is kind of fun. I like to say it. It’s the title. It is Bulgarian analytics startup aims to fix how publishers use data. And I just like seeing the phrase. So if you go through the media shift for Bulgarian analytics startup, you’ll find it.

00:41:58.45 [Tim Wilson]: I feel like the content has to be a letdown.

00:42:03.66 [Steve Mulder]: Can it possibly deliver on that? Well, I would say there are a ton of articles and perspectives across the years about how publishers, media companies, use analytics. There’s a lot of ways to do that. There’s ways to integrate analytics into your newsroom and your content folks, really embedded in the numbers. I think this article is kind of interesting, but what really stood out for me is it jump-started yet another conversation of many that we have at NPR around how do we expose folks in our newsroom and our programming groups to all the numbers we have in a way that’s simple and understandable. They actually, they talked about this company who does something they call a CPI content performance indicator, where for every particular story, they’ll do basically an index, a calculated score around that story’s performance. So it’s a combination of page views and shares and likes. And so they do a calculated average, a calculated score that they’ve made up. And we have this conversation all the time within NPR as well about there’s a certain benefit to creating a nice simple index score for evaluating content. But at the same time, it hides all the really interesting complexity about how individual pieces can be better or worse in certain ways. So is that better? Is that worse? So this article, it’s not a crazy detailed article and I don’t think it has pushed things in a lot of new directions, but Jump started a conversation within our group that’s been really healthy over the last couple of days.

00:43:32.78 [Michael Helbling]: Alright, so yeah, I’ve got a last call. I just recently started reading a new book called Smart Choices. It’s a practical guide to making better decisions and I’m really enjoying it. It’s written by John Hammond and Ralph Keeney and another guy who I’m totally blanking on his name. But it’s, you know, what we do in analytics is help people make better decisions. So I’m trying to learn how to do that. So there you go. Book recommendation.

00:44:02.65 [Tim Wilson]: Okay, I will continue the downward slide of the quality of the last calls. I believe I made a podcast recommendation on the last one. There is no flipping way that we could actually talk to NPR and not have me just totally log roll for NPR. I’m gonna do a dual one and I’m breaking, I’m sure, some internal style thing and that both of these have been mentioned while we’ve been talking. Because it came back to me, the Planet Moeney episode on ARB, not super, super deep from a AB testing. It’s back in December, episode 669. But if you are in the optimization space and you’re kind of sometime trying to talk to a relative or a friend who says, what do you do? That’s not a bad way to say, hey, you know, Listeners to NPR think that whoever what’s ever been talked about has credibility as Steve told us point them to that episode and then I second I’m going to also last call the NPR politics podcast because if you enjoy this podcast if you’re at all a political junkie and that is another one of my many many vices the politics slate I mean the slate That was bad. That was bad. The slate political gap has been around longer than the NPR politics podcast, but the slate political without saying it, they pretty much said that the NPR politics podcast like overtook them somewhere by some measure at some point. I think the quality of having smart journalists talking, everyone from Tamara Keith, who apparently everyone adores, to Ron Elving, who is just like the old man of the sea, the wise. to Sam Sanders. So I kind of think the format’s great because I think they’re five to seven different people and they just kind of rotate through and it’s in theory kind of a weekly roundup, but they’ve been doing, they got to do three or four a week and it’s one, a little mind boggling being on the inside of this podcast and knowing how long it takes us to get from recording to standing it up to their turning stuff around inside 10 hours. So I’m kind of doing a dual last call and they’re all podcast NPR related. Preach. Well said. That’s right.

00:46:29.63 [Michael Helbling]: Well, this is outstanding. And Steve Mulder, thank you so much for coming on the show. It’s been a pleasure.

00:46:36.28 [Steve Mulder]: Great to be here. Thanks.

00:46:37.80 [Michael Helbling]: Awesome. Well, as you’ve been hearing, podcasts can be interesting to measure. So don’t hesitate to reach out on Slack or Facebook. Talk about it some more. If you’re not on the measure Slack, then you probably ought to be. And since we’re talking about podcasts and you want to jump over to iTunes and rate the show, thanks for that. We appreciate that. And, you know, give a couple of listens to some NPR podcasts. I think Steve will know exactly where that bump in traffic is coming from.

00:47:10.48 [Michael Helbling]: That’s right.

00:47:13.37 [Michael Helbling]: That’s right. So, for Tim Wilson, my co-host, this is Michael Hobley, and everybody keep analyzing.

00:47:26.28 [Announcer]: Thanks for listening and don’t forget to join the conversation on Facebook, Twitter or Measure Slack Group. We welcome your comments and questions. Facebook.com forward slash analytics hour or at analytics hour on Twitter. Those smart guys wanted to fit in so they made up a term called analytics. Analytics don’t work.

00:47:51.63 [Michael Helbling]: I’m either anonymous lemur or anonymous wolverine.

00:47:57.20 [Steve Mulder]: I like podcasts. I hear they’re going places.

00:48:04.62 [Michael Helbling]: Yeah, podcasts are something that, you know, we should all keep an eye on.

00:48:10.89 [Steve Mulder]: Only there was better content.

00:48:12.73 [Michael Helbling]: Yeah.

00:48:17.98 [Tim Wilson]: There’s second season kind of blue.

00:48:20.62 [Michael Helbling]: Well, so did True Detectives, so hey, nobody bats a thousand.

00:48:24.70 [Steve Mulder]: Yeah, but the Wrath of Khan was the best Star Trek movie, and that was number two.

00:48:29.05 [Michael Helbling]: Oh, good point. And Empire Strikes Back.

00:48:31.52 [Steve Mulder]: I’m just saying, number twos can be good.

00:48:35.25 [Tim Wilson]: I need to get your brother to come back down to Columbus. I haven’t seen him in three, four years. Yeah, he spent a little time in Africa recently.

00:48:42.10 [Michael Helbling]: I don’t even know if he’s back yet.

00:48:44.51 [Tim Wilson]: That’s good. You have a similar relationship to him that I seem to have with my sister.

00:48:48.32 [Michael Helbling]: Well, we talk sometimes. I just watch for Facebook posts.

00:48:52.71 [Steve Mulder]: Yeah, exactly. That’s right. So that was like right after like Excite and Ulta Vista. Oh, I could go, you know, down search engine history.

00:49:04.97 [Michael Helbling]: Also, my dog is deciding right now is the time to bark. That’s not good.

00:49:09.54 [Tim Wilson]: This does not happen in NPR’s DC studios. Exactly.

00:49:14.18 [Steve Mulder]: Although, it was like a family day, right? Like bring your daughter to work day, something. And somebody went into the studio and like hit a wrong button, a kid did, and all sorts of mischief happened. So, you know, it happens everywhere. Whose kid was it? Actually, I never heard whose kid. Maybe they were trying to keep it a secret. There was too much embarrassment, I don’t know. But the engineers fixed it like that.

00:49:35.20 [Tim Wilson]: It was really Bob Edward snuck in and he don’t did it.

00:49:39.19 [Michael Helbling]: Exactly. Bitter.

00:49:41.57 [Tim Wilson]: I’m a borderline unhealthy NPR fanboy, so… That sentence makes no sense to me.

00:49:49.92 [Michael Helbling]: Nice. So I’m looking here at PodTrack as we’re speaking, and Tim, we have a little bit of work to do to crack the top 10. We call ourselves the number one explicit analytics podcast. That’s what we are. Explicit digital analytics. Yeah, digital analytics.

00:50:12.29 [Steve Mulder]: I think all you really need is this American Life to play an example episode, right, one weekend, and then you’ll be golden.

00:50:19.50 [Michael Helbling]: Well, I mean, if you’re doing all this testing on the NPR1 app, I mean, we should probably just sort of chuck one of our episodes in there and just see how it does. You know, maybe it’s the next breakout hit. I’m sure that the podcaster is just wine and roses.

00:50:40.77 [Tim Wilson]: Rock, flag, and podcast measurement.

Podcast: Download | Embed

Subscribe: RSS