#108: Smart Speaker Measurement with Steve Mulder from NPR

[music]

00:01 (Announcer): Welcome to the Digital Analytics Power Hour. Tim, Michael, Moe, and the occasional guest discussing digital analytics issues of the day. Find them on Facebook, at facebook.com/analyticshour. And their website analyticshour.io. And now the Digital Analytics Power Hour.

[music]

00:27 Michael Helbling: Hi everyone, welcome to the Digital Analytics Power Hour, this is Episode 108. Hey, Google, who’s the quintessential analyst?

00:39 S?: Tim Wilson is the quintessential analyst, you dipshit.

00:42 MH: Have you ever been disappointed in the results you get from your smart device that listens to all your conversations and reports it back to their evil robot corporate overlords? Or whatever is happening in that place. The latest rage in consumer electronics is definitely the smart speaker. And with any new tech comes new business opportunities, changes in user behavior and a new analytics horizon. Hey, Moe, do you own any smart speakers?

01:09 Moe Kiss: Yeah, it’s funny, I don’t at the moment, but I’m on the brink. I was actually looking at them just after Christmas, so maybe this will be my inspiration.

01:20 MH: Yeah, wait for Prime Day. They’re always on sale.

[laughter]

01:24 MH: No. What about you, Tim, what’s your inventory of smart speaker devices?

01:29 Tim Wilson: It is uncountable. We have one sitting around unused. I feel like we should just be using our Echo Dots for spare hockey pucks. And we added a Google Home over Christmas because apparently, there was a Black Friday deal, so we are conflicted in the kitchen between two different platforms.

01:46 MH: Yeah, and I’m much the same, Tim, I think you and I are in the same boat. I literally have an Amazon Echo Show still in the box because I’ve never found a room to put it in yet.

[laughter]

01:57 MH: So lots of Amazon and Google devices. So obviously, with all of that technology, we wanted to talk about how measurement is happening in this brave new medium. And to give us the 411 on this, we sent out an SOS to a prior guest on the show. You might remember Steve Mulder, he’s the Senior Director of Audience Insights at NPR. Before that, he was the VP of Experience Strategy IsoBar. But more importantly, once again, he is our guest. Welcome back, Steve.

02:28 Steve Mulder: Thank you, sir. A long time lover of smart speakers and of the Digital Analytics Power Hour in the house.

02:35 MH: Ah, thank you.

02:36 TW: Have you ever played The Digital Analytics Power Hour through your smart speaker? I don’t think I’ve done that.

02:40 SM: Every single night.

[laughter]

02:42 TW: Oh man, you gotta get to sleep somehow. If you’re not listening to “Scoots” and “Sleep With Me”, there’s always the Digital Analytics Power Hour.

02:49 MH: Oh. Good for [02:52] __. So Steve, you were back, you were with us on Episode 46 where we talked about measuring podcasts, but what have you been up to since? And this is sort of a leading question, but great way to start off the show.

03:05 SM: Absolutely. So the world of podcast measurement, we talked about last time, continues to evolve and occupy a lot of my time and you know, what, it’s really about where people are consuming audio and news and information content wherever they are. And the smart speaker, man, it has emerged as such a huge platform for us. I heard this stat that this holiday season, 8% of Americans got a smart speaker as a gift.

03:33 MH: Oh wow.

03:34 SM: That’s a huge number.

03:35 MK: Are you kidding?

03:36 SM: Yeah, how many other gifts can we say that of, right? How many other categories do you think…

03:39 TW: Yay! Consumerism. Oh sorry.

03:41 SM: 8% got the same thing, it just, it blows me away.

03:44 MH: Yeah, not since the Tickle Me Elmo, right?

[laughter]

03:47 SM: That’s a good one.

03:48 TW: So was the… You guys had the NPR One app before you dove into the smart speaker. And is there kind of a convergence of behind-the-scenes tech or development?

04:03 SM: Yeah, very much so. There’s… Well, we should talk today about all the different ways that NPR content can be accessed all on the old smart speakers, but NPR One is absolutely one of them. And it was in fact the first Alexa skill that we developed. And so NPR One, if you don’t know it, is a continuous listening experience that blends local and national content and rotates through your favorite newscasts, shows, podcasts, news stories and responds to what you like, right? And so that’s a… That began as a mobile app, and we have been using that base platform on Smart TVs and now smart speakers as well. So you can say “Alexa, play NPR One” and you’ll hear exactly what we’re talking about.

04:47 MH: If you’re listening to this through a speaker, then you might have just triggered a skill.

04:52 SM: Yeah, I’m wondering how many times can we trigger our home speaker? Yeah, exactly, on this one conversation.

04:58 MK: Okay, so what I’m probably missing, and maybe it’s because I don’t own a smart speaker, yet, it’s like how big is this? How many people are actually listening on smart speakers? Is this, I guess from an analyst’s perspective, is this something that at some point I’m gonna have to figure out how to pull into my analysis, or do you think this is only super relevant for NPR?

05:24 SM: Yeah, well, just a few stats first. So I mentioned that a huge number of people who got smart speakers as gifts over the holidays. Here’s another one for you: Comparing this past December to the year before, the number of smart speakers in the US, grew 78% in one year, the number of devices in American households. 53 million people in the US, adults, own a smart speaker now. That’s 21% of people in the US own a smart speaker. This is, I’ve seen data that basically the adoption rate of smart speakers has been steeper and faster than any other category of device than smartphones, than the telephone, that adoption rate. And that tells us something about the appetite for this way of interacting with your music, your news, your utilities, your house, your kids, you name it. When you look at some of the research about how families are using smart speakers, it’s transformative and people that start to use them many of them become absolutely in love with what they can do on a daily basis in their lives. And I’m… I think I’m sounding like an advertisement at this point but given the sales that you’re seeing about smart speakers, all the different platforms, I think that’s totally safe to say.

06:42 MH: Now fully 10% of those are going to Tim and me though, so I mean, I wanna like average out the rest of the country. But I totally agree with you. And I would say, Moe, one other thing that I’ve sort of noticed personally but I think is actually having a profound impact in a lot of areas, is actually, having a smart speaker has actually changed the way I type and search phrases in Google.

07:05 MK: Really, how?

07:06 MH: Because now, I used to just type for the phrase that I was looking for, now I type full questions most of the time. I just realized this like two weeks ago, because when I’m asking Alexa for something, I’m asking it a question or if I’m asking Google, my Google Assistant on my phone for something or whatever, I’m asking it a question, and I find that I’m now typing questions into the Google search bar in my computer. And I don’t know to what extent that’s happening to other people, but I’ve heard some anecdotal evidence that the way people are searching is definitely being impacted by smart speakers and how people use them.

07:38 MK: But do you not think that’s also just part of people’s changing behavior towards using natural language and understanding that everything can use natural language now versus…

07:49 MH: I do, but I really think it’s accelerated by voice because I think voice communication is much more normal and informal and hence, just sort of natural. I don’t know, I’m not a linguist, but I think it’s influenced by it.

08:04 MK: I feel like we’ve been talking collectively about the move to voice interactions with your computer for like decades, it feels like, right? And… So I’m reminded, okay, I’m gonna geek out a minute here but Star Trek 4, the one with the [08:17] __.

08:17 MH: That’s right.

08:17 S?: Oh… [laughter]

08:19 SM: Where Scotty is talking into the mouse, and you know it comes back in time like…

08:23 S?: “Computer.”

[laughter]

08:24 SM: Yes, exactly. And you know we’re all laughing, because the promise of people like talking to their computers has been always on the horizon, but one thing that we’ve seen happen and research has borne this out, that people who own smart speakers start using Siri and Google Assistant on their mobile device, and voice interactions in their cars. That seems to be a tipping point to getting people towards voice interactions with everything. So at the Consumer Electronics Show, my colleagues were there recently and they told me about this trash can you can get for your kitchen that is voice activated. So you can say “open can” and not have to touch it to throw away stuff in the kitchen. So will this become that ubiquitous? Will I be wandering around my house talking to the parts of my house on a daily basis… I don’t know but I do think we’re at a tipping point and smart speakers in so many people’s homes is part of that tipping point.

09:18 TW: But backing up to where Moe… Where you were starting… ’cause I do think there’s an interesting part, if you’re a media… If you’re a content creation, which is fundamentally what NPR is, there is that direct opportunity, I mean you guys are even moreso tied because you’re historically an audio format. So that’s about as naturally to me aligned with the smart speaker as possible. If you go back to the early days of the internet, brands weren’t thinking about being on it or kinda where their place was. Now we have obviously Amazon was developing the platform to enable the shopping experience through their channel. But I am kind of intrigued and I just kinda lack vision of where does a… Where if you’re selling wine or you’re selling shoes, or you’re selling apparel or you’re CPG you’re selling toothpaste, one, there’s a question of, “Is it like a mobile app where you say, ‘Well I’ve gotta come up with the skill that gives… That provides some value that is an excuse for somebody to interact.’”

10:22 TW: I have to go out and come up with a reason to engage with that channel, or where, Michael, you were heading around this is changing my search behavior. My understanding is that there is that convergence of the… The answer box in a Google search with what Alexa or Google is gonna return to you. So there’s actually kind of a content marketing, SEO component out there where you may not be producing in audio, but you may need to have an eye towards, can this be found in audio? And I have no idea if there’s any… If I ask a question that Google returns and it’s basically about a brand, there’s no way for that brand to ever, right now, to actually know that they were ranking in the first response for the Alexa skill of search. To me those are like two very different things. But I do struggle with where, to Moe’s question of, if I’m an analyst at a brand, when is this gonna hit me as an analyst? And does it depend on what my company is doing and how likely they’re to be on the adoption frontier?

11:37 MK: Yeah, I’m just gonna add to that, ’cause I was thinking about that exact example of “Hey, Google, order me more wine.” When will I get to a point where that’s now another channel that I need to factor in as like, “Hey that drove this transaction for my customers.” Yeah, I don’t know how far we are from that and how the hell I’m gonna get to it.

12:01 TW: That totally seems like, given your wine gallery gig that you could… That could be something…

12:08 MH: A new frontier.

12:09 TW: You could do. That’s a way to get… Just like a lot of what NPR has done has been to get captive users by saying, “We’re gonna be more where you need to be when you need it.” Right?

12:20 MK: Yeah, yeah, but yet I’m really keen to hear your thoughts, Steve.

12:24 TW: [laughter] The guest.

12:24 SM: I see, like with any new platform, there’s a land rush for brands. Where they try to… They see where the consumers are going, and they try to figure out, where can we create something unique and valuable that supports our brand, that’s consistent with what we’re trying to do, and that drives x sales, listening in our case, whatever they’re trying to do. And so you’re gonna see, just like we have with other platforms, and I could use examples that kind of flopped, but maybe I should avoid talking about things like Second Life where you get land rushes for these new platforms and sometimes they pan out, sometimes they don’t, but the rate of adoption here speaks for itself.

13:05 SM: So we’re gonna have a lot of different companies creating unique Alexa skills, some of which are gonna be simple little games just to grow their brand recognition. Others are gonna be more practical, that we’re gonna to care about driving our business. A lot of it’s gonna be though, media consumption, right? You’re right, this is where NPR’s sweet spot is, in terms of creating and distributing great content, and audio is our sweet spot. So, of course, smart speakers are gonna be critical for us and the value for us is really obvious and has been from the start. For other companies, for other brands, it’s gonna be more of a question mark. And it might take a while to figure out where those sweet spots are.

13:46 MH: But now that you’re talking about it that way, and I’m thinking, well, there are content marketers out there saying, “We’re gonna have a really active blogging,” or “We’re gonna be the experts,” or “We’re gonna have a YouTube channel,” anything that’s “We’re producing content to build trust in our brand”, I guess that’s where, so skills versus… I actually don’t know enough. I know with Alexa… With Amazon, you’ve got skills that you kind of have to activate. I’m actually not that clear whether Google is a similar world, but if I was an analyst and my marketer is saying, “We wanna get into this and hey, we’re gonna take our video content or our written content, and we want to be able to deliver it, not as a podcast, but through this audio channel”, would they need to wind up developing a skill?

14:35 SM: Yeah, that’s exactly right. So skills on Amazon Alexa, actions on Google Home or Google Assistant, and that’s exactly a good way of looking at it. And keep in mind that more and more smart speakers now have screens. So what used to be an audio-only experience is already broadening out. So what you’re describing about companies using video, for example, suddenly the smart speaker is an obvious distribution point for them and no longer kind of a strange one as an audio-only thing. Now, it can do everything.

15:08 MK: But why has it got a screen? [laughter] I’m so perplexed.

15:13 TW: Moe…

15:14 SM: So I can talk to my grandma.

15:17 TW: Yeah. It’s…

15:18 MK: But then what… Isn’t that why the iPad existed or like why we have a smart tel…

15:23 TW: No, but you say… You say…

15:24 MK: My mind is blown.

15:26 TW: Hey Google, show me pictures of what this weird cake cutter looks like. This was actually a Christmas Day example when my daughter was trying to assemble something and we were like, “We don’t know what this is supposed to look like.” Hands-free, she just talked to it, it popped it up and we were like, “Oh okay, I see how this is supposed to work.”

15:43 MK: But won’t that eventually just be on your phone?

15:45 MH: But yeah, you don’t wanna pick your phone up with your cupcake hands.

15:49 MK: But you’re worried, ’cause you’ll be like, “Hey Siri, show me pictures of this cake.” Anyway, this is… I digress.

15:54 MH: It’s about having more screens in your house than anybody else and…

16:01 MK: Yeah, I don’t think this has hit the Australian market quite as…

16:05 MH: Well, and actually that’s very true because I remember last year at Super Week there were very few people I talked to over in Europe who were using these devices yet, so I think the market penetration in the US is very distinct, and it hasn’t really blown up a lot internationally yet. Although, I bet you, Steve, you may even have some insight on that because I don’t know how many people who internationally listen to NPR, but I bet it’s a lot.

16:31 SM: It is, I actually don’t have numbers in front of me on adoption elsewhere. I know that it’s relatively high across Europe, but I don’t actually have those numbers on me.

16:41 MH: Oh, okay.

16:42 TW: Is the ABC, I mean, Moe you haven’t, I guess haven’t heard anything… Now I’m curious, I wanna go see if the ABC is diving into that or the BBC, for that matter, for… So I can hit two other continents and not sound like a dumb American.

16:55 MK: I would be surprised if they were… I mean, I just use the app…

17:00 MH: So let’s get into some brass tacks on just measurements. So I think a lot of people don’t know. First of all, instrumentation, how do I get measurement attached to, let’s say I’m a company and I build a skill or an action, how do I put measurement on top of that? What’s that process like?

17:17 SM: Yes, welcome to the Wild West.

17:19 MH: Okay.

17:20 SM: It’s not that bad. Okay. So to go there, we should talk about the different kinds of experiences around NPR content that you can get on a smart speaker. So you can get newscasts. So, there’s hourly three-minute newscasts that NPR creates that you can get through your smart speaker. You can listen to a livestream of your favorite public radio station via smart speaker. You can listen to a particular podcast via Alexa or Google Home, or other devices, or you can open up one of our custom skills, and these are the things that I think are gonna proliferate more and more from brands. So NPR, we have three skills. There’s an NPR skill that I can tell you about, there’s an NPR One skill and also there’s a new one we just launched, which is the “Wait Wait… Don’t Tell Me!” quiz. So you can say to your Alexa, “Open Wait Wait Quiz,” and I’m hoping there’s another example of us suddenly deploying Alexas all across the world.

18:15 MH: That’s right.

18:16 SM: It’s something that just went live both on the Alexa and Google Home platforms, and so measurement across each of those varies.

18:27 MH: Okay.

18:28 SM: And we can drill into these one at a time, but we can go anywhere first that you’d like to.

18:32 TW: Well, before we hit the measurement, is, when you say a skill or an action deploying on the two, like that feels like so familiar to the iOS and Android. Is it, when developing mobile apps, is it for now, when a smart speaker deployment is happening, there has to be a decision, are we doing this for… I think right now, Amazon still has a much larger share, but Google’s obviously growing and Apple’s got HomePod. Are those completely independent code bases? Is it there’s some base and then the measurement goes on top of it, when does it fork?

19:11 SM: Yeah, it forks pretty early. So, they are different platforms effectively. And your iOS and Android comparison is a pretty good one. And that’s true for developing, development of skills and actions, and it’s also true for on the measurement side, where each of these platforms has its own nomenclature, they have their own platform in terms of measurement and the metrics that they report on and level of consistency around how those metrics are delivered as well. So it forks pretty early and from an analytics point of view, it’s that’s part of the challenge is trying to normalize data that’s different platform, different standards for producing different ways of actually getting the data. It makes our lives a little complicated. Now, we’ve found ways to get around that a little bit in certain areas, but looking at these skills themselves, it’s not as simple as I think we all would love it to be.

20:02 TW: So how does it work? Is it your programming in specific action? I mean…

20:07 SM: Yeah, so. Okay, so for the skills themselves, we’ll start there. For the skills themselves, out of the box Amazon and Google provide baseline reporting that report back on users, all the things you’d expect, users and sessions, but the magic really is around utterances and responses. So think about an utterance as a, “Alexa, do this thing,” and a response is what Alexa would say to you back. So that back and forth, seeing getting the data on those inputs and outputs, it’s kinda like search analytics, right? Like when we had great keyword data, it’s actually pretty similar to that. So there’s a lot of rich data there, and that comes out of the box from Amazon and Google, and it’s super powerful and we use that all the time for optimizing the experience of our skills. So we get that data, but what we have found is on top of that…

21:00 TW: But you get the utterances and responses that’s native as well, that’s the equivalent… You get that with your base analytics, or you have to actually add in…

21:08 SM: Yeah.

21:08 TW: Okay.

21:09 SM: That comes, yeah that comes standard, and it’s reported back through the Amazon Analytics interface and Google has a very very similar thing.

21:16 TW: But is that a web interface, or are you able to pull that data programmatically?

21:21 SM: It’s a web interface, so the analytics are developing and maturing just as the main platform is. So we’ve seen Amazon analytics get slowly better over time already, and now there’s at least an export ability, so it’s evolving as you might expect.

21:38 TW: And do they layer on any sort of aggregated supplemental information they have about that ’cause obviously Amazon and Google both know a lot about the user? Do you get any geographic, demographic any sort of other… Can you slice the utterances by gender or anything like that?

22:01 SM: Not yet.

22:03 TW: Okay.

22:03 SM: And this is where we’re doing a whole lot to supplement this with our own research. So we’ll be doing our own surveys and interviews and usability tests with our audience, so that we can try to collect and get a 360-degree view of these listeners or these users. But it’s early days on the analytics side, so we’re kind of putting together the puzzle if you will.

22:26 MK: And so, you can’t pass. I’m assuming that you can’t pass any kind of attributes yourself like a custom dimension or something, like if you were able to link it to someone’s account or anything like that?

22:38 SM: Not at this point. However, that’s changing. So one of the things that we did was, we realized we want a cross-platform view of this world. We wanted a single source of data that crosses all these smart speaker platforms so we can aggregate, for example, how many people and how many hours are being spent across smart speakers with our content everywhere. So, we’ve done a few things there. One is actually used the Google Analytics measurement protocol to send pings from the smart speaker when certain actions happen back to GA so that we have another source for routing our data through it. And then aggregate the data there as well. So the things we’re using that for specifically is classic audience reach and engagement kind of measures, so that we can look at across all these smart speakers, how many interactions, sessions are we having? How is that trending over time across all these platforms, across all these skills? What are we learning about time spent per platform, time spent per user in a given session? All the sort of classic things that we do in Google Analytics anyway but applying that to smart speaker interactions.

23:47 SM: So, our analytics now are kind of a hodgepodge of skill-specific things from the vendors themselves, from Amazon and Google, adding to that from Google Analytics, all the standard users, reach, engagement kind of things that we would expect from there. And then on top of that ’cause it gets even more complicated. I mentioned earlier that there’s a lot of people that use smart speakers, and they don’t go into our NPR specific skills, they just go straight to listening to their local or public radio station. They say, “Alexa, play WBUR.” And so they just get their local station, or they just listen to our newscast, they say, “Listen to my flash briefing,” and if NPR is in their flash briefing, then they’ll hear the NPR newscast. Because those things are outside of our skills, we need separate ways of getting to those as well and we do that through our other standard ways of measuring livestreams and newscasts. So, let me take a very short detour I promise.

[chuckle]

24:46 SM: So the way we measure livestreams is we work with this vendor called Triton, they’ve a tool called Triton Webcast Metrics and it’s a log-based service for measuring livestream audio listening. So basically, all it does is parse log files, and in this case it looks for smart speaker user agents, parses those out and can tell us exactly how many people are listening to public radio livestreams on smart speakers and when they listen, and so forth. So we use other third party sources to round this all out. So, this becomes very much a puzzle as you can already imagine, of at least four different data sources that we’re using to try to put this puzzle together.

25:29 MK: That sounds insane.

[laughter]

25:32 SM: It’s early days. I hope it gets better.

25:36 TW: On the live streams, is that… And I don’t know enough about how that… Does that mean NPR, you guys are actually hosting the servers that are serving up when requested to deliver the live streams and that’s why there is a log file that’s not… It’s not like a podcast, not the tease of future episode, but I download an mp3 file to my local device. You have no idea whether I’ve listened to it or not, unless there’s a new protocol for that, but again, that’s a topic for another show. When you’re doing live stream, is that always coming back to where you guys have or through Triton, have access to the log saying, “A request was made and X number of seconds were listened to?”

26:18 SM: Yeah, you can think of it… Yeah, you can think of it the same kind of way as podcasts where you’ve got live streams hosted on CDNs scattered across the country at individual NPR Public Radio stations, and then Triton has access to those log… All those individual log files and is aggregating and parsing those log files.

26:35 TW: Okay.

26:36 MK: So can I just go back to the utterances? ’cause that to me is incredibly interesting. So when that comes through, is that almost like an open text response or each of the vendors like do some type of cleaning, so that you can understand the basics of what they were asking? Like how… Can you talk me through what the data actually looks like for that?

27:00 SM: Yeah, so the vendors in this case are absolutely doing their own parsing of what they hear, I mean they’ve got… Obviously that is what their technology, their platform is best at, right? That recognition and parsing into language and storage of that and then what we need to do on the skill side is create the responses that respond to all of those stated intentions and there’s where it really becomes more of a Choose Your Own Adventure kind of book, or maybe I should say, Black Mirror: Bandersnatch.

[chuckle]

27:28 SM: Right now. It’s in terms of all the trees that we need to design out to think about the potential utterances that the consumers say, the responses that we can give back to those sets of utterances and building out a logical tree that represents all the possible paths that a given listener can take through that experience.

27:46 TW: Which I will say having played the Wait Wait… Don’t Tell Me! Quiz, it is… It’s a little mind-boggling when you think about if you’re on a quiz show and somebody says, “What’s this?” And then you give an answer and they’re like, “Oh I’ll accept that.” Right? ’cause they’re… If the answer is Queen Elizabeth, you know they’re probably you could say Elizabeth or you could say Queen Elizabeth or you could say, the Queen of England, and…

28:07 SM: Right.

28:08 TW: It was kind… It was intriguing ’cause there are gonna be a couple of questions where you think, “Wow, I can think of a bunch of different ways to answer this. And it did feel very natural to… I mean my daughter blurted out what the answer was, and I was like, “Oh we’ll see if it can process that. I think that’s right,” and it did. But you’re saying that utterance, what you get… The utterance is technically what Amazon or Google, the words it parsed into text, to kind of exactly as uttered and then it’s the skill would say, “Okay, we’re gonna translate that into whatever needs to happen.” That’s like… That’s neither the utterance nor the response that this is what our conclusion or what we think that utterance meant?

28:53 SM: Yeah, yeah, yeah, exactly. And so it takes a lot of good collaboration back and forth with Amazon to resolve any squeaky points there in that kind of hand-off, as you can imagine. Well, one of the tools that is also in the Amazon Analytics that has really proved valuable is pathing. So actually seeing the most common pathways through our skills. If you imagine this Choose Your Own Adventure tree, we can actually see where people are dropping off or going down side tunnels, were failing to give an answer that Amazon understands and we can then tweak our “interface”, [chuckle] our voice interface to better respond to that. So here’s an example: So if you go to Amazon Alexa right now and you tell it to play NPR, what it does is it opens up the NPR skill, and it says something like, “Welcome to NPR, tell me the name of your station,” or say “Browse by location.” And if you browse by location, you can browse by zip code, or give it a city and state. So there’s a whole pathway built out right there.

29:55 SM: And when we launched this skill, our hypothesis was that people would either know the name of their station and we’d be off and running, or they could pick from a list, and we thought if they’re picking from a list they’ll probably be more comfortable with zip code so let’s lead with that option. So we launched that way, lots of back and forth with Amazon, but launched with that and then looked very quickly at the data to look at the pathways in analytics to see if our hypothesis was right. And it turned out yes, between naming their station or putting in a zip code, that was covering the vast majority of people. But what analytics told us soon after that is there were a lot of people who, when we asked the name of their station, they dropped off, and they gave an answer to the name of their station that didn’t make any sense. So when we looked at the data and looked at the actual utterances, we realized a lot of people when asked the name of the station, were entering NPR. They didn’t really know the name of their local station by call letters or whatnot. So it was great intel for us.

31:00 SM: So, in terms of product, we immediately turned around, went back into the flow that we created, the logic and we changed the prompt. So if the first time around we don’t recognize the name of the station, well now we’ll say something like, “Okay, let’s help you find your local NPR station, tell us your zip code.” Because we know if we can prompt them with a zip code. That was the path that led to best success. So we can use the utterances and use the messaging back to guide them to a pathway that we found through data is the most successful. So it’s kind of a simple example, but it’s a way where the combination of analytics we get has helped us immediately make improvements to the experience.

31:40 TW: And are you visualizing the paths through a Sankey chart or through… I mean does it?

31:45 SM: Yeah, that’s exactly right, and that’s exactly the visualization that’s in Amazon’s analytics reporting right now.

31:52 MK: Wow. So I’m really curious to hear from a business perspective. So you mentioned that you’re sort of looking at audience and reach and engagement, and some of these metrics, what are some of the questions that the business are asking you about this, as it’s in kind of its infancy?

32:09 SM: Yeah, for sure. One of the main questions is, first, is there sufficient audience scale in this new platform, this new space to justify the level of investment that we’re already committing to, right? So, when… Here’s an example, when the smart speakers started coming out and had screens, this was new, and we had folks asking, “Should we come up with a visual version of our newscast for those particular devices?” And that’s immediately a question around investment as you can imagine our default NPR newscast is audio only. So… But looking at the scale of the audience, looking at the growth rate, looking at the research that supported adoption or especially over the holiday seasons, these last two holiday seasons, and looking at also… We do research with folks that listen to our newscast to understand how they use the devices and what they value, and when we asked, for example, people that listen to our newscast already, 92% of them say that it’s very or extremely important to them. That newscast delivered via smart speaker. So that’s a huge number. And we also know that that newscast is reaching a lot of people that don’t otherwise listen to NPR.

33:28 SM: So in terms of our organization’s goals of broadening our audience, creating a more informed public, getting our content out there, doing the visual newscast based on all those data points becomes a no-brainer. So we started working on one and now we’ve got a visual newscast that is delivered through Alexa on a pretty regular basis. And we use data to determine what times of day make the most sense to invest in a visual newscast versus an audio-only newscast. It’s, again, such one example of how we use data and think about smart speakers from a business perspective about where we invest.

34:07 TW: So what other… Are there other parallels to classic web analytics? You mentioned pathing. It sort of feels like the old null search results, is there the equivalent of there was an utterance and we had to give the… I’m sorry, to your example of, “We don’t understand that or we don’t have that,” is that… Are there other parallels to, call it traditional web analytics that you feel like you stumbled across?

34:32 SM: Yeah, for sure, I mean the sort of unknown, other, unidentified, crops up everywhere. One of the… Another challenge that we have, it’s just less parallels of web analytics than other domains that we deal with all the time is, when trying to identify in other data where smart speaker listening is happening, the user agents that smart speakers use are a little less than perfect, so it makes our lives a little more complicated, just like our lives are already complicated in the land of live streams or podcasts. But it’s fascinating how much is similar, ’cause when you think about your experience running through a smart speaker skill, it’s an utterance, and a response thinking about those as page views and thinking about a session as the overall experience you have in a given interaction. You can imagine similar kind of metrics around bounce rate occurring, when people give up on an experience and if Alexa doesn’t understand them the first time around, and they give up, that sure seems like a balance. So there’s a lot in the experience that is pretty similar when you think about it. And that’s helped us get our arms wrapped around how to use this data as well.

35:47 MK: Do you think that there’s also, I guess, the push from the business? I don’t wanna say to repeat some of the same mistakes. But I’m thinking about that thing where people are like, “Oh, if only we could keep them a little bit longer.” And I’m like, “These technology is also a really good example of where someone might have a very short experience, but it’s a very good experience, like you know. I… For me, if I were to use that speaker, it would likely be with cooking, converting units or setting timers, that… You know. Which are really short, sharp interactions. And I might have had a great experience, but from a business perspective, people are like, “Oh but we want them to have a deeper relationship, we want them to stay for longer.” Are you finding any of those conversations happening?

36:31 SM: Yes, yes, and in fact, as we look at defining what are the metrics that matter most that we’re gonna follow as an organization, we talk about a whole range of things. We’re interested obviously in the number of people who are engaged with NPR in whatever fashion via smart speakers. We do look at time because in terms of our content, traditional spoken word kind of content, the time someone spends with us is often a reflection of a good interaction.

[chuckle]

37:00 MH: You got the one case where time actually…

[overlapping conversation]

37:02 MK: I do agree. I do agree.

[laughter]

37:06 S?: And I love it.

37:06 MK: I do agree on that one. [chuckle]

37:08 SM: But I would say this, is that, even in our world, there is lean back listening and lean forward listening. So, if someone throws on their local KQED live stream and who wants to listen to the radio for hours and use their smart speaker as just a radio, that’s great and that’s lean back listening that we totally love to see. But we love even more when someone specifically asks for our podcast or pulls up that new Wait Wait Quiz and plays it. That’s an interactive lean forward experience that we value even more. So, all time is not equal either.

37:43 MH: Yeah, I mean it’s really interesting just to ponder other scenarios, like people who are selling things or how do you incent people to go get that skill or action on Alexa? And I wonder if we’ll even see things like custom or vanity URL type concepts. So, you know where they were when they ask for the skill, and that kind of stuff. So I just…

38:07 SM: Yeah, I think Alexa has… One of the biggest problems with smart speakers in general is the discoverability problem. You remember when we saw?

38:14 MH: Yes.

38:16 SM: When we got smartphones and apps started coming out and the Apple App Store became so big that there was no way to feature or find or discover new things, it was just overwhelming. Smart speakers is even worse because there is no… Amazon and other places, they’re trying to feature and surface all the activities and actions, and skills you can get.

38:38 MH: Right.

38:40 SM: But how do you, as a brand, how do you get the word out about your skill that’s not here?

38:44 TW: Well, Steve, you could email a link to people you’re gonna podcast with and then they’ll say, “That’s cool,” and then they’ll go… And for the first time in two years, open up the Alexa app on their phone and add the first skill in two years. So I think if you can just replicate that a million times, that’s the point.

[laughter]

39:01 S?: A little bit.

39:03 SM: Another great idea is, you’d go on a podcast as a guest and you keep saying, “Alexa, open Wait Wait Quiz.”

[laughter]

39:10 MH: That’s right. No, it definitely feels like we’re in that time of blue links on a gray page kind of a thing.

39:19 TW: But you’re moving towards attribution, right? I think about the NPR One app and how many times I’ve heard that touted when I’m not using the NPR One app. And so, it seems like you’re in a march towards the attribution problem of, “Okay, people probably didn’t just stumble across this. Was it because Amazon surfaced it better? Was it because we invested in promoting it through other channels?” And that seems like it’s gonna be really, really tough, but it’s interesting.

39:51 SM: Yes, it is. Yeah, it’s very tough.

39:53 MK: I also wonder whether Google and Amazon have figured out where the devices are in people’s homes, mostly. Like, would someone who has it in a kitchen use it very differently to someone who has it in their lounge room? Or like for me, I would potentially… Oh, this is awful. I’m gonna say this out loud. I might put in the bathroom because I always listen to The ABC in the bathroom ’cause I always listen to the radio while showering.

40:17 TW: That utterance was untranslatable.

40:18 MK: But…

40:19 TW: You know? It’s beans for dinner.

40:21 MK: But you know what I mean? Whether the way that you actually engage with the device would change based on where it is in the house, I think, yeah.

40:31 MH: I mean, just personal experience. We use it totally differently in different rooms of the house ’cause the kids use it for watching educational content and non-educational content as long as mom’s not around.

[laughter]

40:43 MH: And in the kitchen, it’s for playing music or asking questions. And in the office it’s, I think, just for show frankly, but…

40:52 SM: Yeah. [laughter]

40:54 MH: Every little place is different.

40:56 SM: Yeah, and we’ve seen too in our research that even time of day, you see different uses for the device.

41:02 MH: Yeah.

41:02 SM: So I have no doubt by depending on the person, their location, time of day, we’re gonna see more and more personalization around even how the default voices of these devices respond to people based on time of day, location, where they are. Yeah.

41:18 TW: Oh, I’m gonna… So going back to that very… So one thing you mentioned that Google using the measurement protocol, so one that has me thinking, what winds up being the VID, do you get… Is it like a MAC address or something like that? Is there a visitor ID? Or a device ID that come… What gets tacked into that request?

41:40 MH: Oh, yeah.

41:40 MK: It comes back to cross-device team.

[laughter]

41:45 SM: Yeah, good luck with all that. We don’t get much very helpful in that department. It’s very aggregated and rolled up in a way that isn’t helpful for us. There is a unique… So there is a… Now that you ask that, I don’t 100% know what that field is. I shouldn’t say ’cause I don’t know it off-hand.

42:08 TW: Okay, but I mean it seems good. It seems like because, Moe, you’re right. It’s like, can I roll up… Clearly with Amazon or Google, those devices are somewhat linked. And so if it was passing you that, you’d have a different… Although it’s a weird, it’s a household count, not really a user count. Whereas if it’s no, we’re just gonna give you this device then you’re stuck knowing whether people are listening to, “Wow, this odd person said to fast forward five minutes into the newscasts.” Well, that’s ’cause they were listening on another device or pick up where it left off. I guess that goes back to the early days.

42:45 MK: Yeah, that’s gonna be so…

42:45 MH: The idea of me being personally collected to that, no matter which Alexa or thing I’m using, is pretty cool concept.

42:53 TW: Yeah, except it’s you and Maria and the boys and…

42:57 MK: Yeah, but they all have different voices. So it’s gonna have to get to a point where it’s like, this is user one on the device, this is user two, this is user three. User one is also this person.

43:07 MH: Simple retina scanner solves all the problems.

[laughter]

43:12 SM: They’re getting there, they’re getting there. Did you hear recently that Google, or Amazon I think, that Alexa can now, if you whisper to it, it’ll whisper back.

43:20 MH: Yeah.

43:21 TW: Yes.

43:21 SM: They’re getting better at being more tailored and there were kinds of responses. So not only identifying different people by voice and being able to respond to them differently over time, getting better at that, but being able to hone how they speak. There’s actually, I just heard about Amazon rolling out a new voice so that when Alexa reads news, it’s now got a specific newscaster kind of voice that it’s gonna use. So they’re doing a lot of experiment now in what’s the appropriate voice and interaction to use given the context. We’re gonna see that just cascade. That’s gonna get more and more comment across all these interactions.

43:58 TW: Okay, one more…

44:00 MK: Okay, so I need to know, I need to know. Wait. But Tim, I feel Tim’s got a relevant question.

44:05 TW: Mine is very, very, very, very narrow in sight. Am I right in thinking you never get the actual audio, you only get the text conversion of audio, do you get any audio of the utterance?

44:14 SM: No, no audio. It’s all the text.

44:16 TW: So if you’re getting something of similar pattern that looks like gibberish, and you can’t really figure out what it is, it’s not like you can say, “Oh go and listen to it.” I can’t go and say…

44:27 SM: Correct.

44:27 TW: Okay.

44:28 SM: That’s right.

44:29 TW: Okay, that was it. Just a very tactical… Go ahead, Moe.

44:32 SM: And as a consumer of smart speakers, I think that’s a very [44:35] __ good mix.

44:36 MH: [laughter] Yeah.

44:37 MK: Mine’s a silly question. I’m just like, “Okay, if I’m gonna delve in and buy one, which one should I get?”

44:44 MH: Yeah, and it’s crazy because you have to figure out how it integrates with everything else. So that’s like work.

44:51 SM: Yeah, it’s a lot of what you wanna do with it at this point, right? ‘Cause Alexa got a good head start. Google Home is really good. They’ve come a long way, they’re growing fast like those… Those two just utterly dominate the space. Everybody else is very small at this point.

45:07 MH: Yeah, we started on Alexa and now we’re very much on the cusp of transitioning over to Google.

45:14 MK: I’m just really interested to hear a little bit about the types of people that are using it. Is it like my generation mainly? Or… I don’t know, because I obviously don’t have one.

45:26 TW: Well, yeah, you’re a what is it? You’re a Zennial?

45:28 MH: Oh, boy.

45:29 MK: I’m a zennial.

45:31 TW: That’s fake, fake news.

45:31 S?: My generation.

45:32 MK: I’m in my current generation.

[laughter]

45:34 MH: Pick a generation.

45:36 MK: But yeah, is usage pretty widespread by generation?

45:40 SM: Yeah, yeah, it is, it is, about half smart speaker owners are 45 or older. It crosses the gamut of ages and genders, and it’s really become pretty mainstream at this point.

45:53 MK: That’s super fascinating.

45:55 TW: Everybody, but it’s like I’m asking who uses smartphones, it seems like it’s whoever the early adopters or whoever it’s… Everybody’s gonna be talking about it. But let’s just pretend I didn’t say anything at all, Michael. And you just kinda had to wrap and I’ll cut that.

46:09 MH: No, we could… “Alexa, fix this and post.” Okay?

[laughter]

46:14 MH: Ah, yeah, we do need to wrap up. This is awesome, though, because this is such cutting edge territory. So Steve, it’s so great that you’re our friend, that we knew about this and got you on the show, so thanks for coming. One thing we love to do is do a last call, just areas of interest that we’ve seen recently. So, let’s go around the horn and see what people got for last call. Steve, you’re our guest. And we typically will start with our guests, so what have you got?

46:41 SM: Absolutely happy to. So Seth Godin is an author, philosopher, pundit, what have you that I follow on a regular basis. He recently blogged this thing that I really liked and it’s so short, I’m actually gonna… Actually, I’m gonna read it. So it was called, “Don’t Steal Metrics.” And here’s what he said, “A thoughtful friend has a new project and decided to integrate a podcast into it. Talking to a producer, he said that his goal was to make it, ‘The Top 10 Podcast on iTunes.’”

[laughter]

47:13 SM: Why is that the goal? That’s a common goal, it’s a popular goal, someone else’s goal but the compromises necessary to make his podcast that popular all fly in the face of what the project is for. It’s your project. It’s worth finding your metrics. I love that, right? ’cause it speaks to, when we start with all the time as analyst is always finding what are the right metrics for me in my moment for my project, and working with teams in each of that fundamental project?

47:43 TW: How much mediation did you have to do as smart speakers rolled out around what the right metrics were? Was that collaborative and converged? Or did you find that there was a chasing of vanity stuff that maybe wasn’t really your metrics for NPR?

48:01 SM: We learned as we went. There were a few on there that were vanity because people just want the big number of, “How many total hours are we listening to smart speakers on all of our content?” And there were others that were much more driving the business forward, driving our investment decisions, driving our development of our skills. It’s always a mix and it was a back and forth for us too, especially because the space is so new.

48:25 TW: Nice.

48:26 MH: Oh, that’s awesome. Moe, what’s your last call?

48:29 MK: So I’ve got a bit of an odd one, but it was a really interesting article, actually, that my new CEO put me on to. And it was published on First Round and it’s called, “Humans Hate Being Spun: How to Practice Radical Honesty- From the Woman Who Defined Netflix’s Culture.” So her name’s Patty McCord. And the reason I think… I just found it incredibly interesting from a cultural perspective, but I really was thinking about it from the lens of being an analyst. I love the concept of radical honesty, but I don’t feel like we can always be completely honest. And the reason I say that is, and we’ve talked about it a lot on the show, it’s like sometimes you actually need to watch what you say, and hold people’s hand and if you tell them that you’re uncertain of a number, does that impact your credibility or how much they’re gonna trust you? And so, I was just giving a lot of thought about is radical honesty something that we can actually always practice in our job? I guess, do we need to have a certain stance on it to make sure that we still have that good relationship, but then we can still be honest with our colleagues? I don’t know. Anyway, it was really interesting and led me to think about things a lot. So, yeah.

49:47 MH: I talk about all my problems with Tim. Does that count?

[laughter]

49:52 MK: I don’t think he wants your honesty.

49:54 MH: No, in fact.

49:56 TW: Unfortunately, he gets mine, but he has the right temperament to deal with it.

50:02 MH: Well, Tim, what’s your last call?

50:05 TW: So I’m gonna do a twofer, but one leads into the other. The first is, we always talk about the Measure Slack team. So in case you’ve missed it, if you’re in the Measure Slack and you didn’t see it, not for kind of ongoing funding but for a way to get to a higher level. There is a Patreon page for Measure Slack, patreon.com/measureslack. So if you use that community and get value out of it, and would like to see it be able to be sustained and grown in some ways, consider maybe making a nominal donation. It’s an ongoing, recurring thing. I feel like I should plug that, because I’m totally… My last call is something from the Measure Slack team, so I’m probably butchering the name, but Maciek Stanasiuk…

50:54 MK: That sounds… Yeah, you nailed it.

50:55 TW: Yeah. He posted, he wrote… It’s on Medium, a post called, “State of mobile operating systems privacy in early 2019,” and it was crazy thorough and informative. Basically, going back and [51:09] __ saying what Google and Apple both do, what they covertly do, what they don’t advertise they do, what seems good, what seems alarming. And it was just a really, really well-written, well-researched post. And I got that off the blogs and podcasts channel in the Measure Slack a while back. Michael, what do you have?

51:30 MH: Very nice. Alright, well, I’ve got a very self-serving last call. So, many of our listeners may already know this, some of you might not. We’re slowly making a migration away from Facebook, but we have opened up a LinkedIn group, which is growing a lot faster than we even realized. And so, if you’re not aware of it, we’d love to have you as a part of the group, and so feel free to join. We’re excited about the potential of moving that community over to LinkedIn versus Facebook in terms of communication. Obviously, the Measure Slack is still the number one place to probably communicate with the podcast. And then lastly, I can’t go through this kind of discussion without being reminded of a podcast and a set of articles that came out about a year ago, Tim. I know you’re gonna remember as soon as I say it, which contrasted these concepts of robot assistants and the voice they use. And so male and female voices, and why they are considered that way. And it really raises a lot of very interesting cultural questions about that.

52:41 MH: And so, we’ll dig up an article, there’s about four or five that I found that covered the topic and I know there was a popular podcast episode that I listen to about robotics a long time ago that also raised that issue. Well, I can’t think of it right now. It’s definitely bringing back to mind when Alexa has a woman’s voice and other robots are male voices like why that is, and some of the cultural norms that underlie that from a research perspective, it’s pretty neat. Anyway. All right. Well, thank you everybody. Steve, thanks again for coming back on the show. I know it’s very intimidating because of Tim.

[laughter]

53:16 SM: Yeah, you’re so scary here.

53:19 MH: No, it’s been such a pleasure. Thank you for sharing with us some of the things you’re learning as you dive into this world of smart speakers and assisted, whatever these are called now, whatever this category is called. I think we’re all headed there and so it’s pretty cool to get a little bit of an insider’s view, and we’d love to hear from you. Are you working with these technologies? Are you doing school stuff with them? Let’s keep the conversation going over on the Measure Slack and/or our new LinkedIn group. Or you can also reach out to us via our website. Well, as always, I know that Moe Kiss and Tim Wilson, wanna encourage all of you out there to keep analyzing.

[music]

54:05 (Announcer): Thanks for listening and don’t forget to join the conversation on Facebook, Twitter, or Measure Slack group. We welcome your comments and questions, visit us on the web at analyticshour.io, facebook.com/analyticshour or @analyticshour on Twitter.

[music]

54:23 S?: So smart guys want to fit in so they’ve made up a term called analytics. Analytics don’t work.

[music]

54:31 S?: Analytics, oh my god, what the fuck does that even mean?

[music]

54:39 MH: Oh, well, I was about to devastate all the other people. Give me a chance, Tim. Yeah, what does he know about analytics except for almost everything?

[laughter]

54:52 MH: Are we recording this, Tim? Oh we are, great.

55:02 MK: Oh, don’t.

[laughter]
[music]

55:03 MK: So… Oh man, okay, I’m not gonna keep talking about this or Tim’s gonna yell at me. Tim, do you have some new camera? You look like really dreamy, like you’ve got that special video mode.

55:17 MH: A dreamy camera?

55:19 MK: Look how… He looks dreamy, doesn’t he?

55:22 MH: Wow. [laughter]

55:22 TW: I’m so glad we’re recording.

55:24 MK: Mine’s shit but I’m going with it.

55:28 S?: You know what’s gonna be hot in 2019? Smart speaker measurement.

55:33 S?: Sorry, I don’t understand.

55:35 S?: Tim Wilson is the quintessential analyst, you dipshit.

55:40 TW: I mean, I would go so hard nerd on NPR stuff that somebody else is gonna need to… I’m trying to be better.

[music]

55:48 MH: Okay, as the non-smart speaker owner, I think that’s appropriate. I’ll go ahead.

55:54 SM: I realize I forgot to tell my little thing about how I had this friend who loves his smart speaker and buys things with it. And so I like to walk into his house on a regular basis and say, “Alexa, buy 1 ton of cat litter and confirm.”

[laughter]

56:08 MK: Perfect.

[laughter]

56:11 MH: I’ve also already been the recipient of my son browsing through the Amazon account, adding 17 bouncy houses and ordering them. So, that is something that, you know…

[laughter]

56:26 S?: Rock, rag, and hey, Google!

Podcast: Download | Embed

Subscribe: RSS