0:00:05.9 Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. Here are your hosts, Moe, Michael and Tim.

0:00:22.0 Michael Helbling: Hello everyone. It’s the Analytics Power Hour. This is episode 197. You know, so much of the time, the limitations we run into as analysts are representing our creativity, not just in our analysis, but in our approach to storytelling and creating a narrative arc to the work we’re doing. And on this episode, we’re gonna take a, maybe a slight lateral jog into what might seem unrelated at first glance, but could end up being something that future generations of analysts and data scientists might use every day as tools to help stakeholders and business people understand us even better. So put down your Python for a minute and grab your loot of bardic inspiration and your cloak of stealthy approach. And let’s go on an adventure into the midst of AI and gaming. Tim, have you ever played D&D?

0:01:13.7 Tim Wilson: I’ve never successfully gotten into it. My 20 year old has played a decent amount, but I have not.

0:01:21.3 MH: And I feel like I might know the answer, but Moe what about you? Have you ever.

0:01:26.1 Moe Kiss: It took me a good, like 10 seconds to be like, wait what’s D&D.

0:01:30.1 MH: Oh, Dungeons and dragons. [laughter]

0:01:32.3 MK: Yeah, yeah. I got there. I got there. So this is impressive.

0:01:36.6 TW: I did oddly have like a whole set of like the pewter little figures for D&D.

0:01:41.8 MH: Oh, okay.

0:01:42.3 TW: But I never actually played it like, right. That would’ve been a thing. Would that have been a thing? Like I had a half dozen? Okay.

0:01:48.8 MH: Yeah. And I’m Michael and I have a level seven half orc Druid named folgar the crystaline, named after Folgers Crystals.

0:01:56.6 TW: Michael I’m so impressed.

0:01:58.5 MH: But we needed a DM. I mean, a guest, someone who could be a guide of sorts on this journey. Well, it just so happens that Hilary Mason is the co-founder and CEO of Hidden Door, a startup that is building a social role playing experience powered by narrative AI. She was the founder of Fast Forward Labs, which was later acquired by Cloudera. And she’s on the board of directors of the Anita Borg Institute for Women in Analytics. She is the co-founder of Hack New York, a nonprofit that mentors the next generation of engineering talent for New York’s Creative Technology Community. And today she is our guest. Welcome to the show Hilary.

0:02:34.3 Hilary Mason: Thank you so much. This is very exciting.

0:02:36.8 MH: I… Yeah, I get really stoked about conversations like this only because I started playing D&D during the pandemic. And as I was looking at your company’s website and sort of the things you’re working on, I was like, wow, this is right up my alley now. But I think that might be a great place to start is just talk a little bit about how this all got started and kind of, you’ve held a number of data science roles and data science leadership roles. But this is sort of a little bit of… Sounds like a little bit of a departure, this new venture, so I’d just be curious, maybe our audience could hear a little bit about that story, and then we can jump into more of that deeper technical stuff as we go.

0:03:14.6 HM: That sounds great. It’s actually not that much of a departure if you go back even further in time. So I grew up as a huge sci-fi nerd reading, reading endlessly, playing a ton of tabletop RPGs through college and graduate school and found that it was the source of some of my more meaningful social experiences, this idea that you sit around a table with your friends and together you create an adventure in a world that is exciting and interesting and motivating. And that it’s really about the creative interplay of having ideas together and making that experience together. And there are all sorts of things you can bring to it. Like you can do miniatures, you can Cosplay, whatever it is that works for you and your friends, but it’s something I’ve always loved. And so fast forward now to where we are today, I’m the co-founder of Hidden Door, where we’re building that narrative AI machine for playing through any story with your friends in any world.

0:04:18.8 HM: So imagine being able to take the world of a novel you love, or a TV show, you get really into and then you can all role play inside of that world together. And there are a bunch of things that bring us here. One of which is that I’ve had a tremendous personal interest and passion for this domain for pretty much my entire adult life. And on the other side, we also have the technical capability to finally make it feasible to actually facilitate some of these interactions in an automated way, in a way that was completely out of reach even just five years ago, from a technical point of view. And then to draw on your experience as well. There is a tremendous interest right now in this sort of experience, storytelling experiences, which may have been driven by the pandemic, but has been sustained as people really look for ways to have these sorts of creative social experiences with their friends and their family. So it really is the time for building this.

0:05:18.5 MH: Well, I think stranger things on Netflix also probably helped a lot of us sort of like reconnect with some nostalgia as well.

[laughter]

0:05:25.7 HM: In a good way and then a scary way.

0:05:27.9 MH: Right, right.

0:05:28.9 TW: Those of us of a certain age, I guess.

0:05:31.3 MH: Yeah, yeah. You gotta remember the ’80s, I guess.

0:05:33.2 TW: I mean, was there like the Hidden Door from, I mean, what I understand of what you’re building and even as you’re describing it but it seems like was there like a creative flash where you said, oh, I could do this. I am intrigued by just the inspiration of it seems both so ambitious. And then you’re also like, yeah, yeah but we have the technology to do it. I mean, not to any way… I mean, there’s clearly a ton of work to be done, but like, where did that spark come from, or was there a spark, or did it just slowly build?

0:06:11.1 HM: It was a little bit of both, in that this is not my first company, it’s the fourth, and so I have a set of things I look for in what I want to work on. I look for something that I’m very excited about, where there is that sort of step function in a technology that we can now go explore what it means to implement and build products around it. And also when we started, this was not necessarily going to be a game or even a role playing game, we started actually looking at applying the tech to things like organizing our personal information. So we communicate through like email and Slack and the web and text, and we can easily now sort of model that in a way that makes it much easier to retrieve. For a while, I was really stuck on the idea of building a new search engine, because I find that what Google provides these days isn’t working for me, and I really want to see information synthesized and summarized given my current knowledge levels, and we should have that.

0:07:10.1 HM: Somebody will build it, there are a few folks trying. And so my co-founder, Matt and I spent a while circling around what was personally exciting to us and what was technically possible and building some prototypes until we hit on something that we found really engaging, that when we tested it, our players found engaging, and then that it also has a few features to it, one of which is that the users or the players are people we wanna spend time with. So with all due respect to corporate CIOs and chief data officers, I spent the prior five years having a lot of steak dinners. And I’m pretty excited to spend more time with teenagers and folks who like role playing, those are my people. And then it also has a business model that is not exploitative or parasitic, one that I can be really proud of. Hey, I grew up in social media years and years ago, so I’m very sensitive to some of those things. And then finally is in that open space where… And I think I already said this, there is that technical jump, and we can invent the product, and somebody is going to do it. But now is the time to start to think about what that co-creative experience with the machine learning system can be and how you can use it to open up the spaces to have different kinds of social experiences with your friends.

0:08:35.6 MK: Can you explain to me… You kinda keep talking about this technical jump and things… There are things that we can do now that we couldn’t do five years ago and that you and your co-founder kind of assess things. What does that mean at a nitty-gritty level? What are the things that you’re doing now that weren’t possible five years ago or…

0:08:54.7 HM: Yeah, that’s a great question. And on the technical level, it’s really the use of things like large language models. And I wanna put an asterisk around that, ’cause we need to talk about how those things are used. And then thinking about ways we can do things like text generation and automated abstractive summarization as an achievable task, both from a machine learning algorithmic point of view but also from a affordability of the compute point of view. And then there’s another layer of even having access to sufficient data to start to build models of not just how is language used in storytelling, but how do plots evolve in different sub-genres, and what kind of item would you find in the pocket of the barista on the starship who’s making you your morning coffee, being able to actually inform models that can predict those things. And so yeah, there’s… That sort of gets to the heart of it.

0:09:56.5 TW: And it will actually do that? It will be that if you’re in a theme that’s a outer space starship, green aliens or whatever defines the theme, that it will generate prompts that are seen contextually appropriate?

0:10:13.2 HM: So yeah, I love that you’re having trouble even saying the words, and this is something that we’ve been talking about…

0:10:18.8 TW: I have trouble saying words all the time, if I could get an AI just to talk for me, that’d be perfect.

0:10:24.0 HM: But it… When we think about it as a technical problem, we’re essentially trying to say it generates things that makes sense, what does that mean? And so just to get to the heart of this, one of the challenges of building a product like this is understanding that there isn’t necessarily a notion of quantitative correctness. And so yeah, so you’re trying to build systems that make things that make sense in context, that people will find to be natural, and that’s something that has been really fun to work on as well.

0:10:56.9 MK: My mind is blowing.

0:10:58.9 TW: How do you measure…

0:11:00.9 MK: How do you measure that?

0:11:01.2 TW: How do you measure make sense-iness?

0:11:02.2 MK: Yes, that was my… Straight away, my mind has gone there.

0:11:07.2 HM: It’s how people react and respond to it, so it’s a product metrics question. But there’s… We do a lot more than that too. And I’m just gonna say that one of my hobby horses at the moment is that I think that data people tend to shy away from addressing problems where there is no notion of quantitative correctness. And there are a bunch of them out there well beyond the stuff we’re working on. If… Even if you think about web search, that is perhaps the canonical problem in which… Sure, we have some relatively poor metrics we can try to apply, and then we do a ton of AB testing when you have a scale product. But it’s hard, right? It’s scary. Same for content recommendation, for example. And so you end up falling back to product metrics. So like if Netflix, for example, is recommending that you watch a show, and you actually watch it, that probably… I don’t know anything about their internals, but probably is some signal that they’re gonna use to reinforce whatever widget or component it was that made that recommendation. But in our case, we look at it, we play test extensively. And then we do things like play 10,000 games with the same probabilistic seeds and see what the distributions of outputs look like and see if there’s anything in that that looks odd or feels weird.

0:12:29.5 TW: Because… But I guess that… Like take the… We’ll go YouTube, we’ll go with the more parasitic approach, where if you measure… If you’re trying to say, “We wanna provide an experience that’s engaging and people return to,” that’s where it seems like the… You want to have an experience that draws them back in, but there’s this getting sucked into the horribly unhealthy. What you have kind of very front-and-center is, “No, we don’t wanna do unhealthy, addictive engagement.” That seems like providing a really… Providing a book that I can’t put down, that’s generally considered a good thing, but in the digital world, a phone that you can’t put down is negative. It sounds like that’s the needle you’re very deliberately trying to thread… I think, is that fair?

0:13:27.5 HM: I think so, though and we would be so lucky as to have the problem of generating stories that you can’t put down, and we are certainly aiming at that. Where I think there’s a little nuance is whether the stories start to be… They drive their engagement through things that are damaging to the person consuming them, whether it be in the case of YouTube, misinformation or gradual extremism, and that’s where it becomes really deeply problematic or, not to just pick on YouTube, but encouraging body image issues and the Instagram posts, right? Those products can be damaging and you can see how they would get there because they’re optimizing for those engagement metrics. And there’s also a ton of research that shows that people tend to engage with things more when they are outrageous or provocative versus things that are sort of calm and make you feel a little happy about the world, right? And so it’s sort of the combination of those metrics with some of our unfortunate human tendencies that can lead these products to become quite damaging without any product manager ever saying, “You know what, I’m gonna build a thing that makes kids anorexic, or a thing that makes domestic terrorists.” Nobody… I believe nobody ever went in with that intention. So yes, we’re not doing that, I have to say that very clearly.

0:14:53.5 MK: But okay, Tim, I need to rewind. Back on the metrics thing, so I understand the relationship to product metrics, but maybe… And maybe this is ’cause I don’t understand the detail of the models that you’re working with, but you gave this example of a cup of coffee on a spaceship, I’m not gonna reference the actual terminology there, I’ll just say spaceship. It sounds like what you need to test is how normal, that is in that one experience. But how do you separate that from a good overall story or a good overall experience on your platform? ‘Cause I’m just making an assumption, right? That you have many, many models doing many, many little things.

0:15:42.4 HM: That’s true, we do. And there are a few things to tease out of that I think, one of which is something that we should be very clear about, which is that the model itself is not creative. It is essentially a trope machine, it is excellent at recognizing the patterns in similar stories and I mean patterns in the sense of the kinds of sentences that are written, so we can generate those sentences, but also the plots, the character archetypes, the personalities, the ideas that… For fans of science fiction, you know that when you have a story on a spaceship, there are things the author doesn’t really have to explain, like that you may have an engine that can go faster than light, is a trope you can build on as an author, and so these models get to be very good at essentially becoming these trope machines. And so it’s not to say that the stories don’t have to be good, though I don’t think they would stand alone as the greatest American novel and we’re not gonna be winning the Nobel Prize for Literature with what comes out of our system, what they have to be is memorable, but not so memorable that it’s chaos.

0:16:57.2 HM: So we think a lot about the tuning of the ratio of surprise and delight with things that you expect to see because of what the tropes would predict. And doing that in the context of the kind of story you want to be playing. So is it a comedic story, is it a dramatic one, is it one about… Are you role playing in something like a Bridgerton type world where it’s about relationships and social status, or are you out there in your traditional fantasy setting whacking the bad guys with a sword to get the treasure? All of these things are possible, but it’s really trying to tune that and also realize that this is a social experience on purpose, you can play it by yourself, but it is really better with friends, because the thing that makes it special is not that we need more stories in the world. You can just go to the internet and it is full of fan fiction, it’s wonderful, but we need your story, the one that’s meaningful to us as we’re creating it together.

0:18:04.1 MH: So I’m dying to ask you how you handle, say like in the fantasy context, magic systems, because a lot of times the stories can be very fluid, but the systems have to be very locked down. ‘Cause you just have to apply… Anyways, that might be a really dumb question, I don’t know, but I’m just thinking about it, I was like, “How do you handle that?

0:18:23.1 HM: No, I love it.

0:18:23.6 TW: You’re thinking Brandon Sanderson or… I’m trying to…

0:18:25.0 MH: So we are… So I’m doing a one-shot with some friends right now where we’re doing it in the world of one of Brandon Sanderson’s novels, the Mistborn series, and so we’re having to make up our own rules about… “Okay, yeah. So all the… And sorry for listeners who are not familiar with Brandon Sanderson. A, You should start reading Brandon Sanderson, awesome author.

0:18:43.3 TW: You should start reading Brandon Sanderson.

0:18:44.6 HM: Hard agree.

0:18:46.6 MH: And B, that’s a great series to start with if you’re into it. But anyways, the metal, it becomes the basis for their magic system in that particular set of novels. But applying it to our game, we had to create a structure around that.

0:19:00.9 HM: Right.

0:19:01.7 TW: Is that a D&D game or some other game.

0:19:03.8 MH: It’s basically… We’re making it up as we go along. So it’s built on the concepts of D&D, but our DM basically has created a layer to support that world, so yeah. But this sounds like a system that would literally build that for you, which is just blowing my mind.

0:19:20.9 HM: Yeah, so what we’re doing in our particular approach, is that there is a stats-based system and maybe it’s… When I put that little asterisk around large language models, we’re coming back to it now, because the core problem we’re solving is not one of put text into a model and get text out because that’s not controllable, and it doesn’t necessarily follow the laws of physics or the laws of magic for the world you happen to be playing in whatever those may be. So what we’ve done instead is essentially cut the problem in half and solved the problem of unstructured input to structure, and then we have a simulation engine and a bunch of simulator chunks you can throw in and be like, I want this world to have this sort of magic thing and this vocabulary and dressing around it, and it should work this way, and then we take that structure and we render it dynamically in text an art sort of like a graphic novel that comes together as you play. It’s a very different approach than just putting something into a large language model and taking the output and showing it to people, which by the way would lead to awfulness, and so that’s how we handle the issues of different worlds, having different…

0:20:35.4 HM: We’re play testing a lot with our superhero world, so they are super powers and they work a certain way, but a Mistborn world would have a certain set of… A certain kind of magic and a certain currency for that magic, and so you can pull that chunk of simulation in and be like, Yes, this one for this experience.

0:20:53.4 TW: So there is kind of a finite set of those simulation chunks that you can… If I said I wanted to throw some Douglas Adams in with some Isaac Asimov or something, if I wanted to have the Rules of Robotics in the world, so is that, what your… And that’s when you said you’d run 10000 simulations with the same seed. Is that saying, I wanna take some basic simulations and run them and see if it generates things that are different, but that make sense?

0:21:25.7 HM: I mean that’s how we would test it, but… Yes, so our system, you put in your input, so in this case, so we’ve designed a UX where you throw a few words in it and it writes sentences for you and you go through ’till you get a sentence that describes your character’s intent, we call it, and it can be as lyrical as you like, or you can just throw a muscle emoji in there and a sunglasses emoji, and it’ll be like, Cool, you’re doing something muscly and sunglasses-y and it’ll fill it in for you. And the idea was that you should be able to have your character do whatever you want them to do with just your thumb, and as quickly as possible, we take that intent and then we model it into our simulation, which does in fact have a somewhat… It’s not finite because of the way it’s expressed, but there is an actual manipulation of numbers and stats that happens based on what goes in their rolling of the virtual dice and what comes out the other side, and then what comes out the other side gets rendered for you in text and art. And so, yes, it is coupling that structure to the sort of language and narrative piece, and it does require…

0:22:31.7 HM: I think it’s a much more robust approach for what we’re trying to accomplish and the experience we’re building, but it does sacrifice things like if you are using a raw language model, you’d be able to say like, and now my orc character has the baseball card of the person who won the World Series in 1974 and actually get something back, whereas our system will have no notion of what a world series is. So you sacrifice some of the generalized ability in order to create something that is more like a game than just sort of playing with NLP models?

0:23:04.7 TW: Well, so maybe this is… I don’t know if this is a fair question, but I’m curious, when it comes to build versus buy, it feels like the crude way to put it, but as you’re describing it, there are things that are very much… You’ve said you have had to… You’re building them because they’re unique and different, but presumably there’s also backbone and stuff that… Where does this live? Is this living in some cloud environment, what are the pieces that are like… Well, yeah, yeah, this stuff, the storage is gonna be AWS or whatever, what are the pieces and what is it that you’re doing versus what is it you’re like, Yeah, this stuff is just run-of-the-mill and we can tap right into existing services.

0:23:54.4 MH: Yeah, I mean, I am a huge fan of not building anything, you cannot build unlike any… The fewer lines of code we have to maintain, the better for us, and of course, we are building on the shoulders of a ton of amazing open source work and open model work and data that happens to be out there, and then hacking and making these things work for us, so we have collected a data set of… At this point, millions of stories, including books and also a bunch of just fiction stories that people have written and put down on the internet, we have our own GPUs. In fact, right behind me is a machine with a GPU I bought from a guy off the internet, because it’s really hard to actually come by very beefy GPUs these days, where we do a lot of our experimentation and training, and then one thing I didn’t mention.

0:24:47.9 TW: That’s like the modern day equivalent of the Pentium 5 under my desk that’s running in the website, that’s the…

0:24:56.1 HM: Well, in these days of being a fully remote company, when my co-workers are actually running big models and I hear the fans spin up, I don’t feel so alone in my home office. Like in a way they keep me company. But one thing we do is push a lot of our intense computation offline, so while we’re giving all our secrets away in terms of our live language generation, we actually do most of our generation offline generate tens of thousands of candidate sentences for each plot point, and then templatize those and then at runtime, we rank them and fill them with the stuff in our game at the moment, which is actually how language generation worked before large language models existed, ’cause I’m very old and I built some of those things back in the day. But that actually makes our compute needs in production fairly small, so we’re still running on a hosted service called Render, where you just create a… We have a monorepo, you make a YAML file and then it deploys all your lovely Postgres DBs and services out there, and that’s been working great.

0:26:02.9 TW: So one of the things that you… I think it was from the… One of the blog posts from early last year, was you talked about the see yourself, the pushing against stereotypes and lack of diversity. So how… And maybe this is kind of back around to the same sort of question of how do you let… How do you provide creative freedom without getting things… Without having system biases or other stuff creep in? If you’re pulling in all of these texts, do you inadvertently wind up with texts that are misogynistic and biased, and they’re gonna start pulling you?

0:26:45.6 HM: That’s a good question.

0:26:46.7 TW: How do you kind of… How much do have to very, very deliberately say, “We’ve got to make sure we’re looking out for these things that AI tends to get sucked into.”?

0:27:00.1 HM: Yeah, that’s a wonderful question, because it is a very real danger, building anything with large language models, which are largely trained off all of the English on the Internet, a lot of Reddit. And as you can imagine, anything trained off that data does indeed replicate and even magnify the patterns of toxic and really inappropriate communication that we very consciously do not want. And so we’ve made a few decisions about that, one of which is the reason we have the architecture I just described to you, where we’re not generating text out of a generative model in production, it all happens ahead of time, it gets templatized. A person reads that template and makes sure that it does not contain anything that we would not be proud to put in front of some… And that doesn’t mean that it couldn’t end up generating something completely inappropriate, ’cause again, templates get filled with variables, and you don’t have complete foresight. But at least we have a layer of protection, we have a bunch of words that will never… That are marked in our dictionaries, things that will never be expressed. On the other hand, we’ve made deliberate design decisions around things like gender expressions.

0:28:16.4 HM: When you create a character, you create… You choose their pronouns, those pronouns are respected throughout. We make sure that they’re not tied to the avatar they can have, they’re not tied to the roles that they’re given, that’s all done in a neutral way. And that’s a decision we’ve made, because that’s something we’re very sensitive to. And we’ve tried to be consistent in those decisions so that the outputs don’t reflect those kinds of biases. And at least in our system, we’re building some deliberate product choices to make sure that at least those things that we can think about ahead of time are not being magnified through the models.

0:28:58.2 MK: It sounds like you have made some very deliberate choices, and there are, I guess, these stop gaps to help. But do you as a founder… I sometimes feel like getting this right, it just feels like pushing shit uphill. Like you can have the best stop gaps in the world, and something could still go wrong. Is that something that you personally think about, or is it just one of those things that you come to realize? You try and put in the best protections you possibly can to make sure that everybody has a good, positive, inclusive experience. And… I don’t know. Is it something that you wrestle with, I guess, is what I’m trying to understand?

0:29:41.4 HM: Personally, yes. And I also understand that nothing we do technically will perfectly solve these problems. Because even if we could solve them technically, there’s still issues with things like bullying or inappropriate interactions that can happen. And that might be input and output that is perfectly appropriate in one context, but because of the context that’s not even accessible to us within the application, it’s not. And so the way I think about approaching that is that there is always a human layer of moderation and support available to be summoned to interact with the tooling and the power required to be able to handle whatever issues may still arise. And that is a hard-learned lesson from some of my early experiences working on consumer social media.

0:30:37.5 TW: So this is… So there’s the human moderation, and then I think now… I think if I’m understanding… When you said it’ll generate, say, 10,000 templates, and those are reviewed and scored, is that humans that are reviewing and scoring it or…

0:30:52.4 HM: We have a person reading every template, the ranking is automated. But none of the templates go into the system without somebody at least reading it.

0:31:01.9 TW: Okay. And what’s the ranking? What’s driving… Yeah, I’m kinda slow. Yeah.

0:31:07.7 HM: No, no, no. It’s a complex thing, and we’re jumping all over the place. I should have brought an architecture diagram. But…

0:31:14.7 TW: [chuckle] That works great in an audio format. Yeah.

[laughter]

0:31:19.7 HM: Okay. Good point. But if you imagine… Let’s say we’ve just taking a turn in the game, we’ve each decided what our character’s gonna do. The system has mediated it and said, “Okay,” like, “Tim, you succeed. Hilary, you fail. Here is the consequences of your failure, here’s what happens because of your success.” The change in the game state, so the update to the stats behind the thing based on what we’ve chosen, and then it has to tell us in text, it has to write the bit of the story and pull the art together. And so our system is trying to feel like a graphic novel. So it’ll say like, “Okay, here’s Hilary’s avatar falling on her face and Tim’s character’s avatar flying into the air with his power succeeding,” whatever it may be, and there’s a sentence that describes that. And so it takes that structured essentially representation of the plot at that moment and then ranks all of the available text templates that can cover that sentence or two and then makes a selection based on essentially an arbitrary set of heuristics at this point, which is a nice way of saying, “We made it up.” So we made up a way for what seems good at the time and makes sense, and then it does that rendering and passes it through. And that is.

0:32:35.7 HM: As opposed to some of the other ways you might approach this using large language models where you would essentially have the model accept your input as a prompt and generate some text that you would then return to the user, but you have no idea what it says, you have no control over it, it may or may not prompt your suggestions, it may or may not actually do what you want it to do, and it may indeed just say something outrageous. So in our system, what we’re trying to do is take essentially a data structure and render that accurately, we are not relying on the language model at that point to tell us what should happen.

0:33:15.1 TW: So and is GPT-3 a codes language model? Is that…

0:33:18.2 HM: Yes.

0:33:19.3 TW: Okay, just making sure. So the human side of it, what’s the skillset of the, both the humans who are reviewing the templates before they go out, as well as the humans who will be pulled in to do the moderating, how do you, I mean it’s, crazy is not the right word, like it’s fascinating that this thing is being built and yet there’s a heavy human layer, it sounds awesome that the humans are in the front end as opposed to take meta meadow, whereas these poor humans who have to sit and soak through all this horrible flagged content like that sounds miserable. This actually sounds like it might be kind of fun and interesting, but what are their, what’s their background? What’s their skill-set? What’s their role?

0:34:07.6 HM: I mean, I’ll actually go up a level with that question. Tell you a bit about our team. So our team is a bunch of, I hope this would be a compliment to everybody, but delightful weirdos with a bunch of, everyone has sort of a mix of a creative background as well as whatever their particular skill-set is, and we’re about half folks from tech broadly, so software engineers, ML people, and half folks from the games industry, so the game directors very experienced at this stuff, an art director, who’s worked on the artistic systems and games for kids and cartoons and things like that, and a community manager who has experience in inclusive communities of gamers, and so we’re a very small team, so it is still a little bit of everybody who is doing all of the, both the incredibly exciting generation of 10,000 candidate sentences and then the slogging through and reading of them.

0:35:10.1 HM: And so that’s a shared responsibility, and it is, as we look to build out that moderation team, we look to people who have experienced moderating and supporting inclusive community issue of training and how to handle issues like bullying or let’s say, interactions with potentially non-custodial parents or issues that can arise in any sort of social forum. And we’ve also been pretty thoughtful about how people engage in our platform in the sense that we don’t have un-moderated, even audio communications, so you’re playing together through the game interface, which is mediating what your character is able to do and communicate, so trying to be very thoughtful about all of those things.

0:36:02.7 MK: Have you found that skill-set difficult to hire for? ‘Cause I can imagine it’s not, like there’s not people coming out of university with experience in tech and then related to, I guess, equality on a platform. Yeah, I don’t know what the right words are for that.

0:36:21.9 HM: No, it’s something I think about a lot because the answer is no, but we’re very small right now, so we’re ten people, and we’re going into early alpha shortly. And so it’s not hard, and I found this actually, in every start-up I have been involved with, it’s not that hard to hire at this scale because we are looking for the people who don’t fit in a box, who have… Everyone on the team sort of brings a variety of different experiences and ends up doing a whole bunch of stuff beyond what just their core skill-set is, and I very much appreciate that about small teams. I do expect it to be a challenge to hire for when we get to a greater scale.

0:37:04.6 MH: Alright, let’s step aside, and you know what it’s for, it’s for the Conductrics quiz, that’s right. The quizzical query, the conundrum that pits our two co-hosts as they go toe to toe on your behalf on a quiz question conducted by Conductrics. So let’s talk a little bit about Conductrics, ’cause the Analytics Power Hour, and the Conductrics quiz are sponsored by Conductrics, and they build industry leading experimentation software for AB testing, adaptive optimization and predictive targeting. Go to conductrics.com to find out more. Alright, and I’m super excited to announce Hilary Mason has stayed on to do the quiz with us, so Moe and Hilary, you’re a team, would you like to know who you are competing on behalf of?

0:37:50.8 HM: Of course.

0:37:51.3 MK: Definitely.

0:37:51.3 MH: Alright, so Sean McClain, I think you are sir in for maybe a win here because…

[laughter]

0:38:00.4 MH: There might be. This is a power house. This is a dream team, I’m telling you, it’s a dream team. Alright Tim.

0:38:04.3 TW: This is, I will appreciate the expectations are set really well in my favor for this.

0:38:11.3 MH: Tim, you are competing on behalf of listener, Juliana Jackson. Alright, so let’s jump into it. So here is the question. And it’s a long one.

[laughter]

0:38:25.4 MH: You open a door and walk into a room that appears to be a reading study, you notice sitting in a corner, your friend and Power Hour co-host, Michael Helbling, quietly reading an old, frail, small paperback book as you approach, you try to read the book’s title but are unable to, and you say hello, and then ask about the book. “Oh, this,” Michael replies, it’s the original Choose Your Own Adventure book written by Edward Packard. These are very popular with the younger kids back in the ’80s, they’re fun because the reader is given some control over the flow of the story, in certain sections, you’re asked to make a decision that has an associated page number to turn to, this created forks in the story, and it was kind of cool because it made each book a bit like a role-playing game using a simple deterministic decision tree. Today’s podcast guest, Hilary Mason works on AI games and storytelling, so I thought it’d be fun to revisit a super early, an extremely simple version of this type of conditional storytelling, interestingly…

0:39:26.5 TW: I feel like I already turned to page 12 and drowned in the swamp, but…

0:39:28.0 MH: Yeah, that’s…

[laughter]

0:39:30.3 MH: You have died of dysentery. [laughter] No. Interestingly, while this is technically the first book of its kind, it actually was listed as number 62 in the series. You then say, “Oh, then you must be reading… ” And now you get to choose your own adventure here. If you say “Caves of Time”, go to page 10. If you say “Sugar Cane Island”, go to page 20. If you say “The Colour of Magic”, go to page 30. Or if you say, “So Long, and Thanks for All the Fish”, go to page 40. [chuckle]

0:40:04.8 TW: So those are basically our multiple choices?

0:40:08.5 MH: Alright, so I think what we have to do here is determine which page should we be going to, and that would be the correct answer.

0:40:16.2 TW: I’ll start with saying it’s not 40, ’cause that’s the Douglas Adams, “So Long, and Thanks for All the Fish”.

0:40:19.3 HM: Am I allowed to agree with you? ‘Cause I agree with you.

0:40:21.9 TW: Yeah.

0:40:22.3 MH: Yes, I think all of us can agree that that’s not even close. That is the fourth book in the Hitchhiker’s Guide to the Galaxy trilogy. Yes, you heard me right, fourth book in that trilogy. It’s very important that you understand that. So if you chose that one, you would have lost, if you’re following along at home. Okay, so then we have “Caves of Time” at page 10, “Sugar Cane Island” at page 20, or “The Colour of Magic” at page 30. Those are our choices.

0:40:48.9 TW: I have a really strong urge to think that it’s “Sugar Cane Island”. And I definitely read those books as a child, I think that’s actually it.

0:41:00.2 MK: I definitely read them. In fact, when anyone ever talks about what it’s like to work at Canva, I’m like, “It’s the choose your own adventure book. I love those books!”

0:41:06.2 TW: [chuckle] And then they died of dysentery.

0:41:12.1 HM: It’s funny you say that, ’cause I also read them, but 80s, right? And I just… My intuition is it’s either 10 or 30, I don’t know. But I have no reasoning to support it at all.

0:41:25.1 MK: Do you know, I was leaning towards those two as well, but I have no… I literally have no memory of a single name of a single book, so I don’t feel like I’m help.

0:41:36.3 MH: Yeah, I wouldn’t know either.

0:41:38.5 HM: I have no… If I had to pick one, I’d say it’s 10, I think “Caves of Time”. Like that sounds… It sounds epic.

0:41:45.5 MK: What about if each one of us picks one? Is that allowed Helbs?

0:41:50.5 MH: I think it’s totally allowed. We can… You’re choosing your…

0:41:53.7 TW: Oh wow.

0:41:54.6 MH: Own adventure here, so…

0:41:55.0 MK: ‘Cause then I could pick…

0:41:55.5 TW: So you…

0:41:56.4 MK: ‘Cause I think 30, and then you could pick 20 Tim. And then whoever wins, wins. I mean, obviously Sean’s got better odds.

0:42:04.5 MH: Yeah, in that scenario, Sean, you’d only get half a prize, so just FYI I’m just telling you. [laughter] Alright, so we kind of are doing a little bit of like betting odds now, so we’ve got coverage on “Caves of Time” on page 10, and “Colour of Magic” on page 30. And then Tim, are you doubling down and saying “Sugar Cane Island”?

0:42:24.0 TW: I’m gonna lock in. Yeah.

0:42:25.3 MH: Alright. Dun-dun-dun-dun. Okay, so the answer is page 20. Michael looks surprised. You win!

0:42:35.9 MK: Oh.

0:42:36.1 TW: Oh, my goodness.

0:42:36.3 MH: “Sugar Cane Island” was originally written in 1969…

0:42:39.0 TW: It’s a dark horse.

0:42:41.4 MH: And is credited with being the first book of its kind. I can’t believe you got that correct. You’re even more of a nerd, or should I say quintessential analyst, than I even imagined. [chuckle] Anyways, you know, actually though, Hilary, you were super-close because “Caves of Time” was the first in the Choose Your Own Adventure series, but wasn’t actually the first one written.

0:43:00.9 HM: Oh…

0:43:01.0 TW: Wow.

0:43:01.2 MH: So I think… So actually, I feel like you were actually really close. So, that was super good. Anyway, thank you both. Congratulations, Juliana, you’re a winner, ’cause Tim is such a great nerd. Awesome. Thank you to Conductrics for sponsoring the Conductrics Quiz. And let’s get back to the show.

0:43:21.4 MK: Sorry, Hilary, I’m not gonna lie. I have thought… We have all thought about having you on the show forever, and I did not think this… If I had thought about it years ago, this is probably not what I imagined us discussing. [laughter] And I’m actually mind-blown-away because I can’t even fathom how you and your co-founder would sit down and come up with this idea, let alone execute on it. That’s the bit to me… When I think about anything to do with NLP, I’m like, “That sounds really hard. That sounds impossible.” But it seems like you guys ran towards possibly one of the most challenging areas of machine learning. Looking back, I don’t know, what do you wish you knew? What have you learned? What would you do differently? There’s a thousand questions I’ve got on this.

0:44:16.4 TW: Well, and it does seem like… It’s like you… You kinda like, dis… Like you went in stealth mode, it just was like, “Oh, Hilary’s working on this new thing.” And then it was like… It was a while. [chuckle] I was like, “What’s she doing? I put my name on the homepage… ” But, yeah, so I’ll add on maybe, has it taken longer to get where you’re expecting… I mean, I assume it always takes longer than you’d expected or hoped. But how… You’ve gone through this experience before. This seems, boy, really, really different. So, has the path gone as you expected? How has it, or how has it not?

0:44:58.6 HM: First up, it is very exciting to be here. And yes, if we were having this conversation five years ago, it would have been very different. And has it taken longer to get… Yeah, it certainly has. And part of it, if I’m allowed to be a human being for a minute…

0:45:16.7 TW: [chuckle] But you are.

0:45:18.0 HM: Part of the path has been just like, there’s a pandemic. And a lot of the things that at least I was accustomed to doing professionally just sort of dampened down or changed dramatically. And we have built… We spent about a year just building prototypes and testing them, and trying to understand both at a technical level, is the thing we want achievable, but then at an experience level, is the thing we want… We think, we want to build something people actually enjoy. Is it fun? And that’s really different than many of the products I worked on before. So…

0:45:58.1 HM: Back in 2014, I founded a company called Fast Forward Labs, and it was an independent applied machine learning research and product development group, and we did our own research, which was really cool, and that’s really where my interest in modern NLP got started in that… We actually did our first report and prototype on natural language generation back in 2014, we did a whole bunch of work on summarization, which remains probably the most fun project I think that I worked on doing that company, and then we worked with a whole bunch of clients and partners to build out their machine learning products, and that was everything from like, Can we do keyway on videos, a pizza coming out of a quick service pizza oven, like robotics surgery, video analysis, to customer support via NLP, again, same technology, very different application and where some of the liabilities in that context are actually advantages in writing fiction, for example.

0:47:00.3 HM: And I mentioned not only because there is a thread through all of it. That makes sense. And then again, being a human being with a choice to make, when I finally moved on from Cloudera and I knew I wanted to build another product, I knew I wanted it to be… Again, something with people I wanted to spend time with, something I was personally really passionate and excited about, and also one that has… That’s really hard, where we are able to try to create a new experience around an emerging technical capability, that’s just really fun, and it’s something that is… I feel incredibly fortunate to even get the chance to do this and to do it with a team that is so interesting and fun and quirky to is just really… Yes, that’s why…

0:47:55.3 TW: Do you feel like the traditional gaming companies, the Hasbro bought D&D Beyond, are they thinking along these lines too, about how to be… Not just take a… What was traditionally an offline experience and making it like porting it from, let’s make that experience online, Words with Friends is taking Scrabble and putting it online. Are there the established players or they also, maybe in different ways, but trying to say how can we actually make this unique and more engaging or something, is there a lot happening in that space?

0:48:41.5 HM: There’s not as much as I would have thought, and I have a theory about why. So one of the things I’ve been trying to do is to collect all the cool people thinking specifically about NLP games, and so if you happen to be one, I would love to know you and you should send me note on Twitter or something, and probably we will have a party or something, or a conference at some point, and there are folks, there’s in academia, there’s actually incredibly lively community, and then at the big gaming studios, there is a very active research community. But they seem to be more focused on using machine learning to make the content they’re already paying people to create cheaper, which is a… If I can be slightly critical, is something I saw a lot in my work with the Fortune 500 of the world when I was the GM at Cloudera for the machine learning business, is that it’s very easy to get a clear ROI on something when you add a layer of automation and now it used to cost you so much, and now it costs you less than so much, and you’re like, Cool, I say this much, probably not as much as you’re paying your machine learning engineers, but that’s another question.

0:49:48.2 HM: But there’s not a lot of appetite to try to create the entirely new thing that’s not possible because of the automation, if that makes sense. And I’ve been surprised that I haven’t seen too many folks really trying to create new experiences, so a lot of people making better tools for playing traditional D&D online, and that’s awesome, and there’s a huge hunger for that, but I haven’t seen all that much trying to evolve what’s possible because of machine learning or AI broadly.

0:50:20.1 TW: So there’s automating the DM and doing the… Being the DM at scale versus saying, No, I’m gonna give you a DM who actually has a much richer world that they can facilitate you creating, and you can pick what that… You can pick and GUI what that world is. And the DM is able to pull from a vast trove of… Of tropes, I guess, to pull from maybe… Okay, it’s gonna give me hooked on gaming.

0:50:51.3 MH: It’s about time, Tim. Alright, this pains me because we do have to start to wrap up, but this is so fascinating and I’ve sort of been quiet for most episode because I feel like I’ll just fully Chris Farley on you and be like, Do you remember Settlers of Catan, it’s so cool, it’s so cool. It’s just so amazing. Thank you so much, Hilary, is so cool to have you on.

0:51:18.5 HM: Well, thank you all. This is really fun.

0:51:20.4 MH: So one thing we do on the show is we go around the horn and share a last call. Something that might be of interest to our audience. So Hilary, you’re our guest, do you have a last call? You’d like to share.

0:51:30.6 HM: So I think what I’ll share then is a book that you will all lose yourselves in for your summer beach reading that sort of goes with the theme of getting lost in a fictional world and everyone’s treated, This is How You Lose the Time War by Amal El-Mohtar and Max Gladstone, and it is the kind of book that has both incredible world-building, incredible intense character development, some romance, surprising plots, go pick it up on your summer vacation, you will have zero regrets.

0:52:04.3 MK: I think I’m gonna try it. I’m saying, that’s surprisingly good [laughter] ’cause I don’t actually read a lot of sci-fi but it sounds really good. So… But I’m willing to try like any genre so.

0:52:16.0 MH: Oh, well we’re gonna get you hooked.

0:52:18.6 MK: I know you’re she’s just nudging me in there, like one little recommendation at a time.

0:52:22.0 MH: Sci-fi and fantasy is like the best fiction.

0:52:25.2 TW: I was, as a child, I was into sci-fi. It is Michael who turned me on to John Scalzi, which is just super quick reads and I just devoured his stuff and then it’s like a friend of his who turned me on to Brandon Sanderson. So my Sci-fi and fantasy consumption has gone through the roof in the last six years. So…

0:52:50.0 MH: Well, they’re bringing out obviously the… Of course, I can’t think of the name of it, the dragons one, the HBO show.

0:52:58.7 HM: Game of Thrones?

0:53:00.5 MH: Game of Thrones. [laughter] Thank you. It’s like I’m pulling a complete blank and also Wheel of Time now, they’re doing TV for that too. Okay, Tim, what about you? What’s your last call?

0:53:10.2 TW: So I’m gonna do a quick twofer, and I’m gonna… They’re actually both loosely related to Hilary in different ways. But one, I was talking to Matt Policastro, a past guest on the show, and he brought up datakind.org, which I realised from years ago I was vaguely familiar with, but it’s kind of like for those of us who are coming from the digital analytics world the, it’s like Analysis Exchange was but times like a 1,000. So opportunities to volunteer, if you’re feeling like you’re dispirited and disheartened and burned out at your, whatever your analytics work and in your day job, you’re getting paid for. DataKind have been mulling over what somebody who’s a bit of a Hackworth with limited skills might be able to contribute to humanity. But I was just looking at their website before when I was coming off on our last call and Hilary, you’re technically on their site as an advisor, so I’m like, “Of course, she’s everywhere.” So that’s one. The other is the Everything is Everything 202 Bagel Shops in New York City, which I came across through Philip Bumps, How To Read This Chart Newsletter, but it’s kind of also in the spirit of kind of retro things that these two guys walked the equivalent of five marathons every week in New York City.

0:54:33.6 TW: And one of them would regularly order an everything bagel with scallion cream cheese on it, and then he would rate it. So it’s about as like low-tech human, totally subjective, and then they built a site where you can kind of explore your area, and I should look up what the top one… He has the rock star in each one, I’m not sure what Brooklyn, the best in the borough was, and if I’d really been on my game, I wanted to ask my brother-in-law.

0:55:02.8 HM: So the that, as a native New Yorker, the thing about bagels in New York is that they are all good. So it doesn’t matter, they’re all good.

0:55:11.6 TW: As somebody who grew up in Texas and lives in Columbus, I am like, “Give me a good bagel,” and I don’t have a super refined palette, I’m just like, “Oh my god, a good bagel, I’m a sucker for a good bagel,” which was why I got excited about the project. But the Old Brooklyn Bagel Shop, are you familiar with that? Ever heard of that? They’re the… They were rated as the best in the borough for Brooklyn by this fella, fellow.

0:55:39.6 HM: Alright, well, I’ll have to check it out.

0:55:41.4 TW: I don’t know. But a good bagel… It’s kind of a fun little website ’cause it’s like data viz, overlaid on maps with kind of a retro pixelated like Atari sort of look to it.

0:55:54.2 MH: Nice. Alright. Moe, what about you? What’s your last call?

0:56:01.5 MK: I don’t really commute, so I don’t listen to a lot of podcasts, which people find particularly strange since I help co-host a podcast. But I had the enjoyable trip on a train this morning and actually decided to listen to a podcast, which I can’t believe I didn’t listen to Tim earlier, it’s really annoying that he is right, and he will have mentioned this as a last call before, but I’m gonna repeat it anyway because Tim recommends a lot of shit and you would not be able to do your job if you listened and read everything he suggested. So this is extra reinforcement. Oh my God, Michael Lewis, I just love the man. Why didn’t… I love all of his books. I love them. Why wouldn’t I have listened to his podcast? Anyway, I’m just completely obsessed with Against the Rules now, and particularly the latest season because it is about experts and I really love it. But the one like epiphany moment I had, was in that one particular episode they’re talking about a specific metric, and people try and boil things down to a specific metric, in the finance world, its VaRs, which is value at risk or whatever, and it’s intended to be a specific metric to help you understand the risk of an investment like in a summary.

0:57:14.5 MK: But what often happens when you try and boil everything down to this just like one key metric, is that it actually takes thought away. And that the point of a key metric like that, is that it’s actually meant to be the start of another question, not the answer to the question, and I just… I hadn’t even had coffee yet, and I was just sitting there being like, oh my God, this is… Like we do this, we do this in data all the time where we try and come up with the perfect metric that we can say to the business like, “This is the metric.” but the truth of the matter is, that should be the start of the conversation. It shouldn’t be, “This is how we’re performing against that metric,” end of conversation, and that’s what we so often do. But anyway, I love Michael Lewis. I could listen to him talk about like toilet paper, I swear. Anyway, Helbs, that was a very long-winded last call, I will hand over to you.

0:58:06.5 MH: It’s fine. No problem at all. I love your passion.

0:58:08.6 TW: What’s your last call, Michael?

0:58:10.6 MH: Well, since I mentioned it earlier, please go check into your local library, and if you haven’t yet read any Brandon Sanderson books. If you wanna read that series, the book you’re looking for is called The Final Empire. So that’s book one of a series. So that’s the book we were talking about earlier, but. If you need to… This is another fun one. If you need to sort of get your YouTube algorithm shook up and changing a little bit for you, I recently came across a YouTube channel and I’ve been loving all of their videos, which is sort of weird intersection of British people and Korean food, a YouTube channel called LOLLY. And it’s two British guys, but I think they spent time in Korea.

0:58:51.0 MH: So I’m not sure exactly what their whole thing is, but they’d end up eating a lot of like Korean food and other kinds of food, introducing people to different kinds of foods. And for whatever reason, I just love that channel. So whenever you needed me to get something done and I’m not doing it, I’m probably watching their old videos. Anyways, that’s the YouTube channel that I’m really kind of deep diving on. So if you need your YouTube algo to get up-settled or whatever, check out LOLLY, which is their YouTube channel. Okay. That’s it. We would love to hear from you and you don’t even have to be a gamer. You could just be someone who’s interested in these kinds of topics and how they work. And the best way to do that is through the measure Slack or also on Twitter or on our LinkedIn. Hilary also is on Twitter. You’re actually a pretty big deal on Twitter. [laughter]

0:59:46.1 HM: I’ve just been there a very long time. [laughter]

0:59:48.6 MH: Yeah. And your Twitter handle is @hmason. So that’s an easy one to find and follow, so then I would encourage everyone to do that as well. And right now I think Hidden Door is still in early beta or alpha, but there is a sign up. So everybody go sign up and see if you’re interesting enough to make the cut to be maybe like one of the early people in there or not… I’m in the midst of the sign up process myself, so we’ll see, and I’m now questioning my choices. And I’m like, maybe that’s the wrong games, I shouldn’t say that. Anyways, it’s okay. [chuckle] We’ll get there. Also, no show would be complete without a big shout-out to our producer, Josh Crowhurst. Thank you, Josh, for everything you do to help us get the show out the door. Hilary, thank you so much for taking the time to come on the podcast. It’s so fascinating to hear what you’re building in… And it’s so unique and actually so important, I think. And I think… You’re building it for the world of games, but I clearly see a future where this kind of idea that you’re creating will actually apply in so many different areas, so… You said, “Who’s creating the next level of innovation or the next things… ” You’re creating it, I think. So I’m pretty stoked that we get to talk to you. Thank you. Sorry…

1:01:05.9 HM: Well, thank you. This has been really, really fun. [laughter]

1:01:08.2 TW: Heading towards that Chris Farley. [laughter]

1:01:12.2 MH: Stop it. Just stop. Just stop. [chuckle] Okay. So with that, we’ll wrap up the show. It’s always a pleasure. And remember, whether you are using natural language processing or just Excel, one thing you can make sure to do, and I know my two co-hosts, Moe and Tim agree, keep analyzing.

1:01:32.4 Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at @AnalyticsHour, on the web at analyticshour.io, our LinkedIn group, and the #measure chat Slack group. Music for the podcast by Josh Crowhurst.

1:01:50.4 Charles Barkley: So, smart guys want to fit in. So they made up a term called “analytics”. Analytics don’t work.

1:01:57.0 Tom Hammerschmidt: “Analytics.” Oh my God. What the fuck does that even mean?

1:02:05.8 MH: And there’s no rules, really. You know, so… You know, just do whatever you want.

1:02:10.6 MK: Other than cheating.

1:02:12.3 MH: Well, yeah, but who’d know?

1:02:14.5 MK: I’d know I just would be specific?

1:02:17.7 MH: I don’t think that’s what Hilary would do. [laughter]

1:02:21.9 MK: I don’t feel if any guest is gonna cheat, it would ever be Hilary Mason, to be clear. [laughter]

1:02:27.8 HM: I’m still not sure what game we’re playing, so… [laughter]

1:02:34.3 TW: Yeah, that was total trivia.

1:02:36.8 MH: Yeah.

1:02:36.8 MK: Okay, this is the first time we’ve not had an actual math question.

1:02:40.1 MH: I know, like…

1:02:41.4 TW: I’m now remembering, with Hilary Parker, it was like a variance calculation.

1:02:46.6 MK: It was a calculation that she had memorized at university. And I was like, “Seriously? I didn’t do that at the university.”

1:02:52.5 HM: You know, the law of Hilary is like, “All Hilarys in data science are like super awesome.”

1:02:58.0 MH: They are.

1:02:58.6 HM: Hilary Parker is amazing.

1:03:00.8 MH: Yeah. We’ve never gone wrong.

1:03:04.2 TW: We’ve never gone wrong with a Hilary.

1:03:05.5 MH: Yeah. [chuckle] That’s right.

1:03:09.5 TW: Rock, flag and tropes by NLP. Not my best one, I was struggling…

1:03:16.1 MK: That was a weird one.

1:03:17.4 TW: That’ll be way in the outtakes…

[music]

Podcast: Download | Embed

Subscribe: RSS