00:00:00 - 00:00:45

How does an MM actually work? >> This is what [ __ ] fires me up, dude. Is like this is a game about math and now people are starting to pay attention that the math matters. Gartner says 60% of Agentic AI projects are going to fail by the end of 2027. >> Yeah, it becomes a cluster [ __ ] for a marketer. >> We see a ton of brands that are spending at the wrong time in advance of these seasonal moments. It's like driving down the highway, opening up the sunroof, and just letting the money fly off the top

00:00:22 - 00:01:27

of it. Okay. So, Mike is the co-founder and CEO of Preient, which is a media mix modeling platform. You have a pretty storied career in uh AI sales working for IBM and then I can't remember who you where were you. >> Oracle, IBM, App Annie, and some AI consulting firms. >> And you actually have a pretty interesting story about how you fell into preient, but I don't think we go there today. No. Let's let's talk more about like what is media mix modeling? >> Let's talk about media mix modeling of

00:00:59 - 00:01:55

why it started, what did it do and then how people viewed it in the modern times and then where is it going in the future? Um 1960s you had people that sold a water bottle in a retail store and how did you advertise for it? On a billboard, TV, catalog, radios, newspapers. you're not clicking any of those channels, but you still spent money on them and people walked in the store and they purchased the water bottle. So, what they would do is they would try to gather as much information as they could about how much

00:01:27 - 00:02:15

you spent on each channel and then, you know, how many households did the ad show up on TV? How many households was this, you know, coupon sent to? Long story short, they got a bunch of people together and they try to put as much, you know, marketing data and sales data together in a statistical model to try to figure out attribution and then do some sort of media planning and budget planning. and say, "Okay, for this upcoming year, if we want to sell the most amount of water bottles in this

00:01:51 - 00:02:42

location, you know, let's increase our investment on the billboards and let's increase our investments on the radio based off of like what the measurement is saying, right?" You can't click anything back in the day. So, you're just looking at spend to revenue and what other variables you could layer in there. >> So, a media mix model, MM helps you better understand where to invest your dollars to maximize an output or an outcome. for most brands that's like revenue

00:02:16 - 00:03:07

>> or new c the traditional M&Ms are just like a revenue metric as they started to get more advanced um you can evolve what that output that what that outcome is and that's something you know we we obviously do really well so you people have used M&M's before the internet and then the internet started and you could track everybody around the internet kind of like took a backseat in a little bit maybe they'd run something once a year twice a year or once every other year

00:02:42 - 00:03:44

try to figure out some budgets the internet comes, you can track everybody everywhere and so you have more of a deterministic customer journey. Attribution theoretically was a little bit easier, right? Because you can actually track uh where people are going across device, mobile, web, all that sort of jazz. Actually, this is before mobile and then iOS 14.5 hits. Well, the internet hits and now you can start spending on, you know, Meta, Instagram, obviously Google, then YouTube comes out. But predominantly most of the

00:03:13 - 00:04:12

brands were spending on as you know >> Meta >> and Google, right? Yeah. One ch two channels driving to their um driving to their Shopify store as you well know with your agency. Where do people go? Well, I'm going to go into Amazon, right? I'm going to go into wholesale or maybe I'm going to stand up brickandmortar stores and hey there's these tools like Keen Digital and and Neon Pixel and Tatari and you know all these different CTV providers vibe that now you can go and advertise on

00:03:42 - 00:04:37

connected TV and a pretty like low barrier to entry right just from a technology perspective now an MTA is not designed right multi-touch attribution is not designed to measure those channels because multi-touch attribution is a deterministic model. There's probabilistic and there's deterministic. Deterministic model is going to say Lucas, right, clicked this ad, got an IP address, then might have clicked another ad, got that IP address, then checked out at another time. And now you can

00:04:10 - 00:05:07

match kind of stitch up those IP addresses, say this is deterministically the path that Lucas took to purchase. >> We can 100% give credit to >> 100%. So if you have a 100 conversions and deterministically you can explain um on met and Google maybe because there's more clickbased channels less channels you can explain 80 90% of your conversions just through those clicks >> and for so is is in platform attribution it's deterministic >> it's deterministic in the sense that

00:04:37 - 00:05:37

it's um within the walled garden of the ad platforms right in their within their own ecosystem >> but that's where the problem comes to is they're all taking credit for the same conversion Right. >> Yeah. So if you put Claio, Meta, Google, and you just add up conversions, you have an inflated revenue figure >> over 100%. Right. >> Yeah. So then you have multi-touch attribution, which is deterministic aggregation of all of those channels. >> Well said. Perfectly. It tells you a lot

00:05:08 - 00:06:19

about a little, right? Tells you a lot about those two channels. Um what happens when you go into TV or podcasts or YouTube radio linear out of home? Um >> yeah, it becomes a cluster [ __ ] for a marketer >> because you all you have is a spend signal. Maybe you get some impressions. Maybe you get some reported revenue from whatever pixel the CTV provider has. It >> and this is what I love because I've been in the trenches with marketers. It's like you actually see how those budgets get

00:05:43 - 00:06:35

uh put together and it's very much just you know by the the proclivity of the marketer on that day like this kind of feels right. >> We'll invest here. >> Listen like you >> and I mean that's that's what good marketers make a lot of money. They have like experience and feeling and so that's part of their calculus and deploying >> but it's not math based. >> It's not math based. And listen, I'm not going to derail this too much, but like

00:06:08 - 00:06:58

I fundamentally disagreed with the message that measurement providers were talking to these brands about. And I would come from the space. I'm meeting people like, well, this is my source of truth. I'm like, why are these people telling these marketers that that they're the source of truth is the marketer point blank period stop, right? There's different forms of measurement, right? You have your MTA, that's clicked base. You have incrementality that is this incremental at a point in time. You

00:06:35 - 00:07:20

have an MM. This is probabilistic, right? Statistical AIdriven models. You have a post-purchase survey that could have, you know, forgetful mindset on that as well. >> And the marketers just come in and they just want to feel confident in the bet that they're going to make. Yeah. >> They're going to take their own calculated risk and they use their own intu human intuition. They grew these brands to what they are today. And maybe it was on just a couple of channels and it gets more complex when you start

00:06:58 - 00:07:50

layering on additional channels. >> That's why you have different skill sets. I think different places where marketers really work and entrepreneurs too like you have the 0ero to1 people that are comfortable operating with zero past data and they're making calculated bets like MM is not a good fit for a startup business that is testing the waters for the first time as an and is operating on a few channels like >> right but then you have marketers who you know they're CMOs and they're coming

00:07:24 - 00:08:17

into established businesses that are running millions in in advertising and they have multiple sales channels And that's when you want to use a tool like an MM. >> I mean, you've been in the game for for long enough where it's 2020, 2020, 2021, even 2022. I'm sure there wasn't a ton of buzz around people talking about an MM. It's a lot of buzz around >> post post iOS 14. Like you had a couple you I remember our team, we started digging up. >> You guys actually started building you

00:07:51 - 00:08:47

guys are actually forward thinking on that. >> We we built an MM. We were using lightweight. Lightweight. Yeah. Yeah. Yeah. And we would dig up that 1962 article. There was one I think from the '8s and this was like the old these statistical models for MM and >> forever. >> Yeah. But there's some I remember there, you know, you could build up a model but they were very unwieldy. Like you'd refresh them once a quarter. >> Yep. >> Maybe once a month. They they wouldn't

00:08:18 - 00:09:12

they're not the right fit for every type of brand. If you're only spending on a couple channels, it's like why use an MM? Like you don't really need to do that. Um and then also the the major thing though is education though because you had the all of these marketers who are very used to deterministic attribution or MTA and they like want to know and educating them on MM and incrementality. It's like you have to it's different sides of the brain. It's been a very interesting journey to

00:08:45 - 00:09:44

see people who said, "I've built my own MM before all the way to," hey, my boss heard about you guys on a podcast and I just watched a YouTube video and I built my business on Triple Whale using a a pixel, right? Figuring out a way to simplify that down was something we've getting better and better at. But for sure, dude, it's a hurdle. Like, >> it's it's the tech and it's user experience. We looked at a ton of MM platforms like after we were like [ __ ] like we're not going to be the guys to

00:09:15 - 00:10:02

build this internally and uh I was very impressed with what you've built out of all every single provider because of the user experience and because of how >> unique like you were like you're an Amazon business and a Shopify business like >> we're going to we're going to crush it for you. >> I was just chatting about this today. I mean, obviously up here in New York and, you know, doing the rounds a little bit, but we were talking about the just this MM are in a very teenage sort of phase

00:09:39 - 00:10:29

right now because if you think about what the MM was designed to do, it was designed to do a long-term budget plan and this is how they were designed to do. They were always thought of like a bunch of humans coming together and think of a Neielson. It's a million dollars. It's a PDF report. It's a bunch of macro data, micro data, third party, first party data. And they're trying to figure out what is that upcoming mix for the year. And then you had folks like us start to come in and say, "Well, you

00:10:04 - 00:10:59

could use a still the same probabilistic models, similar inputs, but why can't the model run every day? Why do you have to run this once a year?" >> Yeah. >> Right. Why can't you do it at a campaign level? And so then the ecosystem started to take the existing research paper and what you see from a lot of the newer MMs that are in the space, it'll run weekly or monthly um you know, at a tactic or channel level. But the the thing that I think a lot of folks need to know is the

00:10:31 - 00:11:20

MM can produce the results, but the MM has to recalibrate. It has to like essentially start fresh. It needs to get a fresh start. And a lot of these model models are refreshing every 3 months. And so we just had a brand come on and they were using an MM. And the guy was like, if the model's refreshing 3 months ago, the model's going to think it's like tariff times. There's too much time that has had been happened into it between the between when the model refreshes. So, this is what people are

00:10:56 - 00:11:45

starting to wake up to now. It's like, okay, well, we use it for this long-term budget plan, and we didn't need to refresh it for a full year. And then we come out, we're like, we can refresh this thing every single day at a campaign level, very granular. And then the traditional research that's in the space now is is it okay? And is it safe to be using an MM that in a dynamic way that only updates every 3 months, right? And that's where you see a lot of the problem with the open sources like with

00:11:20 - 00:12:09

the Robins and with the Meridians, right? These things are expensive to run. and they're clunky. >> They're also not really optimized for e-commerce like these like questionable like seasonality components, >> brand drives. >> Yeah, that's those are some of the things I was learning about and we're like, okay, someone's going to build a solution for this. So, so talk to us about the math like how does how does an MM actually work? >> So, the models are going to look at all

00:11:45 - 00:12:32

of your historical brought up a good price seasonality. It's going to ingest all of your historical data. So, it's going to look at your revenue by day, your new order orders by day and new customers by day. Through Shopify, through Amazon, and through retail. It's going to pull your GA data, right? And so, for us specifically, we look at conversions that happen through GA. We build subm models that have paid, organic, and direct, right? And then you have uh all your attribution data from

00:12:08 - 00:12:56

your app channels. So, you have your spend, impressions, clicks, sessions, reported revenue from meta, Google, whatever those channels are. So, once you ingest that data, what are you trying to figure out? Well, you're trying to figure out one thing is the seasonality of the business, right? You look at seasonality from an annual perspective, you know, a quarterly perspective, a monthly perspective, a weekly perspective, and you're trying to quantify like a baseline what is the impact of just the worth of the the

00:12:32 - 00:13:29

value of the brand in these sort of seasonal moments. The most important part or very important part about an MM is you have to try to control for everything. everything meaning of all the things that could drive somebody to the website to convert. And what is that really? It's a viral moment. It's word of mouth. It's brand equity. It's seasonality. It's just randomness of some somebody being thirsty and walking into a store, right? Once you quantify that and you get kind of comfortable

00:13:00 - 00:13:59

what that baseline is over time, then you can start layering on what you can measure. What's that? The spent across your paid channels. Well, how do you get confident in an MM? in an MM is you back test against the actuals. So historically, hey, I am going to see if I could predict last December of your revenue and I'm going to train on everything up to that and then I'm going to do kind of blind. It's like all right, I know how much they're spending. I know their seasonality. Could I

00:13:29 - 00:14:20

predict how much revenue do they did in December? And that's one of our stronger spots is, you know, that kind of out of sample back testing. So basically like you potentially you have like infinite numbers of scenarios >> coariants is another thing like these coariants could be we're working with the retailer now it's like there was a bunch of the fires in California what was the impact of stores being burned down to their whole business what's the impact of the number of new stores

00:13:55 - 00:14:51

opening what's the impact of foot traffic or what's the impact of being Guitar Center and having a rock concert uh a half a mile from your store in Detroit that's going to drive right and so these models they want to try to figure out and explain as as many signals as they can. >> Interesting. How do you how would you feed it data like that? >> You give we have a very simplized data schema where you can kind of enter in um events if you will or same thing with an influencer. You pay an influencer

00:14:23 - 00:15:09

upfront a big chunk of change. You know, MM like to see what we call time series data. >> Time series data. >> Yeah. So, how do you normalize? We have some great resources internally that work with influencer data, podcast data, radio data. And we have a really good ways of kind of normalizing that. >> Do you look like at like impression like impressions from a source like over time if it's an influencer? >> Impressions are a wonderful input to have. Obviously, we love to have the

00:14:46 - 00:15:29

spend. We kind of we kind of estimate out what like a spend decay would look like. Um >> yeah, like if it was a one-time payment from an influencer. Yeah. >> How does that look like? Let's say you have like a 100k payment to an influencer. >> A lot of it comes down to like the intuition of what the marketer believes. You always want to meet the market or what their intuition is, but it's like all right well over time they're going to see X many many impressions over time

00:15:08 - 00:16:03

like how does that correlate to the spend and then you know hopefully the models are back testing with good high levels of accuracy with those assumptions and you can start to just get more and more confident in it and how those are set up. >> Interesting. So basically you have flexibility to add different types of data sets. >> Absolutely. weather data, gas price data. Um, >> you know, we have some doomsday prepper companies that like >> I love that [ __ ] >> You know, like shit's popping off in the

00:15:35 - 00:16:37

world. Like their sessions are going up and you can measure those things pretty easily um if you can control for the right things. >> So basically like so MM's been around for a long time. Um it became more popular post iOS 14 because people started to realize deterministic attribution was kind of a sham. >> Yeah. went up top up up a funnel. People were like, "Sh," that's why you saw a lot of the MTA tools bolt on an MM recently because the their clients are like, "Well, I'm spending on YouTube and

00:16:06 - 00:16:56

I got a million impressions and I got 4,000 clicks." Probabilistically, you know, the delta between the impressions, the clicks, some of those people went to Amazon. >> Yeah. No, it has an effect. >> Search them. Who takes the credit for that? Last click takes the credit. Especially if you have a high AOV brand. was looking at brand $800 AOV and I was like let just >> Yeah, you can't use last click. >> Dude, it was a they had four search campaigns. It was a 400 rorowaz like a

00:16:31 - 00:17:27

289 and they're spending on nine channels. >> Yeah, >> it's crippling if they don't have an MM is crippling to try to go scale those channels, right? If you don't have that >> bas basically like would you say this is accurate? like MM is a much better way to categorize the impact or quantify the impact of your brand awareness advertising. >> People want to feel confident that they're when they show up to work and they're making a bet on the top of

00:16:59 - 00:17:35

funnel in a very difficult channel to measure. They didn't waste that money. It's like or they're continuing to waste the money because they don't have a right way to measure it, right? >> Yeah. >> So, they won't feel confident. So, Eminem's top of the funnel. And the other thing is that, you know, I think we're particularly well known for is is MM tells you what you should go do next and then predicts what's going to happen. >> How does that work?

00:17:17 - 00:18:04

>> Well, we're looking at things like um for us specifically and, you know, give away some of our secret sauce because it's, you know, public pretty public now, but like we've been able to come up with some research so we understand the the ratio between top of funnel and bottom of funnel. Meaning like, hey, you have these seasonal moments coming up and we know these are sort of tentpole moments for you. We see a ton of brands, a ton of brands that are spending at the wrong time in advance of these seasonal

00:17:41 - 00:18:30

moments. It's like driving down the highway, opening up the sunroof, and just letting the money fly out the top of it. So, some cool things our model will do is we'll say, "Hey, we know when you need to start spending on your top of funnel at which campaigns on which campaigns based off of the predicted impact of just seasonality doing its business, right, for your brand, right? You you have to have spend on your top of funnel to fill your bottom of funnel. you have to spend on the bottom of the

00:18:05 - 00:19:02

funnel, right? To show up when these people are going to be coming there. So, that's one way to do it. The next way to do it is this those saturation plots I was telling you about. Now, the saturation plots are essentially, you know, you have your spend and whatever metric like CAC, new customers, revenue, rorowaz, how much should you spend on each campaign? And so when you think of Neielson that would do that once a year in a model with our platform, you can go select whatever campaigns you want,

00:18:34 - 00:19:18

enter in a budget, click a button, and 20 seconds later, it's going to tell you to increase your spend on these campaigns and decrease your spend on these campaigns. It's going to predict out what that CAC or new customer will be at an individual campaign. Since our models run every single day, we'll actually tell them after a few days if we think we're on track, if they think they're on track to hit the forecast or not. Right? George box, famous statistician, all models are wrong, but

00:18:56 - 00:19:39

some are useful. How do you make the most useful model? Well, you tell them ahead of time when you think you're going to be wrong so they can make changes to it. >> How long does it take? Like, do you quantify model accuracy? >> Oh, for sure. Yeah. >> How do you how do you think about that? >> We we predict out what we think is going to happen and then we look to see how well we did the prediction or not for an individual campaign. >> There's actually a lot of risk on that

00:19:17 - 00:19:57

because people are making money moves basically. >> And we can see if they make changes though. We'll track to tell them exactly how much is spent and we'll tell them if they're on track to take our recommend. We keep a very close eye on the models will keep an eye on that. The other question I think you're asking was is like, well, how long could does it take for you to start getting a kick signal if they're going to be right or wrong? >> Yeah. >> If you run a 7-day op a weekly

00:19:37 - 00:20:28

optimization model and you have a price point of $2,000, right? You're probably not going to be able to pick up a whole bunch of signals as quick. If you have a CPG brand, you have a $40 AOV, you have more conversions. That's when we can start too. Um >> I'm curious in like like anecdotally like when brands onboard with Prussian how drastically are they changing their immediate mix. >> 5 to 15% sometimes up to 20 depending on the risk tolerance of the brand. We'll tell them not to onboard to the platform

00:20:02 - 00:20:56

if they're not ready to do that. >> You have to >> you have to make changes. >> What are you going to do? Minemm hate uh cake static consistency. Um, I'll sometimes look at some clients and I'm like they're spending at, you know, $300 a day for like three weeks and they go up to $500 a day for like three weeks and the model wants you to see you spend and kind of build these scatter plots and that's how these saturation they want to see scattered spend and that's

00:20:29 - 00:21:42

how these saturation plots start to form. >> And so in Q4 like you have big spend drive seasons for e-commerce like >> Oh, for sure. >> Do you give people scenario planning? That's exactly what the model will do is we run these scenario plans. Um we're a few weeks out from it, but um these these scenario plans will largely be powered by um brand specific agents that are just you'll be able to go ask questions. Hey, we have a target that we need to hit over. How can we,

00:21:06 - 00:21:47

you know, decrease our CAC by X percent over the next quarter? How much money do we need to spend to hit this revenue target? And these agents are going to go out and come back with very specific information and a lot of very come back with very specific recommendations. >> You need a great model to do that. >> Well, this is the thing and I'm going to go off topic for a little bit. This we've been chatting about all day and I'm here in New York talking about all week is compound AI. This is going to be

00:21:26 - 00:22:24

the future. Guy, the founder of Data Bricks has been writing about this compound AI. And you think of Historically in AI, you'd have this like one monolithic model, right? You had a fraud model, you have a forecasting model, you have an optimization model, and they all kind of stayed separate from each other. The Gartner says 60% of aentic AI projects are going to fail by the end of 2027. Why are they going to fail? Right? compared to the competition or compared to folks like us that are building these compound AI models.

00:21:54 - 00:22:48

Essentially, compound AI is a whole suite of machine learning models that talk to each other and then the agents talk to each other as well based off of the specific tasks and functions that they could perform this the specific tasks that they perform. So you could have an audience agent, you could have a creative agent, you could have a saturation agent, you could have a forecasting agent, and imagine putting a bunch of PhDs into a one room to cover one specific general topic and their expertise in each one of them. And then

00:22:22 - 00:23:29

every single day they get new data and they get smarter, right? And so, um, I think that's going to be a big part of where M&M's evolve into as an as a we look at measurement as like a measurement intelligence, but we're moving towards um, you know, a universal intelligence, a universal automation. >> What data sources are very valuable to you that you don't currently have access to? >> Um, it depends on the I say it depends on the category of the brand. Um, you know, we

00:22:56 - 00:23:49

just started we just signed like our first airline betting sites financial services company which is much different inputs or assumptions than a consumer brand or retailer but even within like um you know within the e-commerce space >> like do you have broad consumer spending like data >> you get more a lot of these like more I'd say enterprise different ver these they they spend a ton of money on just third party data >> yeah like Neielson >> yeah Neielson's a great example of that

00:23:22 - 00:24:21

Neielson was a data provider and then turned into an MMA like a lot of Gartner has their own MM like all these data providers that have access to this data they try to create the value of including that into an MM um for the brands though it's pretty straightforward like straight up it's like can you figure out what is the seasonality of the business can you figure out the buying cycle of the product is a huge one they call these ad stock functions and why does that matter right because in MM Lucas spends today

00:23:50 - 00:24:54

on you know his uh his his bracelet but somebody could purchase two weeks later. A group of people that came from the awareness of the day you spent at that time. So MM needs to figure out like, hey, we believe that conversions within 21 days after a day of spend could actually be tied back to that spend. Typically with brands have a higher AOV, that consideration cycle's longer. That's a huge part of an MM to try to figure out like at what's the amount of appropriate time that you can go back and redistribute

00:24:23 - 00:25:36

that credit to for those conversions. So on a more tactical level like so if an MM gives channels credit like for where they drive revenue >> y >> how you know if Meta drives Amazon sales and Shopify sales how do you think about how to segment that spend on a P&L basis because I think a lot of marketers it's a bit antiquated they put all meta- spend in DTOC >> yep >> and so you have different contribution profit on a financial basis. So it actually makes it tough for CFOs to

00:25:00 - 00:25:52

actually like >> listen I think like >> deploy spend >> there's no magical answer there. I mean this is why you you deploy you deploy an MM is there's only there's you know $100 million the brand made 60 million on DTOC $40 million on Amazon right pumping a bunch of Amazon ads but it's not going to drive 100% of the revenue on Amazon. There's that big pie piece of the pie that's missing. It's like, can I get a little bit of a signal? I talk to brands

00:25:27 - 00:26:11

all the time. I'm sure you as well. It's like, hey, I I turned off my I cut my spend down on Meta and my Amazon sales went down, right? What's also interesting is, hey, we're pumping Tik Tok shop. We're seeing a bunch of crazy information between Tik Tok shop and Amazon, which has been very interesting, right? People are Tik Tok shop's a big thing. There's a big halo effect on Amazon. >> How do you guys think about that? >> Um, >> it's so funny.

00:25:48 - 00:26:35

>> It's an awareness. Tik Tok shop is >> Yeah, it's discovery. >> It's a discovery play. We have a a a big pan company that we work with. We've worked with a few of them, but it was funny. The data was showing that people were going and looking on Tik Tok shop and then they were going to Amazon, check some reviews, and then they were going to buy the pans in Costco, right? And the guy was like, he's like, I know these are all statistical things, but back testing was insanely accurate,

00:26:12 - 00:26:55

right? Intuitively, it was like, this matches some of our intuition, and then it's one of these things you're on like a more of like a >> How do you test for that? Yeah. Well, it's a lot of like back testing and then they'll run incrementality result incre they'll run hold out tests at the same time. If the hold out tests are pretty close to what our measurement is, then it's like, all right, like this is starting to add up a little bit. >> Wow. Yeah. I mean, every Tik Tok I mean

00:26:33 - 00:27:28

we run Tik Tok shop. >> You guys doing Tik Tok? >> Yeah, we do. And like my whole thesis is like I want the platform to be profitable, but I just don't think like it's not yet a place where people feel comfortable buying their consumables. >> Agreed. they discover and like if you have a direct response like impulse purchase you buy there. Yeah. >> But like I'm not buying high AOV products there. >> What do you think is like a good category for Tik Tok shop and not a good

00:27:01 - 00:28:07

category category? >> I think like gadgets are good like electronics, gadgets, anything that's demonstrable like cheaper, lower volume. I mean, we've gotten other categories to work like hair care, fragrance, um, more like I would say premium CPG supplements, but I just feel like people like buying those products better if they're consumable on Amazon. And Amazon has this moat. They have they have the fulfillment network mode. >> Like, it's going to be a long time for

00:27:33 - 00:28:29

before Tik Tok can compete with that. >> Have you guys started talking about like the retail media networks at all? Has this come up? Yeah, I mean we do retail media. >> RMN's are big. I actually wanted to ask you like what you feel about RMN's because I I ask everyone like, you know, I think Amazon has a great retail media ecosystem, but I think a lot of the search retail media networks like I just can't I just don't understand how valuable those are going to be in a

00:28:01 - 00:28:49

world where you're like on Comet or Chat GBT getting, you know, [ __ ] bought for you. That's a whole another category like these probabilistic models and curious to see what data you get from that but we could have a whole session on that. These media networks everybody's going to create one that has some sort of marketplace or an audience. Go Puff like you know obviously Instacart, Gouff I mean >> Door Dash >> Door Dash I mean now we're we're as we're starting to expand into new

00:28:25 - 00:29:22

verticals um we're look we're doing a ton of research in this you have like Travelloity travel the travel platforms have their own media networks now fararmac pharmacies have their own media networks now everybody is trying to monetize the inventory and the audiences of people on there where our value ad comes in is well what if somebody is on you know target and they see, you know, Lucas's black bracelet and like, "Oh, that looks awesome. Let me go check out some of the reviews on Amazon." So,

00:28:54 - 00:29:49

we'll be able to quantify and as a third as an unbiased third party platform, right? Typically, the ad networks enjoy us as a partner because we're giving that additional credit from last click on Amazon or Shopify and then redistributing some of that back up to their media networks. I think they're here to stay and I think they're going to get wildly verticalized. I think the the yeah I mean retail media is obviously you know it's a huge part of the the the ad spend I can't remember

00:29:21 - 00:30:21

the exact figures but like it's growing at an astonishing pace >> in terms of how people are investing and that's because you have >> like a lot of them are in closed loop ecosystems like if you look at Amazon like their attribution data is really strong if you look at DSP for instance >> um >> and I think people want other outlets to spend especially if It's as close to the point of purchase as possible. It's like if I can, you know, on my target spend, if I know it's driving one P target

00:29:51 - 00:30:41

sales and it's driving velocity is like awesome. I'm going to keep spending there. >> You 100% might as well. One thing we've been getting asked is like, do you think that there is is there a halo effect from Amazon onto DTOC? >> Um, for the top of funnel advertising, yeah. >> Yeah, >> I think so for sure. Do you think people go to Amazon and like search something, see it, and then will go over to the website and then purchase from there? >> I think in certain categories like

00:30:16 - 00:31:05

Whoop, I was actually thinking about this Whoop. I would not buy >> Great call, dude. I wouldn't want to buy from Amazon. I don't doesn't feel as authentic, >> but I would see a Whoop top ofunnel ad on Prime. >> Yeah. >> Or like, you know, streaming. And then if I'm on the wearable category or I'm on Amazon and I get with an ad, I would be like, "Oh [ __ ] I should buy that." Actually, it's a reminder. And then I'd buy on DTOC because their DDC experience is

00:30:41 - 00:31:24

awesome. >> It's great, right? >> Yeah. But I don't think there's many categories like that. >> I don't think so either. >> I don't think because we've been some people have said like, "Hey, can you do the halo effects on Amazon to DC?" And like we've asked our clients being like, "Is that here's the five things that we can go build is one of those things. Building a birectional halo effect onto Dc from Amazon." And it hasn't been

00:31:02 - 00:31:39

>> like no but Amazon's a it's a demand capture platform. >> It's a demand capture platform. But the point you just brought up about the categories, it it hit it felt. I'm like, actually, that checks out because I would do the same thing. Yeah. >> But I don't think it's enough categories. >> I think as Amazon builds out, like they just had that Roku partnership like they're doing a lot more streaming like >> there could be an additional Halo

00:31:21 - 00:32:26

effect. >> Yeah. Yeah. Yeah. >> When when do you think of your brands like do you talk to of your brands when you or clients or friends in the ecosystem? Um what's like the pulse or the take or the perception on measurement in your a facade? Is there too much stuff? Is it >> No, I think people actually care about it. I think they they it's like super critical and important. I think it's still at the point like you know I was pushing MM like post iOS 14 and like yeah

00:31:52 - 00:32:46

>> I am telling you like those uh refresh meetings quarterly like >> like you know people listen they're like oh this is interesting but they were not using it to make decisions right >> and I think now we have a software layer that's like what you're doing that's actually getting people to make decisions that help them you know drive better returns >> um I do think there's by and large like you're just scratching the surface. I think most of the market is still stuck

00:32:20 - 00:33:13

in their antiquated ways of just having marketers or agencies create spend optimization signals based on like whatever they're feeling. >> Y >> like I have clients today that literally look at channel rorowaz >> within the platform. >> Yeah. Yeah. Yeah. And we've pitched you and I have joint pitched >> Yeah. deals and like I don't know what it is like what why why don't you adopt but people are stuck in their ways. >> They they are they built good businesses

00:32:47 - 00:33:50

on >> it's gonna take 10 more years for you to get full penetration on MM. >> It's going to take time you know we're actually moving away from the measurement space in in a sense where um really excited we we announced it last week a new model release called Omen. Um we are an intelligence layer that sits on top of North Beam that sits on top of Trip Oil. It sits on top of um MTA incrementality post-purchase even other MM can come in here and here was the ethos is >> oh you have other how does other M&M

00:33:18 - 00:34:31

>> MM um >> how does that work? >> An MM is going to always have what they call a prior. This prior is a prior belief of a prior belief of what a what what was this measurement on this time to inform the recommendation engine right it always has to has that and typically the mm is supposed to be they call it a calibration with incrementality data right that's what people say because they believe that incrementality data is you know kind of that source of truth at that point in time so we have our own

00:33:54 - 00:35:11

halo effect measurement right but we have a validation before a calibration layer. Meaning, let me give you an example. We say that this tactic has a 32 rorowaz. MTA says it's a 34. Post-purchase p survey say it's a 36. Meridian calls it a 41. And the incrementality data calls it a 44. Right? So, you have all different forms of measurement that have different levels of rorowaz for this tactic. Now what we now do is we back test each form of measurement within our models and we give it to the marketer and say you

00:34:33 - 00:35:17

don't you want to you don't believe our math that's okay you know I don't want to be in the he said she said anymore here's the data you pick which form of measurement you want to inform our optimization models >> I like that >> right because well why >> choose your own path >> choose your own path well since our models update every single day at a campaign level you're going to get the feedback loop if something's going to be work All models are wrong, but some are

00:34:55 - 00:35:46

useful. This is the most useful model for forecasting and optimization. We use state-of-the-art techniques for the saturation plots. >> People have different preferences, too. >> 1,000%. And so, you know, we're actively chatting with some of the people in the ecosystem that I think people would be like, "Wow, these guys are doing business together." >> Yeah. >> Um, and it's just it's part of our strategic direction as >> that's great. You want to be the layer

00:35:20 - 00:36:24

on top. universal intelligence layer for enterprise automation is where we're going to take this business. >> Are you uh I feel like I have to ask this. Are you building a chat interface like people want to talk to the the agent the strategist? >> Yeah. Um initial prototype went live last week. You'll be able to and it's very cool. You'll be able to ask very like what is the relationship between my top of funnel and seasonality or how much does seasonality play a role

00:35:52 - 00:36:51

in June? Right? When should I cut most of my spend just because seasonality is going to do its work? Those are just more of the simplified questions. But imagine if you said, "Hey, I have a target I need to hit over the next three months and how do I do it?" Eventually, we're going to buy that media automatically for them for the channels that you can buy it. I think dashboards go away. I think people are just going to get a Slack message or a text message that says, "Hey, you're on track to hit that

00:36:22 - 00:37:12

uh you're on track to hit that target." Or just like Google Maps, I found a five minute faster path. You can stay where you're going if you want, but I found a five-minute faster path. a models are going to go back to a CFO and a CMO and say, "Hey, I found an opportunity to find a 5% incremental lift over the forecasted baseline that we're predicting now. Do you want to execute?" Yes. No. More info. Drops down more info. Gives them the information. Yes. Execute it. And then the models just

00:36:46 - 00:37:35

start buying. So, what do you need a dashboard for? >> No, you just want to talk to someone that knows what they're doing. I >> think I think that's wildly the future. Now the agents that don't have the computer learning system that gets smarter and smarter, right? >> They're at a disadvantage. >> Well, it all matters, dude. Look at Clara. Clara had 7 700 customer services reps and I was like, "Okay, we're going to fire them all. We're going to put

00:37:10 - 00:38:08

agents down on them." Catastrophic. And this is just for customer service chat bots. When you're talking about media budgets and highly, highly dynamic, complex, competitive situations, one extra zero, you're [ __ ] You know what I mean? Chevrolet has a chatbot that they're selling cars for a dollar. Air Canada had chatbot that gave wrong information from a legal perspective. >> There's a lot of risk. >> It's a huge amount of risk. Yeah, you need great It's interesting because like

00:37:40 - 00:38:29

the the competitive advantage just becomes like the data source, the model >> and the sophistication of the math. Yeah, the tech. How good is it? >> The math matters and I try to talk to people all the time about that. They don't give a [ __ ] about it to be honest with you. It's like >> do you I took a recommendation. Did you make me more money or not? >> Yeah. Yeah. That's performance marketing right there. >> Performance marketing, man. It's a But

00:38:05 - 00:38:49

it's all dude, it's all the same. like we're working with a a billion dollar betting site and they're like we want to be able to run ads and predict new customer deposits for the football season by Sunday football games, right? They were only thing they care about is did I get new more new customer deposits? Did I sell more shoes? The industry doesn't matter. It's all the same question. >> So Mike, how big is your team right now? Like to do all of what you're describing

00:38:27 - 00:39:26

like how how much did it take? I know you guys raised a you raised a $10 million series A, right? >> Yeah, raised about 18 million so far. We spent three straight years of R&D. 80% 90 89% of the company was all you know multiple PhDs from Stanford and computation like just absolute [ __ ] beast. We spent three years building this model completely from scratch questioning all the other open source research papers. That was the hard part up front and now the team is starting to balance more out on the commercial side.

00:38:56 - 00:39:52

So was heavy tech but it's just really mean. Shout out to Laura um sales rep. like her and I just turned it on over this last year together. Uh and you know we're next phase of the business of growing that team and capital and the next stage is you know coming around the corner. >> What uh what can you share with us about the future? >> Um I think you're going to start to see a lot of surprising partnerships, dude. Yeah. I think you're going to be like these these guys are buddies. These guys

00:39:25 - 00:40:18

are friends. >> They're homies. >> Yeah. I just >> What about North Beam? I don't know about that. I I think like um you know I've I've met pretty much all of the all of the other measurement founders. I haven't I haven't connected with her. They built a 90 90% of our tier one clients are running I would say a combination of us north beam in house or or measured layer in there as well. But >> where does triple well fit into the equation? >> I saw triple everywhere man. We I still

00:39:51 - 00:40:49

we still send triple and north beam northbeam three to five deals a week easy deals that will come in just met Google on Shopify. um proven validated multi-touch attribution solution that it's that's great. Um I don't know what's going to happen with with MTA. My buddy Hans from Broomate, shout out to Hans. Goes takes me skydiving and stuff. He was just sitting there talking about like his measurement stack is largely starting to migrate away from the MTA because they have such a good pulse on met Google. you know it

00:40:20 - 00:41:13

in and out and it's like I'm really trying to dial in on my retail presence and I'm using you know post-purchase surveys. I'm using incrementality and mm um to try to validate that. But so I think to answer that was really long-winded answer. I think um I think you're going to see our team grow pretty quickly. You're going to see us verticalize into different verticals and you're going to see us talking less and less about measurement and more and more about automation. So on the on the

00:40:46 - 00:41:39

product side, you have like smart PhDs, engineers, like people building product, and then it's you and Laura on sales. >> I think mostly Lauren Will Holtz, like he's >> I know Will, >> he's strategy and ops, but like he's in the thick of it. Him and I have been partnering more on like the airlines, the more of like the up enterprise. >> Yeah. Yeah. Yeah. >> Yeah. The thing about Mike is Mike is a uh he's a sales machine. Enterprise sales machine. Yeah, it was it was

00:41:12 - 00:41:52

interesting coming into selling to the I love the Shopify community more than when I was at IBM wearing a suit and on a plane all the time. Like I've met some of the like I met you, you know, I met some of my dearest closest friends of people that are just building cool ass [ __ ] But it was interesting me to come in different >> I'm like okay like you just took the credit card and you ran it. Like >> you're like I need to get the [ __ ] out of this. Listen, it's there's just a lot

00:41:33 - 00:42:22

at stake for people building this [ __ ] and I respect that and they're making like big decisions off of this. It's like any day could be your last. That's that's how it feels. >> Um, but the ecosystems is just like I mean when when you think about your agency, right, and you're starting to see the LLMs and and I mean you just were so like more full service. You've already thought about building MM but like when did matter start to like really pop in your head being like are

00:41:58 - 00:42:49

we going to put hands on the keyboard and we need to differentiate ourselves in a way? >> Um I think LA actually last year in can like one yeah one year ago. Well, because me and my co-founder, we just started, you know, we were on the sidelines, you know, when you're within an agency, you have this vantage point where you like you, you know, the software providers, you see them, you see them go through their cycles of like fundraising and the peaks and troughs and you know how they kind of navigate and then you

00:42:23 - 00:43:12

also see the brand side too. You see like a ton of >> absolute crashes. >> You get that you get the holistic view. >> Yeah. Yeah. It's kind of like it's consulting, you know, you just get crazy exposure or like VC. >> Yeah. And you know the you know the ad networks you got reps at meta and so you get >> but we we saw the AI wave coming and we were like whoa like we need to we need to future proof the agency one but also like there's a huge wave happening in AI

00:42:48 - 00:43:38

that we need to be a part of >> just as like entrepreneurs like just >> we need to have >> you got to show up to that race. >> Exactly. It's like generational opportunities. >> What's next for you guys? So, I mean, we've divided the we've divided the the pie in that, you know, we have these two sister companies. We have Dark Room, which is the agency. >> It's growing at like 60%. >> Wow. >> Love that. >> Cash, you know, 30% margins like

00:43:13 - 00:44:02

>> Yeah, >> it's doing well. Um, and then we have this technology business which is it's it's in its infasy. We really started building past 6 months. Jackson designed the interface. He's a world class designer. >> That's right. That's right. You guys are OG on the design front, >> dude. He's a world class designer, one of the best in in the game in this ecosystem by far. >> I'm excited to meet him. >> Um, he's a great guy as well. Um,

00:43:37 - 00:44:29

>> and two engineers cobbled together that prototype that that you saw that went went a little viral and we got a ton of signups. But our whole thing is like we want to we kind of want to on the agency side we want to build like a palunteer type like you have engineers and highlevel strategic consultants working on commerce businesses but they're empower empowered by like a commerce layer like a software layer% >> that software layer is going to be matter. >> It's going to be something that

00:44:03 - 00:44:46

aggregates data from a bunch of different data sources and does a lot of the the execution work that we've traditionally done >> in our outsource hubs or with our agency works. the junior talent that's just like [ __ ] executing. >> Yeah, 100%. >> So, that's the that's the path. Right now, we're thinking about how to we're probably going to like hold co that. We're talking through like is that a hold or are these two separate entities? Yeah. What does that look like?

00:44:24 - 00:45:17

>> I mean, I think you go either way. One thing you just said that's very much spot on of your strategy is having kind of a labs approach. We call it precient labs. This is just like the consultant side of things. We're taking in skunk works R&D working with different design partners. see what sticks. So, you're not like dumping in all these engineers to go build something. You kind of test it out in like a little like dev environment, if you will. Um, and then be like, does

00:44:51 - 00:45:33

it make sense to layer this into the product or not? >> That's why we created two separate companies. We're like, this needs to be unencumbered by like the priorities of the agency. >> Wildly smart, but you can use the transfer learning from one to the other and then monetize the relationships. It's a that's actually a beautiful >> Yeah, it's it's complicated, but like we're this is what we're talking through right now. Yeah, it's a good flywheel

00:45:12 - 00:46:01

though. Like this is like >> continuous learning cycle. >> Our thing though is like we think we can bootstrap the software business to like 510 million ARR. >> No, they're doing >> we're like we don't think we need to raise funds for that. We just like >> you're in a good spot for >> takeida put it into the business like keep it lean >> non-dilutive capital. >> Yeah. You could do, you know, series B is on our horizon, man. And like, you

00:45:37 - 00:46:17

know, we're we're on the hamster wheel and we started on the hamster wheel and there's no way off of it. No, we built all the models and we had to deploy them and it was to deploy. >> You need cash. You need a lot of money. >> Very specialized engineers and research scientists to be competitive in the space. >> Let me ask you though, could you think you could have done it, you raised 18. Do you think you could have done it leaner? >> It's hard to say, dude, because like we

00:45:58 - 00:46:43

hired the best and the brightest to go get these models deployed. But how do you determine the like how are you like oh you know what we're going to raise 10 like we need 10. >> It's a good question. There's a lot of assumptions that go into it. But my co-founder is just a tenure guy in the space. He was like it's going to take me worst case scenario you kind of always plan for the worst. Worst case scenario it's going to take me 18 months to get these models pipelines built models

00:46:21 - 00:47:11

deployed UIUX things everything up and running. you raise enough money to make sure you have those engineers for you know three years >> and you you tell the board and your investors like don't expect any revenue to come for the first two years like if you want to if you want to if you if you think the future is AI and right we were building AI in 2020 when we first started building these >> Yeah true >> and >> no cursor back then >> no cursor back then it's funny dude the

00:46:45 - 00:47:32

I was on with some consultants the other day and he goes I have a Christmas present for you because I was talking to them about the way that we've we use a lot of neural networks in our models which is very unth thought of in the order of MM space. Everybody says we have this Beijian model. This is like the big hot thing. We have this Beijian >> Beijian models. People love the Beijian model, >> right? It takes you know super expensive to run and slow and so we have a lot of neural networks in our models and this

00:47:09 - 00:47:57

is what our our value prop had been. Google comes out with a paper last in June last month. It said NNN next generational neural networks are the future of marketing measurement. We've been building these next generational neural networks for five years. And that's why we're so excited about this compound AI. It's like the flexibility of what these neural networks can do. We have Beijian models. We have four different AI models that are all all ensembled together that they all speak

00:47:33 - 00:48:18

to each other and break everything apart and do it fast and cheap and that's why you can run it every single day. But this game is this is what [ __ ] fires me up to it is like this is a game about math and now people are starting to pay attention that the math matters. >> Yeah. It's math. It's >> education, >> sophistication, it's tech, and then it's also like speed and knowing where the market's going. >> That's my favorite thing about this ecom

00:47:55 - 00:48:47

space and like startups. It's like you it's really a testament of like how much of a [ __ ] like savage you are at executing >> and like what is your risk tolerance to be like like you know chatting with some VCs I'm like listen like there's we want people in your corner that's going to say you know we're going into this world we have no [ __ ] clue what's going to happen but we know that there's going to be a lot of movement from an artificial intelligence space we know exactly what

00:48:21 - 00:49:13

you have here if you want to go make some sort of pivot because you see something happen in the ecosystem we have your Right? And that's going to be the people who are ready to take a [ __ ] swing when they see something and do it confidently with no ego and just let it rip and you want to win. Like there's going to be some absolute behemoth companies that are going to be produced outside of this just this next generation of of AI. And you know, I think it's I didn't think I didn't know

00:48:48 - 00:49:30

Matter was coming out that fast, dude. It's like it went you guys did it quick. It's out. And now you get it tested and you just keep iterating on it and >> Yeah. Yeah. Yeah. No, it's moving fast. Let's see. I I'm excited for for you guys to chat. We'll see if we can integrate. >> Yeah, that'd be great, dude. >> Dude, thanks for coming on. I appreciate it. We actually finally got this done. We for for the audience. We had to reschedule this like I think three times

00:49:07 - 00:49:42

twice in can cuz this guy was uh he was doing too many sales. >> Yeah. >> Too too many meetings. >> But uh >> Well, I'm glad I'm glad I made it happen. And uh >> yeah, this was great. >> It's good to see you. Let's go get a beer. >> Let's grab a beer. Yeah. [Music]