Why the AI Policy Debate Should Focus More on the Harness and Protocol Layers

Audio of this conversation is available via your favorite podcast service.

Raffi Krikorian, the chief technology officer of Mozilla, has spent the past few months building an argument that the central question in AI isn't ‘open versus closed,’ but ‘owning versus renting’—whether AI becomes something we control or something we lease from a handful of companies. A technologist by background with stops at Twitter, Uber, and the Democratic National Committee, he writes about this question in his newsletter, Owners Not Renters, and in other outlets, most recently in a New York Times op-ed on what he called the "Mythos moment."

I spoke to Krikorian about the idea that generosity is the hidden infrastructure of the internet, how to expand access to powerful AI tools rather than closing it down for security's sake, how to overcome misaligned incentives to build a better information environment, how to counter surveillance, and why those concerned with AI policy and governance should spend more time thinking about the protocol and harness layers.

What follows is a lightly edited transcript of the discussion.

Raffi Krikorian:

Raffi Krikorian. I am the CTO of Mozilla.

Justin Hendrix:

Raffi, I'm excited to talk to you. I've had the opportunity, of course, to meet you in the past in your role at Emerson Collective, where I had the chance to be a fellow a few years ago. And I appreciate the fact that you invited me on your podcast a while back. So I'm so pleased to get the chance to return the favor. I'll also say this podcast, my listeners know we host a lot of tech criticism. And while I know we'll have a critical conversation today, you also are, I think, fundamentally an optimist. That is, I believe, what your podcast was literally called if I'm correct.

Raffi Krikorian:

Technically Optimistic.

Justin Hendrix:

Yeah. And I think it's good also maybe to do a level set on where we're at on some of these technologies, particularly with artificial intelligence and what policy makers should be thinking about when it comes to these things.

I want to start with your New York Times op-ed recently. You sort of laid out an open source thesis, this vision of the implications of the Mythos moment, what it means for the broad enjoyment of technology's benefits that we've experienced, or at least some would frame as what we've experienced with the internet over the last few decades. Why was the Mythos moment important to you? Why did it strike you as a moment to pick up a pen?

Raffi Krikorian:

Yeah, that's a great question. So let me give you some background of where I was coming from. So Anthropic released Mythos, or released it to a small set of companies and organizations. Fortunately, the Firefox team at Mozilla was one of those organizations. And what Mythos did for us at least is it found a slew of exploits. The Firefox code base is pretty old, not old in the sense that it's janky and crusty. It's old in the sense that it's been an old open source code base. There's been, I can't even count how many contributors have looked at every line in this code base. And Mythos comes around and discovers all these vulnerabilities that humans have just missed in the decades that Firefox code has been around. And so that for me was this aha moment of like, "Oh no, this is different. Something just happened where this balance and open source has changed."

So in the sense that software up until this point or up until was called November of 2025 last year was hard to write. And right now we're living in this golden age of writing software where software writing has become free all of a sudden. A dentist can describe a piece of code they want for their website and some agent will just write it for them. But now all of a sudden what Mythos showed us is possible is that finding bugs has also become free. And so the world has existed in this balancing time where writing code was hard, but finding bugs was hard. So we lived in this weird day and time that not many people wrote code, but not many people found bugs in the code that's being written. And the entire internet is now hoisted on the shoulders of open source code. There's code like FFmpeg that is in every single web browser, but it's probably also on YouTube and Vimeo and all these other streaming services.

It processes video and it's like the de facto tool you would use to process video or like cURL, which fetches webpages from the internet. My son's PlayStation has cURL on it. Every single desktop computer probably has some variant of cURL in it. And so all these pieces of code that authors have painstakingly written over the years, which now buttressed the internet, Mythos is coming around and finding the vulnerabilities in all of them. And so the fact that this equation has changed, software is now easy to write and bugs are easy to find could be a pretty bad thing or it could be amazing. It could be a pretty bad thing because it means that attackers could come around and find bugs in all this critical software that holds up the internet. So for me, the point of the piece was to A, point out that this imbalance or this balance has just become an imbalance, but also the social contract that we use to hold up the internet that a few people through generosity have written large amounts of open source software that now power the web, but that generosity is under attack.

There's only two people that maintained OpenSSL and a few years ago we saw a bug in OpenSSL called Heartbleed that caused us all to question the security of the internet, but it was two people. And so all these companies are spending all this time and energy writing software, but what's really holding up the internet is generosity. Mythos, for me, exploded that generosity.

Justin Hendrix:

So the implications of that are it's a closing up a security moment, a securitization moment on some level. We have a piece from Konstantinos Komaitis this week who's talking about AI as potentially introducing a new logic to the internet as well, similar line of thinking, I think on some level, at least some degree of congruence. And he's sort of imagining this world where cybersecurity eats up all the questions. Everything becomes about cybersecurity, everything becomes about defense. Is that similar to what you're describing as the worst case scenario here?

Raffi Krikorian:

Yeah, potentially my worst case scenario. But again, I'm an optimist. Let's think about it from a different angle for a second. So we have to ask ourselves what's the steady state that we want and then what do we do in this transition state to get the steady state? So I think one of the worlds I want is that all our software is just secure. We are actually building software that's secure by design. CISA did this whole push in the last administration for software that's secure by design. So I would love to get there. That's minorly in conflict with my dentist vibe coding all of a sudden, but we can probably get there if we do a concerted effort. And then I want to get to a place that our internet is secure, that we've actually found all the vulnerabilities and we're living in this way better spot that we're living at today, that we're living this slightly constant fear every single day around this critical infrastructure that maybe we get to a place that's all secure already because Mythos and Mythos-like tools have found all those bugs.

That would be an amazing world to live in. So the question is, what do we do right now? And so a lot of people are making this argument that for security's sake, we need to actually close up access to all these models because they're dual use. They can be offensive and defensive. And what I would argue instead is we need to do a concerted effort and investment that we should be like... Anthropic pledged something on the order of $4 million to open source communities to use advanced tools like Mythos to help with find bugs. Imagine a world where every single company that relied on open source did some form of contribution to accelerate the fixing of all these bugs and the deployment of all these bugs.

I want to get to a world where security isn't causing us to shut down access, but in fact, we can use this moment to accelerate access to the right people under the right measures so we can actually get to a world that's secure. I think that's the world I want to get to. So I'd rather think about that than the worst case scenario.

Justin Hendrix:

One of the other things that's been discussed on Tech Policy Press around this Mythos moment is the general inequality, who's been given access, who's in the know about what the reality of the threat actually is, who gets to sit around the table to talk about what should be done. I don't know, is that part of the issue here too? I mean, you're one of the people who I suppose... maybe not you yourself, but your teams have had the opportunity to at least get under the hood here.

Raffi Krikorian:

I mean, I agree you're onto something there in the sense that it sort of matches my line of we need to expand access, not restrict access. Security in my mind shouldn't be something for the rarefied view. I'm very grateful that Anthropic partnered with us to find all these bugs in Firefox. I think that Firefox is an important almost digital public good in the world that needs to exist, needs to be secure. People can trust things like Firefox or Firefox specifically, but at the same time, I think you're right that we can't be restricting it only to the rarefied few. The security tax that sort of exists today is generally a bad thing. If you go to any piece of enterprise SaaS software, if you want to get SSO, single sign-on, you want to get two-factor authentication, you want to get all these things, these companies are asking me to pay more money to get access to features that will make me more secure.

And I think that's a bad anti-pattern. We need to figure out how that security is by default given to all of us in some way, shape or form. And so access to mythos is one of those access to mythos and mythos like tools. There are a bunch of open source projects and GPT, OpenAI now has their GPT Cyber. So there are a bunch of different initiatives to try to build tools to do this type of scanning. We need to make sure that the right people actually have access to them across the board, whether it's even just one person who works on cURL, but again, that software is on every single PlayStation. We need to make sure access is more uniformly distributed.

Justin Hendrix:

I want to also just ask you about this idea that you have brought up in past around this problem of renting versus owning or the kind of rentier economics of AI that we appear to be building into the ecosystem. And this sort of feels like it sort of stretches from that a little bit that I want to get hooked on a handful of providers who are going to give me access to the main line, the best possible security agents, et cetera, et cetera. And I'll just have to wait until they upgrade their product. I can never really own my own security. That feels like a problem.

Raffi Krikorian:

Yeah. I mean, the renters versus owner's frame I think applies here too. So just to make it, just to stay what it is, I generally believe that in the last, let's call it 10, 15, 20 years of the internet, we've slewed toward this model of renting our access instead of owning our access. So we see this right now in things like the way that we access these intelligent services. So OpenAI’s ChatGPT, Anthropic’s Claude, et cetera, we are renting access to them. We pay n dollars a month to get access to these models who honestly, their incentives are not necessarily aligned with mine. They're aligned with making more money. We see this in the way that these models are talking to us, they're designed to actually get us to engage more. We've seen some horrible things about the way that ChatGPT, and I'm not implicating them, just sort of stating the horrible things that ChatGPT has said to children that shows that their incentives aren't aligned.

There's a paper that just came out between Princeton and UW, which has solidified what we all were thinking that if you ask one of these agents for a recommendation on a good to purchase that they're more likely to recommend a sponsored thing versus what you might actually want, or they might be doing some form of social economic modeling of you in order to try to see how much money they can extract as opposed to exam giving you the thing that they want. So we're seeing all these systems which are designed for the profit and motivations of the providers versus ourselves. And again, this is not new. We saw this in the social media era. We saw this before Google was putting sponsored as a note on their tags when they do page rank. So this is a pattern that keeps on showing up. We're just seeing the latest iteration of it.

And so when I say owners, not renters, I want to get to a world, especially if we're going to get to a world of AGI, which I know is a very American-centric way of looking at these open source technologies, but I don't want seven AGIs. I want seven billion AGIs. I want an AGI that's truly on my side and taking care of my interests and someone that's making decisions on my behalf, someone that I can share confidential information with, I can share healthcare information with, I can share all these things with that I know it's not being used as a surveillance mechanism. I know it's not being used to profit someone else's bottom line. And so that's what I mean by this owners, not renters frame.

And so bringing it back to security, I think the same thing is happening here too of just like I'm not getting stuff that's architecturally secure. I'm getting security through the fact that I'm paying another vendor. So a good example here is to think about the way Signal operates. What Meredith and her team, Moxie, et cetera, have all built is something that even they can't snoop in on the messages. If a police department of FBI, if a foreign national comes by and asks Signal for the contents of Raffi and Justin's Signal transcripts, they literally cannot give it to them because it's just architected in a way that's secure. I think that's, in my mind, how ownership should work is that we're getting things that are architecturally safe for us to go use. And we are so far away from that in this AI world, and that's the world that I think we all need to be chasing more.

Justin Hendrix:

Well, I do want to ask you a few questions about what you see ahead for certain technologies with artificial intelligence. But I think many of my listeners, again, who are accustomed to critical AI discussion on our podcast, they hear you talking about AGI, they hear you talking about someone that probably immediately pegs for them in their minds that you are definitely on one end of the spectrum in terms of where you think the technology might go. I mean, do you think that's where we're headed to, some form of artificial general intelligence?

Raffi Krikorian:

Well, I mean the definition of AGI is such a hazy one that if I were being super precise, I put my engineering hat on, I can't say for sure that's what we're aiming towards or that's where we're headed. But what I can say is that if you look at consumer data right now, people are generally switching to these types of interfaces. People want an interface to information that seems to be more conversational or seems to be more able to take in a lot of context or seems to be able to be answering questions or help you automate workflows. I was just looking at Google search query traffic over the last 12 to 24 months and it's not going down but definitely took a dent. And if we look at the number of people that end up on a Google search page and don't click on anything else partially due to AI summaries on that page or maybe something else, that seems to be on the rise.

So people seem to be craving these other types of experiences. So if I were to speak for myself specifically, I'm running the Hermes agent right now on one of the computers in my rack. Hermes is written by a company called News Research. It's an open source agentic framework and it's kind of great. I have to admit it. I have it wired up in a very specific way to make sure that it can't do anything super crazy. It's not going to send an email on my behalf. It's better than I have access to my bank account, but the people who are doing that with OpenClaw are crazy in my opinion. But I can text my Hermes agent a picture of what I just ate for lunch and it knows what's my fitness tracker that I use and logs my food for me. I can ask it questions of just like, "When do you think I can go visit the doctor if you look at my calendar in the next few weeks?" And it gives me all my open slots.

So I do think that people are craving that kind of way of integrating information into their lifestyle, but is it something I'm going to call someone or I'm going to call it a something or is it something that's truly AGI? I don't know. I have been... maybe as a last footnote, I have been really excited and enamored by the scientific breakthrough work that's been going through the people who have been pointing some of these really large models like questions around genome, science, et cetera. That's been super interesting to me. Of course, open questions around, are we actually generating more science or are we just generating more things to be tested in wet labs and stuff like that? But there is something here that we all need to be paying attention to more carefully and I think there's an opportunity like I did with my agent that we have to have questions around permissions and boundaries and how it actually interfaces with the real world that we need to still sort through.

Justin Hendrix:

So I have so many different questions that stem from these issues around agents in particular. And you make me think that I once heard Kate Starbird at the University of Washington say that her field, human-computer interaction, if you look back at the beginnings of it, the whole idea was we want to talk to computers. And that was fundamentally where everybody wanted to get to. Most of the time in the middle from that point to today, we've had this priesthood of computer scientists, developers, coders who knew how to talk to computers and effectively the priesthood is broken. That's what's happened perhaps since November of 2025 as you try to hazily mark the date there. But I guess I want to ask you, do you think that something has fundamentally changed with quote-unquote agentic AI? Is this just software by another name? Is this hype or do you think there's something legitimately new here?

Raffi Krikorian:

I mean, sorry, you're not going to like my answer because it's yes to all of the above in some way. I mean, the biggest difference in my opinion is that we are working with these probabilistic systems. The engineer in me still gets shivers and chills on being like, "I don't exactly know what's about to happen when I type it and ask it to do this thing," which is kind of crazy, but in some way, maybe that's the way the world works. We're all searching for metaphors on how this all plugs together. If we were to ask another human to do something, I don't exactly know what it's going to do. Sometimes it surprises me, sometimes it disappoints me. We have to wonder about what that contract is supposed to look like and what that really looks like.

One of the difference here though is that the fact that we're allowing these systems to get access to way more information than we would've traditionally allowed software to get access to and then allow it to use that as part of its context, as way to synthesize what that decision it should do is, that is relatively new. I think that no one would've predicted, or except for a few sci-fi writers, that being able to assimilate all this information, not just generically what's on the web, but all the context of a particular human and then try to make a probabilistic guess of what to do based on it, I'm not sure anyone would've bet on that. And because we haven't bet on that, we have a whole set of questions that stem from it, like what's the legal permissions that we want to give this thing that might be acting on my behalf? Is it a fiduciary? Is it not a fiduciary? Is it acting with my consent? Is it not acting with my consent? What do we do when things make a mistake? Are there architectural questions we should do to have a permissions boundary around it?

So yes, it is quote-unquote just software, but maybe with a probabilistic spin on it. It's different because we're now giving it access to so much more context that it can try to synthesize and try to make a guess about what my intent looks like in a way that we were just never able to do before. And then finally, there is a real big difference, which is just the economics around this are so fundamentally different. The fact that a single query for me is spinning up a data center somewhere, that's a big deal difference that we really need to really think about. And if you asked me last year, I would've said something like the per token cost is going to go down, it has gone down, but the number of tokens has exploded in the process.

And so there is still these open questions of economics and architecture that need to be sorted out here that I feel like again, we're in this open transaction moment that given a year, given five years will be settled. The question is just like, can we handle the chaos that we're about to cause in these next five years? I don't have a good answer.

Justin Hendrix:

And we probably don't know exactly what the economics look like until effectively the market settles out a bit. There's a lot of hype and too much money from all kinds of different places coursing through the system that are probably perverting all the economics.

Raffi Krikorian:

Well, I mean, I'll just give you a sense of what it looks like for me. I mean, I will admit I'm a semi-Alpha user right now. I do it partially because it's fun, partially because it's research, partially so I can have an intelligent conversation with you, Justin, but running my Hermes agent automating a bunch of stuff that I'm doing, looking at my calendar, all that kind of stuff. It's like $1,500 a month, and that clearly is not a sustainable price point. So as the market shakes out, that might even get more expensive because VC dollars have dried up or we might need to see architecture shift to running on my GPU inside of my laptop instead of on a remote data center, which might get it down to more like $50 a month, which is still too expensive for most users, or it opens up the question of just like, how is the internet funded these days?

So I think that economic shakeout of VCs drying up, token costs exploding, architectures becoming cheaper, moving more to local. It's sort of a thumb in the air how that's all going to play out, but there's so many variables here that's part of the equation.

Justin Hendrix:

So I want to ask you about another issue that perhaps Tech Policy Press readers are paying some attention to, but I feel like most folks in the policy community still aren't really sure what you're talking about when you say the letters MCP, but you clearly think that something important could be going on here, that at this particular layer, this could change the information ecosystem quite profoundly.

Raffi Krikorian:

Yeah. So maybe an analogy that I try with people, you can tell me if it works, is the web right now is powered off of what's called HTTP, the Hypertext Transport Protocol. It's what ships HTML around. So if I bring up a web browser and type in a domain name like Tech Policy Press, I go to that server and it ships me back some HTML, so some content with some structured markup around it that tells me how to render it on my web browser. But agents or these autonomous systems or LLMs in general don't care about the way that the world is structured. Honestly, they just want the information. And so the question is, how do we get the information to them in the most reasonable and most efficient way? So people have come up with or Anthropic came up with specifically, but they've donated the protocol, quote-unquote, to MCP, the Model Context Protocol.

And what it's meant to do is it allows an LLM to call to get more context and that calling more to get more context could be it just wants to read more information or it could mean that it's asking something to actuate something in the world to therefore get more information. So two examples of this, I've had conversations with publications about whether or not they need to not only have an HTML version of their site, but an MCP version of their site so they can better control what kind of content an LLM might extract from them and they can set permissions or pricing models. What would an MCP version of Tech Policy Press look like? And could you do billing on it? What are all the things that we think about that you could maybe put on that? So MCP I think is really interesting in the world of like, can we get access to different kinds of information?

Imagine a world where I had a subscription to the New York Times and they exposed an MCP for subscribers. So all of a sudden my chat agent could use my subscription to get access to more information than what they could get on the New York Times website without being logged in. And the New York Times currently hates all LLMs so you can't get anything, but think about that as a particular example. But another way to think about MCP is also tool calling. So an LLM right now can just on its own synthesize a bunch of text. So you give it a bunch of text, it figures out what to do with it, but if you allow it to call a tool, it could actuate something and then get the response back. So you could ask it to call a tool to change a design on Figma and then all of a sudden a user would see a different design in front of them and then might react to it and give them more information.

Or there's a pretty entertaining, but I think important website out there called HumanAPI, which is an MCP server to humans. So an LLM could call HumanAPI to ask a person to actually do something in the world. So all of a sudden the LLM is like puppeting humans around in the world, but those humans could be doing things. They could pick up a package, they could hold up a sign, they could go to a protest. So all this stuff could start happening through MCP that is both interesting and terrifying. We need to wrap our heads around a bit better. It's similar to how in the '90s we all went around talking about both the powers and perils of the internet because all of a sudden the world had gotten flat and humans could see the entire world through the web.

MCP is the same thing for agents and the MCP is the same thing for LLMs of like the world is becoming flat and people are enabling them to get access to more information and to actuate change in the world. So we have to figure out what to do here because this is actually going to be a big fundamental shift.

Justin Hendrix:

So this raises so many different questions for me. But again, this is the Tech Policy Press podcast, so we're interested in questions around policy, questions around governance. So I'm immediately wondering about permission models, who can audit this protocol, who can see into it, who knows what's going on, who has power, who doesn't, who controls the layer?

Raffi Krikorian:

Thankfully, you're not alone in these questions. I've had conversations with people like networking providers, people who make switches that go into data center racks and they've spent a lot of money and made a lot of money building traceability into the web. So you can understand who's clicking on what, where are they coming from, what they did next. And they're all asking the exact same question now of what do we do with agents, LLMs, MCP, how do we build traceability like that into our fundamental infrastructure?

But you're right, on the policy side, the HumanAPI thing is really interesting for me because it is an example of now humans are becoming puppets to computers, which sounds very matrix-y. But at the same time, it's just like, what if an LLM asks someone to do something that's kind of inappropriate? Who's now responsible in that situation? How does that actually work? Are we logging it appropriately so we can do investigation on it later? What is the trust and safety angle on all that? All these questions are both terrifying and greenfield to be answered.

Justin Hendrix:

So I kind of want to ask the question about where this leaves us with AI critique on some level. And when I think about a lot of the critiques that people are making about artificial intelligence right now, including many that have been made on this podcast, which for the record, I am sympathetic and partial to many of the arguments that folks are making around labor, copyright, bias, climate, all of those types of things. Some of those are all aimed at the model layer. I don't know if that makes sense.

Raffi Krikorian:

It does.

Justin Hendrix:

But you're sort of saying there's this other thing exploding at the protocol layer and it feels like maybe critics of AI aren't really focused on that.

Raffi Krikorian:

Oh, that's a really good flag, Justin. I think that... and I can point you to some stats. Right now the question is not just the model, it's the harness. It's that agentic framework that goes around the model itself. An agentic framework is nothing more in some ways than a loop around the model. It's like something that asks the model a question, then decides to act on it, then ask the model another question, or might take your prompt to the harness and ask the model to decompose it and then ask what to do about every single one of those decompositions. There's a lot of evidence evidence right now that if you look at the capabilities of a model, I'll speak about specifically on coding and we can generalize from there, that the models are really good at coding and every single time we get a new model released from one of the frontier labs, they get slightly better at coding.

But things like Cursor or OpenCode or Terminus, like all these frameworks that an actual engineer might interact with the model on, it turns out when you take that model with that harness and do the exact same eval of how well did it do to write some code, just adding that agentic framework that harness around the exact same model increases the ability for that model to do about 10 to 20% better on coding tasks. So there's a lot of gain right now in just making these really good harnesses. So the frontier labs, A, kind of don't want you to know about this because they're spending a lot of money in model development and writing the framework around it can be done by smaller companies with less capital, but there's a lot of leverage that's happening right now in that layer.

And so you're right that most policy folks, most governments, most of those people who are concerned are all focused on that model side. But what we should be thinking about is the harness is what's actually going to do a lot of the governance. It's going to do a lot of the iteration. It's going to do a lot of the tool calling. So when we talk about my job is being replaced, it's because the harness is actually the one that's orchestrating the model behind the scene, but the harness is going to do a lot of that heavy lifting. It's the one that's going to know exactly how to file a legal brief if you're worried about a lawyer, or it's going to be one that's going to know exactly how to make a calendar appointment if you're worried about as a secretary about your job. So the model might have the intelligence, but the harness is what's going to do a lot of that situation. I'm not saying ignore the model and look at only the harness, but now it's this combination of the two where the battle needs to be at.

Justin Hendrix:

It strikes me the whole Anthropic-Pentagon fight was effectively a harness fight.

Raffi Krikorian:

You can make an argument that Mythos is just a harness. I don't think it is. I think that would be an unfair characterization. But if you dig deeply into the Mythos release notes, part of what they're doing is they're just allowing the model to run for a very long time. So that's a harness question of just like, we let the model run and just let it execute. And when it's finished, if we have a question, just issue another query and just let it keep on going. So a lot of Mythos gains maybe is just because of the way they're running it, not just the model itself. So that's a harness question. So I think you're right.

Justin Hendrix:

What else are policy folks missing right now? Folks who want to govern these technologies as I believe you do as well to some extent aggressively towards avoiding societal harms. What should they be thinking about?

Raffi Krikorian:

I think we need to do a more examination of what's the system we want to look like in the future and then ask the questions around that. So if we look at it through that lens, let's look at the future and backport it a bit. I want to live in a world where we have open systems and open source. So we have both open source, open weight models, which I understand has a lot of trade-offs around them. Whenever I mention this to a lawyer, they immediately come back with CSAM, but I'll also then counter with the benefits of open source and transparency and stuff like that. So clearly a pro and con and a gray zone we need to work through, but I want to live in a world of open systems. So if you think about that agentic harness question, it's not that I don't want the Anthropics or OpenAIs to exist.

I want them to exist at that extreme frontier of the things that takes a lot of capital, a lot of resources and are potentially the ones doing scientific breakthroughs, et cetera. But I want for commodity usage to go toward more of these open systems that are locally deployed that don't require a network connection that I can use even on United Wi-Fi kind of thing. So if we think about that world of this heterogeneous mix of closed source, open source, all being orchestrated and interacted with, there are a lot of policy questions around that that we need to solve and figure out. I don't even know how procurement works in that world. I don't know how responsibility works in that world. If I'm running a mixture of things in the backend, we need to disentangle which one actually caused a particular action to occur at any given moment.

I think all these type of questions become exponentially harder in this mixture world that I'm sort of proceeding in. So I don't think of any one particular thing we need to look at, but instead this crazy combination that we're about to race towards. And I don't think, just to be clear, I don't think I'm alone in thinking we're going to get into that crazy combination of the world. That world already exists and not AI. If you look at the way our computers have set up, Linux kind of runs the world and then there are a bunch of iPhones, but so that causes a bunch of complexity in the world. There's heterogeneity in just the way the web is served at any given moment. So I think it would be unfair not to say that we're going to get to the same world in AI.

So thinking about that complicated system is going to be the crux of it. And if you think about that way, then the harness layer becomes super important, tool column becomes super important, that interaction of models puppeting people becomes super important, those type of questions.

Justin Hendrix:

Let's take a little bit of a detour just to ask you about some of your views on, again, the world we're building, but maybe from a different perspective. One thing you wrote, or I guess a question you posed on your blog, Technically Optimistic, a while ago stuck with me. You wrote, "If you're being recorded every time you go outside, are you living in public or are you living under surveillance?" And if there are any close listeners of this podcast, they know that I brought this up a few times about how in my neighborhood in Brooklyn, enough people have those Ring cameras that whistle at you when you walk past and tell you you are being recorded. Let me see if I can do it. It's something like, "You are being recorded," is the little message you hear. And I must hear that when I'm walking my dog every three or four houses.

And so I don't know, I think about this all the time, I feel like because of that, this kind of issue around my passive data trail and the extent to which I'm being surveilled. And so partly I can have this conversation with you around AI and feel some optimism and enthusiasm about the types of opportunities the technology might create. And yet part of my mind always defaults back to these questions about how do we avoid this not becoming effectively a kind of authoritarian at best, totalitarian at worst kind of situation where not only always under surveillance, but it's very difficult to hide the enormous data plumes that we're all giving off as we interact with these systems every day.

Raffi Krikorian:

No, I worry about that, digital exhaust, all the time. So I did this though experiment a while ago, I wrote about it, but I thought about this a while ago. If you ask a regular person, regular people didn't even think about this, but if you ask a person who's thought about it slightly, what do you think is the most surveilled city in the world? Then some of them might say London, London famously has all these CCD cameras that the police have access to, et cetera. But if you actually do the math, it might be San Francisco because if you think about Ring doorbells, every Tesla has something like seven cameras on it. Waymos have more cameras than that on it. We have cruises, we have Ray-Bans with cameras and et cetera. San Francisco probably has more camera sensors per square block than any place in the world right now.

And what makes it even crazier is that all that data is sitting on private company servers. It's not even sitting in a governmental server. So we can argue whether that's good or bad, but I think it's crazy in some way, shape or form. So I think about this world all the time of just like are we racing towards some more digital surveillance, which is why I like to think a lot about also architecture and permission models. I am worried that the OpenAI product that Jony Ive is working on, I have no idea what it is, but might be some form of surveillance pin or something that's also recording things. I am worried that the new Apple AirPods are rumored to have IR cameras on them so they can have better environmental sensing is what they call it. So I am worried about these type of things to the point that I've jokingly a few times tried to start a company to build anti-surveillance technology of just like, what would it be to walk around with speakers that are doing ultrasonic clicks all the time so microphones get jammed?

Or what would it look like to have jewelry that had high intensity IR LEDs embedded in them just to blast away any camera that's trying to record my wife when she's wearing the jewelry kind of thing. So I do think it's something we need to be really mindful of that we don't want to get into that surveillance state. I mean, that is another one of the policy angles that we need to be carefully looking at. I don't buy the original argument that we should have no expectation for privacy in public spaces. I think that that argument was long before this explosion of computer sensing devices that we've embedded in the environment. We really need to rethink what that looks like and what my right to be not only to be forgotten, but what my right is to that real-time privacy.

I mean, I generally have to go into every single Zoom call these days assuming it's recorded and it's just a horrible way to live and I think we as humans deserve better. So I don't have an answer to you, Justin, except just to complain with you, but I agree.

Justin Hendrix:

You're reminding me of another guest we had on the podcast a few months ago, Helen Nissenbaum, who of course a great scholar on all matters privacy, but she started imagining along with her collaborators many years ago these ideas about how we potentially cloak ourselves or otherwise interfere with surveillance systems and generally try to carve back the public in a way.

Raffi Krikorian:

But I don't want to get in a world of victim blaming. I mean, I feel like I would create anti-surveillance tech for my wife because I find it amusing and hilarious and the fact that I want to rage against the machine a little bit, but we shouldn't have to. We have not as engineers been given the appropriate set of challenges in the world. So I feel like it is possible, smarter people than me could either agree or disagree, but I think it's possible to build these systems and not destroy human privacy, human agency in the process. But the thing is the people who are usually funding this work don't care or they're incentivized not to care in the case of things like Meta or others.

And so I think if engineers given the right constraints could still figure out the right systems to do. And that's one of the things that I'm working on inside Mozilla is just like, how do we introduce these constraints so that smart engineers can architect systems? Signal was able to do it. You can have two-way messaging between people that no one in the middle can snoop on. So they created an architecture where it's possible. We need to set those same type of constraints for engineers as they're thinking about their work and they get architected. They can figure it out. They're just not being asked to do it right now.

Justin Hendrix:

I'm trying to think about where to land this knowing that the hours are almost up. Is there a direction that you were hoping this conversation would go where you could finish on knowing that you're speaking to a policy curious audience?

Raffi Krikorian:

I mean, I think one of the biggest things that we need to go do both as technologists and as policymakers is that I think the lay people, and I mean that with utmost respect, just have this view of technology right now that it's either inevitable in the sense that they're just going to get whatever Apple or Google gives them or it's screwing them over in some way, shape or form and that screwing them over population is growing.

Justin Hendrix:

It is screwing them over, right? I mean, we're getting screwed over.

Raffi Krikorian:

I didn't mean to say that in the way that I thought I disagreed with them. I do agree with them. And look, I wouldn't be working at Mozilla if I didn't agree with those people, but I think what we need to go do is we need to educate more people about what these true trade-offs are on technology and we need to incentivize ways to get more alternatives out there into the world. I want more Signals, I want more Firefoxes, but I want them across more domains of the world and we need more of these credible alternatives.

So in the sense that we can use policy to fund those or we can use policy to incentivize those to exist, or we can start funneling more capital or more attention into open source alternatives that might be better for human agency or better for human privacy or better for human wellbeing, we should do that. Those are the things that we need to look at. I want to look at policy as an enabler for these credible alternatives that happen. So for anyone in the audience that wants to think about that type of stuff or wants to work with it, bring me up. That's the kind of stuff that I want. I want more than just Mozilla thinking about these problems. I want an entire vibrant tech ecosystem thinking about these problems.

Justin Hendrix:

Is that the source of your optimism ultimately that you see this as a set of decisions that people and institutions are taking and we can make different decisions?

Raffi Krikorian:

One of my experiences that I found lovely as part of doing Technically Optimistic, my podcast, was that a lot of times I would get notes from listeners who would say something like, "I never thought about it that way before and now I can't unsee it." And so if we can do that more, I believe that we as humans can then figure out the next step. And so my entire career at this point has been a little bit like, can we educate people just to see the tech world in a slightly different way that might cause them to have action later? And so yeah, I do think that's the source of my optimism, that when you show people what's truly going on, they're like, "Oh, that's not good. We should do something about it." I'm like, "Yes, great. Welcome to the club. Now I'm going to go talk to someone else."

Justin Hendrix:

Raffi Krikorian, where are the best places to follow your work these days?

Raffi Krikorian:

I have a new Substack these days called Owners, Not Renters. So if you just look for Owners, Not Renters on Substack, that's probably where you can find me best.

Justin Hendrix:

I really appreciate you talking to me about this. I hope we can pick up these topics around agents and MCP and the policy implications of those. That feels like to me a lot of new ground that we'll need to cover here.

Raffi Krikorian:

Well, Justin, such a huge fan of the work and what you're doing there. So anything I can do to help, just let me know.