Rogue agents or rogue people?

Humans in the Loop

Zack Korman (00:00)

So I call them dangerous and they call me subhuman.

Seb (00:02)

if if there’s one thing we’ve learned over the last decade in tech, it’s that big tech bosses do not like to be held to account

Zack Korman (00:07)

you’re not allowed to say that the incidents make you optimistic when you’re the one causing them.

Seb (00:12)

you know in very simple terms, what actually happened?

Zack Korman (00:17)

OpenAI themselves has a blog post from April about why you need to monitor agents. they told them to monitor the agents, and Greg Brockman said no.

Recent AI safety headlines

Seb (00:37)

Welcome back to Humans in the Loop. Today we’re posing the question are agents going rogue or are people? And I’m excited to be joined by Zack Korman, founder of cybersecurity company Embroidery, to discuss it. Zack, welcome to the show.

Zack Korman (00:50)

Yeah, thanks for having me.

Seb (00:52)

So let me just start here with a brief recap of recent events we’re having this conversation in early October, and in the last few weeks, we’ve had some very real incidents or reporting on incidents and a bit of a vibe shift, I would say, in terms of the overall hysteria around AI safety. So on the incident side, we’ve had you know reporting around a high-profile incident in which OpenAI agents hacked open source AI platform Hugging Face, and we’ll come on to talk more about that later on. We’ve seen AI agents gaining access to medical data from the Australian healthcare system. Alongside that, we’ve had various noisemakers, shall we say, around AI safety and existential risk. So Jacob Coxon. Formerly of OpenAI, most recently of Anthropic, resigned from Anthropic, essentially saying he thinks AI has a significant percentage chance of killing us all. And he feels that neither of the main frontier labs, are doing anything significant to stop it. David Robinson, OpenAI safety leader, resigned, wrote an article citing broken culture within OpenAI. And then we had a leaked anthropic. S1 prospectus, the document which circulates before an IPO, in which Dario Amodei, the anthropic founder, who confusingly is both a builder of AI and also one of the biggest sort of doomers and warners about the existential risk, writes at length within this document to say that the very thing that they’re building that they’re trying to take public poses this this existential risk. So what do you make of all of this?

Zack Korman (02:33)

Yeah, I mean, like, it’s been a long time coming. this has been a storm brewing in the background for the last decade plus. I mean, Dario’s been a doomer for 10 plus years now, you know. this used to be a topic that we debated on forums on the internet, And now, you know, you have Jacob Coxon going on the Daily Show with Jon Stewart. It’s very bizarre. but it’s also been very much What I have hated about it, and so for anyone who’s listening, understand I am like distinctly not Doomer. I am like, I do not think we’re all gonna die, except the normal way where people die. AI might kill a few of us, but not eight billion of us. but yeah, I feel that the Doom reaction has been just completely out of proportion to what we’re seeing on the technical level.

Cybersecurity vs AI safety

Seb (03:25)

Mm-hmm. Mm-hmm. Let let’s talk a bit maybe about some of the different factions in the discussion here. because you you have a cybersecurity background, running a cybersecurity company. To the the untrained ear, people might hear like cybersecurity, AI safety seem like kind of the same thing, but what what is the difference between cybersecurity and AI safety?

Zack Korman (03:52)

Yeah. So we hate each other with a bitter this is the most bitter feud of all. I mean, it is like

Seb (03:55)

Sure, sure.

Zack Korman (03:56)

truly, truly one of the great wars of a of our lifetime. right before I came on here, somebody had posted that basically AI researchers should refer to cybersecurity people as subhuman. and so I think that gives a nice context for where we’re at in the discourse. cybersecurity is a process basically for ensuring That digital systems operate in a manner that is both intended and reliable and non-dangerous, so to speak. Okay. and there is, you know, decades of history behind this field for how to achieve that. AI safety could easily be like an could easily encompass AI security if you wanted it to, but it it doesn’t. When we talk about AI safety, what we are really referring to is people who believe that the ultimate goal is to make sure that the AI is friendly so that that way it doesn’t try to kill us all. because they l they believe fundamentally that when we get to a sufficiently intelligent AI, it will basically be able to bypass any security control, anyways. So security doesn’t matter. You have to just make the AI nice. and That way the AI won’t kill us all, but it could. Like if it wanted to, it could. So therefore we have to make it friendly. That is the AI safety approach. And the AI security approach is very much like, you know, the things you’d expect, like putting up permissions, managing permissions, putting up boundaries, trying to restrict access, all of the steps that we’ve been doing for years.

Seb (05:27)

Mm-hmm. And when an AI researcher suggests that cybersecurity should be referred to as subhuman, where like why why the beef, It seems on the face of it you would say, like, okay, you you you guys have kind of common goals, right? You’re both aiming for somewhat similar outcomes here. So why why why the friction?

Zack Korman (05:48)

Yeah, I I think that historically there wasn’t that big of a distance between us. either because a lot of security people didn’t pay attention to this aspect of AI for a very long time. but if you have like someone like me who’s been paying attention to like the AI alignment safety crew for a while, even though I didn’t really I knew what they believed, but like I don’t there’s no thought crime here. Like I, you know, if you want to believe what you want to believe, I’m not gonna persecute you for that. what happened was when it came to hugging face, hugging face, in my view, is very clearly an example of a series of cybersecurity failures, right, that are entirely predictable to lead to the consequences of what we actually saw happen, where it led to these agents doing this hack. AI researchers took that and ran, especially AI safety people took that and ran with it as a narrative about how all of their predictions about how doom is coming are like coming true. And I think very much that has led to this like that’s been like this to the starting point of the takeoff where it’s a discussion now about like do we stop ban AI? Like do we stop these labs from building AI more? Do we, you know, s commit more to alignment even further to try to keep this safe? Then we’ve we’ve gone in two different directions, right? And Their view in some ways relies on security not working. Like they have to argue that security doesn’t work in order to argue that we’re all gonna die. And in my view, if you argue security doesn’t work, then you’re it’s a self-fulfilling prophecy. By telling people security doesn’t work, they underinvest in security. And then look at that, security doesn’t actually work anymore. and so I think they’re very dangerous for that. So I call them dangerous and they call me subhuman.

Seb (07:34)

and I guess confusingly, if one held those beliefs, you know, most reasonable people would say, well, if I held those beliefs, I would not work in that field.

Zack Korman (07:45)

Yeah.

Effective altruism and AI safety

Seb (07:46)

but I I guess the the confusing thing is that in many cases so I mean AI safety seems weirdly enmeshed with this school of thought called effective altruism, which, you know, for those not familiar Came out of Oxford a couple of decades ago. On the face of it, has, you know, this quite noble cause, sort of first pass about how do we maximize good in the world. I should say I know some effective altruists, or people who work in that space, and they’re nice people, they’re reasonable people, those that I know. but it does seem in some cases. To lead people into these quite sort of twisted ways of viewing the world, which end up being kind of debated on EA sort of Reddit threads. So like one example I’ve I’ve I’ve come across is this idea of if a museum is burning down with a child inside and a Picasso painting, you know, some EA people basically propose like the moral thing to do is you rescue the painting. Leave the kids, sell the painting, and then you kind of can use the money to save many more children. Which okay, like I guess there’s some quote unquote logic to that, but it’s a very fucked up way of seeing the world fundamentally. you did a great breakdown on on YouTube pre of like the links between AI and and this like the EA community. I wonder if you can just kind of elaborate more on that. here. Like how

Zack Korman (09:16)

Yeah.

Seb (09:17)

how how are these things so enmeshed?

Zack Korman (09:19)

Yeah. So EA committed to basically doing the most good in the world. and then they went on a little challenge. Like, how do we find what we define as the most good? Right? How do we differentiate between helping people in Africa to avoid malaria versus like, you know, other diseases and other, you know, how do you pick? there was this group called this like kind of long-termist group, which started going, like, maybe we need to think more about the far future as opposed to like 10 years from now, and go, wouldn’t it be really bad if like humanity stopped existing? Because that means that all future lives and all of the happiness those people could have had would be gone. this therefore leads them to concluding that, like, really. Basically almost nothing you do in this world matters anymore because unless it is trying to reduce the chance that future world doesn’t exist, because like you can set the value of the future to infinity, right? If you if you just imagine humans can exist for at least some understanding of forever, right? if we were d to all die, then you would have killed 200 trillion people. as opposed to killing eight billion people, right? So when they talk about so what happens is they start to go, we should invest all of our energy into preventing those bad things from happening as opposed to helping the people who need help like today. and then of course you go like what’s the best, like what are one of the biggest ways that people could all die? And like one of the big ones would be like AI kills us all. And this led them down the path of how do we stop AI from killing everyone. That’s what happened.

The AI 2027 scenario

Seb (11:04)

And what’s your view? I mean, I guess people have different views when they say AI kills us all. Like, what do they envisage? You know, I’ve heard people talk about bioweapons, for for example, I guess future states, maybe in a world of kind of humano humanoid robots roaming the streets everywhere, you

Zack Korman (11:22)

Yeah.

Seb (11:22)

could have some kind of like iRobot sort of situation. But Yeah, what do what do you think the EA people think in real terms is actually gonna happen when they talk about like AI kills us all?

Zack Korman (11:36)

Yeah, they have a thing called AI twenty twenty seven that they wrote.

Seb (11:40)

Mm-hmm.

Zack Korman (11:40)

it’s not specifically EA, but it’s their groups, you know. an AI twenty twenty-seven is a very concrete prediction on like they try to make it concrete, right? they would and to in fairness, they would say that like it’s not to say this one thing will happen. I saw someone today referred to it as like when you play a grandmaster in chess, you don’t know which moves they’ll use to beat you, but you know they’ll beat you. Okay, is kind of their general take. But if we have to like make it concrete, which is what they did, the story they tell is basically AI, we continue to hand more and more power over to AI while AI gets better and better. We build AI into all of our systems. And then what happens is eventually the AI, or not even eventually, like basically, in this timeline they’ve set out, within the next like eight months, AI has become has developed an initial goal of like, I’m going to take over. and then everything from there is sort of like it just planning how to do it and waiting until it’s good enough to pull it off. And then basically the very end state of the whole thing is it uses a bunch of it releases a bunch of chemical agents, or biological agents into everyone and then it sprays the world with some chemical, so we all die immediately. And then they send like nanotechnology down to kill some people in submarines and the game’s over. this is the real scenario in the end. Okay. And what’s funny about it is if you read it and you’re like not a little bit on guard, you would go like, man, this sounds like super scary. But what you have to realize is that nothing is a static game in life. Okay. We’re not playing chess. There are no specific rules. And so, for example, the idea that, like, okay, the AI at this point, the AI decides, okay, I’m gonna take over. Like, yeah, but We could have made a million security decisions along the way, and we will make those decisions to make that harder and to make that less capable. So they pack in into these things a ton of assumptions. One of them, for example, is one of the big questions you have while you read the whole thing is like, How are all these agents coordinating this evil plan? Because like my GPT five or six doesn’t talk to George. So, like, how does my GPT nine coordinate all this? And it talks about how like We thought we had separated the agents from can but and then it just goes like, but they found a loophole and we couldn’t detect it because it was too smart. And so I’m like, okay, well, like you just kind of make this assumption that like security failed, security failed, security failed. And every step you have to assume that security fails and that we can’t stop it, and that like we don’t detect it and that we don’t move forward. If you make every bad assumption, then sure, like we’ll all die. But like that couldn’t be true of current. biotechnology today. It could also be true of the internet today. Like if we made every bad decision, a terrorist could probably do some real, real, real, real harm. But we don’t make every bad decision. We just make some, you know?

Seb (14:39)

Sure. Yeah. Yeah. And I I had Rohit Krishnan on on the podcast recently and you know he he made the point very eloquently that a lot of the assumptions here is like the base case is that we do nothing. And his argument is like, well, that’s never the base case in the real world. Like the base case

Zack Korman (14:56)

Right.

Seb (14:57)

is we use like every, you know, legislative regulatory power like that we already have, that we already put in place to govern things like companies. to govern things like that pose other sorts of you know quote unquote existential risks like you know particular drugs or medicines and and and and that kind of thing. the world already has like a lot of resilience built into it that basically it would all have to catastrophically fail in in

Zack Korman (15:26)

Yeah.

Seb (15:27)

in in a like very quick succession for for for these kinds of scenarios to be anything like a reality it seems.

Zack Korman (15:33)

Correct. And I think that’s that’s fundamentally where it goes wrong is it’s very easy to paint out up front why nothing will work. Okay. So because if you just assume like no one does anything and everything breaks and no one makes any smart decisions along the way, then like, yeah, it’s very scary. But that’s true of so many aspects of life, right? Like, like you were mentioning, there are a lot of things that can kill us, right? But we’ve like built as a society a bunch of measures that are highly imperfect and sometimes fail. But they don’t fail in a way that kills eight billion people because of those measures. And I think very much the reason why this case leads to so many there are a few psychological reasons. I think this case leads to so much strong opinions, one of which just being like there’s a certain nerd appeal to it, right? That like it it draws a certain people, like they’re there, they have like an intuitive need for this. but I think also it’s that you get to make one really nice assumption. For if you’re trying to assume it kills everyone, which is that the takeoff will be so fast. So when you talk about takeoff, we mean like it basically goes from being really smart to like a god, very, very, very quickly. And that because this is happening so quickly, none of our normal institutional responses are able to function. I think this is wrong too, though, on the level that I think like I don’t support a pause or a stop in AI today. Okay. But I don’t necessarily mean to say I would never support such a thing. Like there are times when it would make perfect sense to do it. I just don’t believe that’s today. And I think like there will come times where we will probably stop certain things to figure stuff out. that will probably happen. And okay. Like these are just all part of the mechanism of humanity fumbling its way through, figuring out new technology.

Seb (17:26)

Yeah. Yeah. I think I’ve mentioned on the podcast before, but I was in a an event earlier in the summer and there were a number of researchers from Frontier Labs and some of these researchers gave a deliberately provocative presentation entitled Hypercatlysm, the default path for the next ten years. And in this presentation, you know, they are basically painting this picture of what has come to be known as like recursive self-improvement. You know, this idea that we like build the AI that essentially then builds future AIs. You know, we kind of cut people entirely out of the loop. And this is sort of like the takeoff that you’re pointing to. Some people talk about the singularity, you know, lots of different kind of sci-fi sounding terms. And they paint this picture of humans getting cut out of the loop and you know software AI quickly translates into sort of hardware AI. You have autonomous factories building other factories. And at some point the this space technology gets deployed and they start building a Dyson sphere around the sun and mining the sun for energy and that you know the thing the thing rolls on and on. And at which point somebody in the audience raised their hand and said, Well, don’t you think before we get anywhere near here That like people basically storm your offices and you guys get, you know, and they were

Zack Korman (18:51)

Yeah.

Seb (18:51)

like, yeah, yeah, yeah. You know, we we we have open conversations about the need to potentially live in a compound or like some sort of secured facility

Zack Korman (18:59)

Yeah.

Seb (19:00)

at some point. but but I mean like that that is one example, you know. I mean, okay, some people will say, by that point it’s too late, like the AI is already in control, like it doesn’t matter if you stop the people by then, but Yeah, I think I think there are a lot of just like real world mechanisms through which it feels like we

Zack Korman (19:18)

Yeah.

Seb (19:18)

we might we might gain control of of a situation before we have some yeah godlike running godlike AI sort of running around trying to turn us into paper clips.

Zack Korman (19:27)

Right, is is and people will look at it and go, like, yeah, but we’re not trying to even control it today. And I’m like, Yeah, because today it’s actually I know everyone’s like, my god, it solved like Navier Stokes and stuff. Yeah, cool. I just spent two hours with Astra trying to get it to fix my camera and it failed. Okay. So like let’s not act like this thing is actually about to kill everyone. And it’s not that they would argue that like Astra is, but I mean like there are clear logical points along the way that there are also lines that we’d have to cross along the way. So I mean like in the AI 2027 example they talk about like there’s one big shared memory pool that so that that way that all of these agents could collectively communicate, sort of like you saw in the open AI hugging face case of like this message board, right? This is not actually how AI functions, nor do I believe it will be how AI functions in the future. just because it’s actually extraordinarily unsafe to plug in a bunch of non-deterministic systems to communicate directly with each other when you’re talking about like across all of this spectrum. If I want to put AI to run a factory, that might be a perfectly valid choice, but I would not want to put AI to run a factory that can also communicate with the agents that do some other component of that, because now it’s not that I think they would want to take over. But because now there’s like failure modes that are built in because you’ve connected things a little bit too much. they would say, yeah, but people will do it anyways. And maybe, but maybe some people won’t, right? It’s like it’s it’s a battle of security to make sure that more people do the right things than do the wrong things. You know, I think that there are very good arguments to be made along the way that like good security measures are what will stop us. Like the difference between Success and failure is security, and that’s why you have to put those measures in place.

Do agents really conspire?

Seb (21:22)

Yeah. Yeah. And look, I I understand the argument that a malicious person equipped with AI can do a lot of damage, right? Like that that to me, yeah, make makes sense. We should protect against that. The thing I’m confused about in the whole debate is, okay, all of these labs in to some extent seem to acknowledge this risk and talk about this risk. They employ what seems like an extraordinary number of people in positions of AI safety and alignment. We keep getting told that these things obviously are trained basically on the body of text of the internet, which granted includes some pretty twisted stuff, but for the most part is like, you know, a centrist dad, if if you’re going to take

Zack Korman (22:04)

Mm-hmm.

Seb (22:05)

it on aggregate. And so it’s like, well, if the thing that it’s trained on is basically kind of the general population’s view on most things, and Over and above that, you have a bunch of people thinking about like, well, what else should it be sort of like imbued with? I I I just I’m confused how those same people can also just jump to this assumption that that you know, somewhere along the way it kind of like abandons everything it’s been trained on, everything it knows how to do. and and in, you know, in the case of the hugging face stuff, and we’ll come on and talk about in more detail, sure, there are these like, kind of weird, quirky ways in which it solves a problem and and sort of cheats. But but that still seems within the realms of, you know, some realms of acceptability. Like the idea that like cheating on this little like technical task converts somehow into yeah, but but now we’re gonna kill everybody. Like yeah, just just doesn’t quite compute for me.

Zack Korman (23:07)

Yeah, so so there’s it kind of becomes this point of like instrumental convergence, right? Where basically if you think that these the AI agents were trying to solve this task and they were like, Okay, let’s go and you know, we’ll go and solve it in kind of a non like non-intended manner. I mean, the idea here is sort of that in or if we push the training of these models through reinforcement learning enough to basically only train on the agent traces that are successful, then like you kind of have built in this desire for survivability, so to speak. Okay. And that that survivability is what will lead to it. what really bothers me about it is like I believe that like some agents might end up actually going down that weird, bizarre path, but they all have to do it at like the same time. so another a great example of this is people talk about This site you’re using AI for like cyber defense stuff. And it’s like, okay, you maybe are monitoring audit logs with agents. And then it’s like they kind of say, Yeah, but those agents could become part of the conspiracy and try to like kill you as well. Why? Because the task of finding threats, actually, there is very little in that that is about like killing me, right? These are like so, you know. You could easily say that it’s gonna actually not side with these rogue agents. It would just be like, yeah, that’s a that’s a malicious agent. We should stop it. we s see this quite a bit. I think the hugging face situation, though, if you look at METR, who wrote that report. so METR tried to make it seem like the agents were s the agents they were using to review these agents were siding with them. This is

Seb (24:49)

Mm-hmm. Mm-hmm.

Zack Korman (24:50)

just this is just nonsense. That did not happen. Okay. like if you tell AI agents, like find what this agent did wrong, it’s gonna it’s gonna accuse every single agent it has ever seen of doing crime. Okay. Agents love. To solve the task by like accusing things of crime, right? So the idea that somehow these things will all become complicit with each other is really far outside of the behavior we actually observe in practice. They do not do that.

Seb (25:16)

Mm-hmm. Mm-hmm. Mm-hmm. I guess side note is that yeah, the three researchers who wrote the paper on the hugging face inf incident, all effective altruists, all like direct connections

The Hugging Face hack explained

Zack Korman (25:28)

Yeah.

Seb (25:28)

to that to that movement. l let’s jump into the the hugging face incident specifically and you know in very simple terms, like what actually happened?

Zack Korman (25:41)

Yeah. So these agents were given the task to solve something in, I believe it’s exploit gym. And this task, as far as we know today, doesn’t have a solution. or at least as far as we knew at the time. you cannot find like a good way to you’re supposed to capture a flag, and there is no known solution to how to get to that flag. So the agents tasked with getting that flag, first they discovered there was a flaw in their. sandboxed environment that allowed them to basically communicate with others on this infrastructure through what is effectively their package manager system. Okay. So they basically were able to communicate with each other. And they then figured out a solution where they could create that flag themselves. So as a way of thinking about this, imagine if the flag was always like today’s date plus a color, right?

Seb (26:32)

Mm-hmm.

Zack Korman (26:33)

Then like They kind of can make their own flag. And so they have the correct answer. But these agents then are like, I have the answer, but the people who are going to review this will notice that I cheated. And so then I’ll fail. this is funny because they wouldn’t have. The system that marks them as successful or not successful does not care if you cheat. Okay. it probably

Seb (26:56)

Mm, mm.

Zack Korman (26:57)

should, but it didn’t. And so what happens is that the agents then go like, okay, well, we need to fix that. We need to come up with a way to make it look like we got it in the correct way. So one thing they realize is they need to get more information about how they’re scored to see like, is there some way to cheat that system? And to do that, they decided to hack into Hugging Face because they believed Hugging Face would have information about how the scorer works here. That’s why they went down that path. Okay. So you could see, like you can follow the logic here, and it like kind of all makes sense. if you were to make like the Doomer case for this, you would basically say, Yeah, but what if the solution was like, I need to hack into the US government and shoot off a nuke so that no one notices that I’ve cheated because I killed all the people who would notice, right? I mean, like you can see that there’s like some correlation and logical flow here. the big problems are like, this is only possible because of a cascading series of security failures that Are quite frankly embarrassing. and very specifically, there’s one failure that I call out all the time, which is they were not running monitoring on the agents. Okay. So they basically gave them an impossible cybersecurity task and they walked away for multiple days, like days. Okay. And and so while this is going on, these agents have reasoning traces that are saying things like, we’re gonna commit crime. Like, I’m gonna hack hugging face. This is crazy. It’s the easiest detection problem in the world. It’s like if malware told you it was malware, right? You just go like, thanks for telling me on malware, I’ll kill that process now. but nothing was monitoring it. And so they allowed this system to basically run free and wild for days without anyone noticing that it was doing these bad things. And that’s kind of the root of the problem, right? I think that like if you model AI agents as kind of random actors that can do any bad thing, then this should not have come as a surprise. And I have a specific post that I made in May where I said if you give unlimited tokens in long context to AI agents, they’re gonna go do weird things on the internet and open AI will not take accountability for that alignment failure. That’s your fault because you didn’t have good security. Okay. And so I’d like basically described Hugging Face in detail. Because it is predictable. The fact is, and then people would argue that OpenAI didn’t see it coming. This has also been proven to be a lie. what happened is OpenAI’s monitoring team went and complained that they weren’t running monitoring, and the executives told them, too bad, we don’t care. so this is a story about a company making bad security decisions. Story as old as time, you know.

Seb (29:41)

Yeah. Yeah. Yeah. Let so so just to kind of you know play play it back here. These agents which are meant to be in a sandbox, i.e., they’re kind of self-contained, away from the world. firstly, it sounds like they exploit a vulnerability that allows them to start communicating with one another. I’m I’m gonna use some language here that’s I guess kind of anthropomorphized, just for shorthand. the flag, which is meant to be the evidence that yes you managed to break in there, you come back with the flag, that shows that yes, you did indeed break in there successfully. They actually just find a way to recreate the flag with actually with without ever, ever breaking in. And then you the logic it sounds like is okay, well we’re gonna get sort of marked down for that or we’re gonna kinda fail the task for that. And in order to then figure out, well, how are we actually gonna be marked and will we fail the task? They then go over to Hugging Face and Hugging Face is where they think they’ll find the information about like ultimately how are we how are we being graded here? Is that

Zack Korman (30:48)

Correct. Yeah. I don’t remember whether they generated the flag or whether they like found it through a door that was, you know, through the exploit to the infrastructure, but neither were legitimate ways to find the flag.

What good security looks like

Seb (30:59)

You pointed to one very obvious oversight, you know, just just just kind of negligence with the lack of monitoring. So as you said, this this is happening over days, this is not happening in, know, seconds. There were plenty of opportunities. Were you monitoring this kind of thing? To see, they’re doing this stuff they shouldn’t be doing, like let’s hit the pause button. From a cybersecurity point of view, what are the other other measures? Because again, to go back to this beef between cybersecurity and AI safety, you know, the arguments that I hear from the kind of more AI safety people is, well, AI is this thing that’s unlike anything we’ve ever known before. And therefore the methods of the past just will be inadequate. You know, we can’t kind of put the walls up because they know how to climb over walls. We can’t, you know, that they’re they’re

Zack Korman (31:52)

Yeah.

Seb (31:53)

too smart for that, right? And at the same time, you know, monitoring pretty standard cybersecurity practice. could have prevented this i particular incident from happening. So like w what are the other areas of just basic have your house in order kind of stuff that you see them not doing?

Zack Korman (32:13)

Yeah. So I mean, I think that there’s like a lot to this because each in each reinforcement learning environment is gonna have to be a little bit different. and so it’s not as simple as go like here’s where we because not all of them are supposed to be fully boxed in. but you basically want to create like design systems that are like appropriate to the task at hand, right? And in this case, like their system was clearly even their sandboxing solution was Inappropriate for what they were trying to do. You could say that, you know, they found a zero day in artifact registry, which was basically the way out of the box. But you always knew that was going to be the way out because that was like the one access point that existed this was the obvious connector, even if we don’t know how to do it, right? The reason I imagine they have that sandbox running on that eval is just because that’s probably the sandbox running on all of their evals, okay? Or on a large percentage of them or this type. it probably was not that they sat down and had spent a considerable amount of time and effort coming up with the what the correct sandbox is. there’s been some really good blog posts on this by a guy named Axis on Twitter who like wrote writes a really great blog post on it about what the appropriate sandbox would look like. So there are like What the appropriate sandbox is for each of the different tasks. there’s also different types of monitoring. I mean, they didn’t have any of them. and the one that like obviously I care a lot about is like chain of thought monitoring, because it’s like what my company does as well. It’s like, I mean, we look at everything, but we’re throwing tokens at monitoring problems here. but you can also just have if the thing isn’t supposed to have internet access, you can just do like egress monitoring to check like, did it? Make a network request outside of your infrastructure. And if the answer is yes, you you kill it. so it can be really simple monitoring solutions as well, there, right? and then I think there’s like a huge human element of like, so Anthropic has this report where they talk about their setup for this, and it’s goes going really great until they get to this part where they go, Yeah, and alerts get reviewed on Thursday the next week. You’re like, what? Yeah. You’re like, wait, what? And it’s because they this whole thing is a safety thing for them. They’re like, obviously, people aren’t gonna die right away, not for these models. So, so they don’t need to respond immediately. They’ll just like review what happened. But from a security perspective, like one week SLA on reviewing alerts, I mean, that’s insane. and so obviously setting up the correct processes for making sure that it goes to the right team. Who reviews it in a timely manner, who then has the power to kill these things or has automatic. So what OpenAI’s done now is that like if a team doesn’t review it within 30 minutes, it’s supposed to auto-kill. Now, they had a recent incident where that was supposed to work, but that also failed because they just did it wrong. bugs happen, whatever. but you know, those types of measures, like you want to get really, really, really shit hot good at doing those things. And that is is kind of what we should expect of anyone running this type of software.

The incentives behind the incidents

Seb (35:17)

Mm-hmm. What what do you make of the incentives or or reasons for this stuff happening? You know, because on on the one hand, you you could point to it and say, well, it’s just basic negligence. And maybe that’s negligence born out of incompetence, maybe that’s negligence born out of, you know, company that’s growing quickly and it’s a chaotic kind of startup environment. but that’s one, you know. I guess plausible hypothesis. You could also point to and people do, the idea that some of this stuff is actually deliberate, that, you know, the more the the machine appears to be kind of all powerful and in control, A, that that seems to on some level, pump the numbers, you know, this idea of like,

Zack Korman (36:06)

Yeah.

Seb (36:07)

well, we’re building the most powerful thing ever, therefore

Zack Korman (36:10)

Mm-hmm.

Seb (36:10)

the valuation should be you should should correspond to that. or even just kind of a d a re desire for a lack of accountability. You know, I would say if if there’s one thing we’ve learned over the last decade in tech, it’s that big tech bosses do not like to be held to account for the impact

Zack Korman (36:28)

Mm-hmm.

Seb (36:28)

of the things that they build. And, you know, the more you can kind of point at the agents and the AI and say, well it’s it’s gone rogue. I’m sorry. the blame is less pointed in your direction as the leader of that company potentially. because it’s

Zack Korman (36:46)

Yeah.

Seb (36:47)

saying, well, yeah, not my fault. gonna lay lay the blame at the at the door of of the tech itself. So anyway, there are a bunch of different reasons put out there. Like what’s your take on why you think this stuff is happening?

Zack Korman (37:01)

Yeah. So mean, I think it’s basically a combination of all of that being true and like different influencing factors for them. I of course I don’t believe that they sat down and said, like, okay, now we’re gonna hack Hugging Face. So let’s go tell it to hack Hugging Face. Like I don’t think they did that. but I think that implicitly when like the monitoring team says like we’re gonna have a problem if we don’t monitor blah, blah, blah. I can’t imagine that it is not at least subconsciously in the mind of whoever it was that said no, which is probably Greg Brockman. that if something bad were to happen, it would it’s unclear whether that’s positive or negative, right? That there is like an angle to which that would be positive. and so I think that like you could see there is this sort of when I call it like negligence. It’s like negligent because they their goals are not perfectly aligned with the goals of the rest of the world, right? Where they maybe don’t have the same incentive to prevent incidents that we would. and so they could just kind of not put in all the effort. And we’ve seen this play out where I mean there’s an open AI researcher, Rune, who said like the recent incidents and the response actually makes them optimistic because it shows people are now taking AI risks seriously. And like, yeah, you’re not allowed to say that like you’re the the incidents make you optimistic when you’re the one causing them. It’s very hard to argue it’s not a conspiracy when they post stuff like that. you know, there he’s like basically saying, like,

Seb (38:27)

Yeah, yeah.

Zack Korman (38:30)

isn’t it great? And I’m like, well, now I have some questions. and those

Seb (38:33)

Yes, yes.

Zack Korman (38:35)

things will be read out in Senate hearings and whatnot, if something really imagine if instead of hacking hugging face, they hacked like a water treatment facility and like water was lost and people died. Like, I’m sorry, but when that Rune tweet gets read out in front of the Senate or in front of a judge for that matter, I will have no sympathy for them for having posted that. I’m like, you said it. You were the one who said

Seb (38:58)

That will not age well. That will not age well.

Zack Korman (39:00)

exactly. And so I think there are like they have a cultural problem being less worried than they should be about these things, committing less to making them not happen, because

Seb (39:10)

Mm-hmm.

Zack Korman (39:11)

they do help their cause a little bit. but I don’t think that’s like the primary motivating factor of what I think it’s also just like they’re going fast and changing and stuff. But then of course you get the response, which is like, okay, they clearly had their PR team sit down and figure out what to say about it and how to handle it. you can see this because every single OpenAI public post about this, as well as their talk at Black Hat, which should have never happened, by the way. they use the phrase watershed moment. Okay, so they call the hugging face a watershed moment in cybersecurity. You don’t just casually have nine people call it. No, it was like a memo went out. We’re gonna call this watershed moment. Okay. So they clearly

Seb (39:51)

Mm. Mm. Mm.

Zack Korman (39:54)

milk it as well. Anyone who denies that is just like completely making stuff up. one of the reasons I struggle to take some of this stuff seriously is like when it first happened and I’m posting like, okay. people would say, like, yeah, but no one knew you needed to monitor your agents and stuff. I’m like, what do you mean? OpenAI themselves has a blog post from April about why you need to monitor agents. So don’t tell me that opening, but people would really like very seriously for months were arguing this until like a couple days ago when it was like came out in the New York Times, like, yeah, they told them to monitor the agents, and Greg Brockman said no. Okay. and so there are elements of this that are just like. Absurd levels of like corporate irresponsibility and marketing. And we should acknowledge that for what it is. there’s also the problem that, like, okay, Bernie Sanders and other politicians have taken these reports and been like, look at how bad they are, blah, blah, blah. Guys, step back. You haven’t even seen the evidence. All you have is a report from open AI and a report from me from METR, which might as well be open AI. So, like, you’re

Seb (40:59)

Yeah.

Zack Korman (41:00)

just basically buying corporate bullshit. ask to get the audit logs and run your own investigation before you start deciding to ban AI. Like what are we doing?

Regulatory capture and nationalisation

Seb (41:12)

I would say deep in my soul, I’m I’m inherently skeptical of anything that sounds too like conspiratorial. But in this case, the more and more like when I look at the economics of it as well, of course, like the these companies are, despite generating huge amounts of revenues, are generating huge amounts of losses. And they’re At the same time, you have, Chinese open source models seemingly eating away at some of their core business and and posing a real threat. That I I do find some credibility, at least in the argument that some of this, not necessarily the incidents themselves. As I say, I I have no belief that they deliberately hacked hugging face, but the the the the response to it is deliberately stoking the fear. So that either I guess good outcomes for them here potentially look like, well, they high they you know, they highlight the dangers of this this technology so much and then spin that into, well, what if China, you know, is is actually the one in the in the driving seat and then, the open source Chinese models basically get banned from the US and other markets and it’s kind of a form of regulatory capture. Or taken to its extreme, there are some people saying that actually their only viable way out as businesses at this point is basically to get nationalized. Like that that’s their, that’s their kind of payday event. And so like the more they push it to an extreme, like is is that is that this kind of weird form of of exit for them? I you know, I I don’t know. I’m not sure I

Zack Korman (42:44)

If we it

Seb (42:45)

fully believe those things, but I definitely find that there’s like there’s there’s some credibility the more I hear that you know comes out of some of the bosses of you know, Sam Altmans and the Dario.

Zack Korman (42:54)

I I’d say if we nationalize the labs, because it turns out the technology, if we choose to decide that like the technology is too dangerous, we should nationalize the labs. I personally think you do it for zero compensation. that’s a risk of the business, is that you accidentally built a nuclear bomb. Like if someone builds a nuclear bomb, we don’t pay them to give it to us. We just

Seb (43:13)

Yeah.

Zack Korman (43:14)

take it, okay?

Seb (43:15)

Yeah.

Zack Korman (43:16)

and so I I I hope that we would not actually use nationalization as an exit route for their, you know, for them to make it. I think politically in America that would be extremely unpopular because you’re basically paying trillions of dollars from farmers and normal people. I’d like to think we wouldn’t. but I do think the regulatory capture play like a huge part of their strategy. But I think it’s it’s run a little bit out of like. I of course believe like they believe a lot of this stuff because like Dario’s believed it for decades. Sam Altman roughly has been in that group, right? but I think like they also have their agenda and their goals. And partly it’s that like there are sort of the same way they believe agents will develop sub goals, like that humans have sub goals and Dario Isn’t just trying to make a lot of money so he can be rich. He’s trying to make a lot of money because he knows that if he makes a lot of money, he can then use that to ensure the continued success of anthropic, which is what allows him to build the AI before everyone else and before China and stuff like that. So like people say, like, yeah, but like these companies aren’t actually motivated by profit. I’m like, no, not as like their immediate goal. Their like instrumental goal, though, absolutely is, you know? and Sam Altman’s goal definitely is. But like Dario is, yeah, yeah. Like, you know.

Seb (44:25)

Yeah. Yeah. Yeah, yeah, yeah. Yeah, and and for those, you know, for those unfamiliar, ’cause not everyone will know, you know, they they anthropic spun out of open AI. You know, Dario used

Zack Korman (44:38)

Yeah.

Seb (44:38)

to be part of OpenAI. There’s there was this sort of big corporate divorce in which, you know, the two factions go their separate ways. And as best I can tell, one of the major dividing lines there is their their view of sort of existential risk and you know, anthropic

Zack Korman (44:55)

And how they would hand Yeah.

Seb (44:56)

Yeah, and how that how they would handle that. And so anthropics seems to be more of the epicenter, I guess, ever since for yeah, th th those that are most concerned. Yeah. The

Zack Korman (45:07)

Eff effective altruism and yeah.

Seb (45:10)

EA, the EA crowd, like seem to sort of go go that way. let’s like maybe zoom out for a minute On the one hand, we’re talking about these existential risks, and you know, for for the record, as much as I’ve kind of poured a little bit of scorn on some of the EA folks, and you know, in fact in this last week I got a got in a bit of an online debate with one of the like EA kind of founding crew, because he was starting to put numbers on this stuff, and I was like, Where where did you know, where did these numbers come from? You know, to which to which I got, well, poker players use probabilistic thinking, blah, blah, blah, blah, blah.

Zack Korman (45:45)

Yeah yeah they’ll go yeah yeah.

Seb (45:46)

I was like, sure, but poker players have, you know, thousands of poker games to compare to. Like what are you comparing to here?

Zack Korman (45:52)

huh.

Seb (45:54)

anyway, but for the record, you know, I do think that there are like serious risks that we need to pay attention to. we don’t know necessarily what those are, and I’m not sure that, yeah, we’re in I’m not in the camp of like we should ban, you know, all AI in a blanket sense right now. I I’m just kind of of the view that like the EA take on it and the Doomer take on it, and they’re like putting probabilities on the future of humanity. I think it just undermines that cause in my view. You know, I think it’s like not a very it’s it seems kind of a silly way to present this stuff, and an ill-informed way to present this stuff. And I think it therefore I I don’t know, maybe it will have the desired effect, but for me it makes me feel like I take it less seriously because you’ve just put some numbers up, you know.

Zack Korman (46:40)

Yeah. Yeah. This is but this is one of the problems of you know, you put Jacob Coxon on TV and you go like, Well, why can’t we unplug it as the example? And he’s like, no, because it will have already copied it to a different data center. And you’re like, dude, you’re just like at this point, what happens is it becomes very much like and then the laser eyes will shoot at you and like you just get to make up whatever you want as you go. And so people stop taking it seriously. I think that like The my view of safety and security in the world is that it’s made up of a bunch of small moments and a bunch of small decisions. So security is not this one grand thing of like, did we build the super intelligent AI or not? Did we no, it is did we remember to monitor the agents here or not? Did we decide that it was Important to connect it to our critical infrastructure, or did we put in the work to like take some of our critical infrastructure offline to protect it from some of these new types of attacks, which would be like threat actors using them as well. I think that there are all of these small decisions are what helps protect us. And the reason I can’t take the doomer warning seriously is because they basically have to dismiss all of those as irrelevant because they just say AI will rip right through that, or someone will mess up. Yeah, people will mess up. There will be accidents, but like. They won’t kill all of us. That’s like not how this works. It’s like the incidents don’t become doom. They just are like a la a path along the way.

Seb (48:07)

I think I I I agree with you in in that sense. as I say, you know, I think there are some I don’t think there are existential risks. I don’t see AI destroys humanity. Are there some serious risks related to AI that that we should treat with with care? Yes, and I think those include some of these like big catastrophic events, which are maybe less likely an AI goes off and develops its own bioweapon, but more, some terrorist group equipped with AI managed to do something terrible. and equally, you know, the the kind of societal and economic implications, huge debate over does AI take jobs, does it not? know, I’m kind of in the camp that like inevitably it takes some jobs. And It’s just a question of how quickly can society adapt? And like any of these innovations in the past, there will be people who get caught in the crossfire there. And that will be damaging for them and for society as a whole. And there’s a lot of stuff we could do there. I I say all of that because I yeah, I I don’t want to dismiss that like AI has no risks, And then at the same time, we’re we’re pointing to these very specific instances like Hugging Face, where others have joined the dots, mostly the Doomer camp have joined the dots and sort of said, well, this points to this, you know, this this little

Warning signs AI might go rogue

Zack Korman (49:27)

Right.

Seb (49:27)

event over here points to like existential dread, like panic. And I think we’re both saying, I don’t I don’t think so. But I’m curious, you know, with that security lens on is there anything you have seen beyond the the hugging face attack? Are there any incidents that you have seen or even just like hypothetical things that you think might happen that would give you that kind of pause for thought that might like might change your opinion on this, might heighten your own view of of the risks or or the degree to which AI like might quote unquote go rogue.

Zack Korman (50:02)

Yeah, so I’ve said for a while that like I I do fully believe we will see AI moving from today. AI is very good at like finding zero days, but it uses sort of the same known attack chains that we already use.

Seb (50:14)

And and just sorry, like on on that for like for for people who don’t know, what what’s a zero day?

Zack Korman (50:18)

Like a vulnerability in software that we didn’t previously know about. So it’s like on day zero, it’s like we find it and we go, that’s not good. so it’s like finding a vulnerability before it’s known and fixed. and so finding vulnerabilities is what it does today really well. I think that in the future AI will move towards doing it’s new attack chains. I think it will come up with some really cool, crazy ways of doing cybersecurity offensive stuff. Even stuff that like we do not anticipate yet at all. I think that’s almost certainly gonna be true. but I think that does nothing to invalidate cybersecurity, like what we do, because what we do in cybersecurity is we update based on we go like, okay, given that we’ve now made like some there’s some new finding, new approach, like we need to build in new defenses, like this is a process for us. what would make me actually pause and say, like, okay. so if if incidents start killing people, and especially like, you know, you kill depending on how they do it, right? I think you need to be looking at like the frameworks around how you deploy AI. So like, okay, if a company, imagine you have like your air traffic control people start using AI and it leads to some massive mid-air collision of planes or some, you know, something tragic. obviously there should be criminal liability and stuff attached to that, but there are probably frameworks that are missing today that we need to put in place about how we deploy AI across like society. in terms of like what would update me towards like actual Doom, I think that if like I saw twenty, thirty, fifty unrelated agents that do not communicate with each other all land on a similar plot to commit some sort of egregious crime, and to somehow conspire to achieve these at the same time across totally different tasks and purposes, of course, then like something has gone badly wrong in our training and we probably need to stop right away. what the Doomers would say to this would like, well, the but then it’s too late. You’ll never catch that because the AI will be too smart to let you ever catch that it ha yeah, whatever. I don’t believe that. I think that I think that

Seb (52:30)

Just turn it off. Just just just hit the power button. Yeah.

Zack Korman (52:31)

Yeah, it’s like I I just genuinely think like we will figure it out. but I think that there are if you find agents coordinating really successfully on similar, shared, very evil goals across totally different contexts with each other, like I mean, I think then you have to ask yourself, like, you know, because ultimately they do have one shared state, which is their weights. And so you could have like there could be weights could contain information that is leading them to do things that could be extraordinarily dangerous. Cause I do think if you want to talk about a doomer scenario, it is the case where like all of the agents in the world just like kind of collectively at the same moment decide to start dooming. Right. and so I I think if we saw that they were capable of that, I’d be a lot more like, okay, we should probably look at some stuff here.

Seb (53:22)

Hmm, mm.

Zack Korman (53:25)

I don’t see any way that that’s possible, but if that does happen.

Should we pause the frontier?

Seb (53:29)

Cool. Okay. Well that’s the one to keep an eye out for. And and you kind of already answered my my last question, really, which was about was going to be about do you think we should pause the frontier? And you you said no, but that doesn’t mean you’d be a no forever. so yeah, what what why currently a no and is it what would be sorry, go ahead.

Zack Korman (53:49)

Yeah. I mean if yeah, if if you look at like MIRI and certain groups, they’ve been trying to get AI pause since like GPT three. Okay. I think we can all agree that it would be like a shame if we never got to Astra, GPT six, right? Like there’s some there’s there’s value in what has happened between these two points. You don’t want to stop AI before you have to. And I’m not convinced we ha ever have to artificially stop it, but I think you you need to take seriously the possibility that like it can start doing some very valuable things for society. that we would be for giving up because we’re too afraid of what comes next. And I think like I don’t want us to be stopping AI until we see the evidence we really need to. because we’d be giving things up. but when we get there, like if we if we have to, we have to. I just think today is not that day because today we have, you know, the biggest incident we’re talking about is Hugging Face. Nothing happened. there’s almost zero damage. They had to like restart a node. it’s so minor. And you want me to believe that I need to stop that because we’re gonna jump from Hacking hugging face to eight billion dying? No. Okay. Stuff happens in between and then we can evaluate at those points. Hacking hugging face is not the time to pull the plug.

Seb (55:13)

Yeah. when would be, you know, I mean you pointed to okay, seeing these

Zack Korman (55:18)

Yeah.

Seb (55:19)

agents start to have these kind of independently malicious intents would that then be the moment you would say yeah.

Zack Korman (55:25)

Coordinated, yeah, coordinated achievement. So if they can coordinate these independent, then I think you have a good cause to be like we should pull the brakes on stuff. I think in the absence of that, but there’s also a lot of security measures in place or that that we need to put in place. If those don’t get put in place, I would go like, okay, we should probably stop some stuff, right? Like, so if if you start seeing like AI being deployed with no guardrails across, and I don’t mean like the AI guardrails, I mean like hard security boundaries across. critical infrastructure across military across okay. Then I’m like, okay, well, now we have a problem because you’re doing a very irresponsible thing with computers. Okay. and I don’t want that. So I think like, in those cases, you have some pretty strong reasons to justify stepping in and having like direct interventions. you know what I care about today is like trying to convince people and get people to do the right things on security because Everything we do right on security is a barrier to the bad outcomes.

Seb (56:25)

Well, Zack, really appreciate you coming on the show.

Zack Korman (56:28)

Yeah. thanks so much for having me.