What AI is good at

Humans in the Loop

Seb (00:51)

Welcome back to Humans in the Loop. Today we’re going to be revisiting a simple question. What is AI good at? There’s an unbelievable amount of noise on this topic, ranging from the AGI pill tech bros talking about abundant superintelligence to the Doomers predicting end-of-world scenarios to the skeptics dismissing AI altogether. For the sake of today, I wanted to step back from the noise to look at the evidence. And to help me do that, I’m excited To be joined by Rohit Krishnan. Rohit is a polymath. He’s an engineer, economist, investor, former hedge fund guy, startup founder, and writer, all rolled into one. In his excellent substack, Strange Loop Canon, Rohit writes about AI based on deep experimentation and analysis. So, you know, he brings exactly the sort of insight I think you want to listen to on this topic. Rohit, welcome to Humans in the Loop.

Rohit Krishnan (01:43)

Thank you very much. Thanks for having me, Seb.

How to judge what AI is good at

Seb (01:45)

Let’s you know before we come on to judge good and bad in the world of AI, let’s let’s kind of start by defining how to actually judge good in this context. how do you go about that trying to decide whether or not AI is good or bad at a task?

Rohit Krishnan (02:01)

it’s extremely hard. And I think the key problem with a lot of the discourse is also the fact that like knowing what AI is good at is itself a bit of a skill. I mean, there are one way that I think about it is that anytime you’re able to make a measurable benchmark, an AI will get very good at it. It’ll get good at it much faster than you are comfortable with, and it will get good at it. In a way that exceeds human performance pretty fast. Another way to think about it is that anything that does not is or cannot be put into an easily markable exam, AIs end up not being that great at, at least for now. And that means like gaps effectively will remain. So, which is a long way of saying that the only way to know if an AI is good at something usually is to try it out on that thing. It’s knowing that pre-hoc beforehand is incredibly difficult. And like

Seb (02:59)

Mm-hmm.

Rohit Krishnan (03:00)

if if the labs could figure out a way to do it, they’d love that, but they can’t yet.

Why benchmarks only tell part of the story

Seb (03:06)

And and let’s talk about benchmarks for a minute. There are a lot of different benchmarks out there for you know different different tasks. what are the benchmarks that you find credible, shall we say?

Rohit Krishnan (03:19)

I mean, i this is a hard question because like I don’t think any individual benchmark is all that credible. like if you do well in one set of benchmarks, the chances that you will do well in another set of benchmarks is very high. There is high degrees of correlation. So as a consequence, an individual benchmark doing pretty well doesn’t really kind of mean all that much per se. I mean, I’ve built benchmarks, I’ve seen benchmarks. I mean, some of them are really interesting in in different ways. The way I look at it is like ability to kind of solve hard problems that I encounter on a daily basis in my work. And not hard like solving a Millennium Prize hard, but hard just like normal work hard. And some of those are things that the models are still really bad at. So the way I look at it is very much like For new models that come up, subject to how good they are and subject to how much bandwidth I have, can I do a vibe check on them is sort of the the default method that I kind of use. And if as soon as one gets saturated, usually you go to the top tier of the models because they’re usually better, faster, and cheaper. And then you can use those in order to kind of push whatever piece of work you want forward.

Seb (04:34)

Mm-hmm. Mm-hmm. Okay. So there’s that like, yeah, we’ll we’ll come on definitely and talk more about some of your your kind of hands-on experimentation. But I guess what we’re saying is, yeah, there are these benchmarks, they have some use where you wherever you have a a benchmark, the models can be very clearly trained, you know, towards that benchmark. I guess there

Rohit Krishnan (04:56)

They get better.

Seb (04:56)

are a whole bunch of things that just exist in this far more like subjective space that it’s

Rohit Krishnan (05:02)

Right.

Seb (05:03)

just much harder to kind of train a model for or to just determine whether or not the model was actually any good.

Rohit Krishnan (05:10)

Yeah, correct. I think the I think like normal one weird thing that we have learnt over the last few years is that like human work is very hard to put in a box. Even the like seemingly most normal, mundane pieces of work are really hard to constrain into a box such that you can provide it to like like an AI to do. So one way to think about the benchmarks and stuff is that like what is it actually measuring and what is it going to be really good at? Even something like coding, where these models are undeniably superhuman in many, many, many ways. Like the hardest part of coding, as anybody who has coded or like has lead engineers knows, is kind of telling them what to do or like figuring out what to do. Once you figure out what to do, the implementation is usually not the hardest part of the problem, right? Because like you have false starts and you come back and whatnot. And the models are getting better at judgment, but they do not have good judgment with this at all. Sometimes they’re too timid, sometimes they’re too narrow. They kind of get hyperfixated or focused on some ideas that kind of come their way. They do not explore the white space well enough. All things that presumably with enough scale or with enough ability can get solved. But like to me the question of how do you know if a model can do something question, the answer almost always is to try it out. And then you will see like the fifty ways in which it falls short, as opposed to like the one big way in which it falls short.

Where AI is already really good

Seb (06:43)

Hmm. So, I mean, you you you touched on engineering there, and and I think that is one of those areas where, you know, undeniably, client of mine, CTO a startup, they have, a bunch of historical technical debts. They’ve talked about for for years now this at migration, gonna be months of work, kind of, you know, grunt work, frankly. Nobody’s looking forward to it. And basically the CTO goes away last couple of weeks, shuts himself in a room with Claude Code for two days, and and and essentially does it, you know, like takes this whole thing that would have been months. And yeah, anyone close to engineering teams, product development teams is seeing these kinds of examples, which are like kind of mind-blowing and and and quite transformational. Beyond coding or or building on that idea of of the example I gave there, like What do you see in your world as being the core superpowers where AI in its current form is already just really, really good?

Rohit Krishnan (07:51)

I mean, it changes sort of month to month, but like I mean coding is obviously great. The latest generation of personal apps, personal intelligences like Muse Instinct, these things, they’re remarkably good. I remember running things via open claw before and it was like a pain in the ass. But now like things work remarkably well. So like a large chunk of my personal admin kind of gets done by them. They’re very good at computing.

Personal AI agents in practice

Seb (08:18)

Tell me to sorry to interrupt there, Rohit. Like tell me how you’re using that, right? Because I’ve you know, there there are lots of podcast appearances or whatever in which you see someone from Silicon Valley like gleefully describe, here are my seven different agents and they’re all doing this kind of stuff and and I I I I find I find it

Rohit Krishnan (08:35)

So I think I’ll

Seb (08:36)

hard I find it hard sometimes to know how much of this is sort of like real versus performative. So yeah, tell tell me like in your kind of personal admin, individual admin, like what what are you using, what’s working for you,

Rohit Krishnan (08:49)

Yeah.

Seb (08:49)

what’s it actually doing?

Rohit Krishnan (08:50)

So yeah, I mean a lot of that is kind of fake or like done with no real benefit to you at the end of the day. for me, like I use Instinct and I use Muse. I’m trying both out because like why not? they do roughly the same stuff. Muse is by Meta. They’re out of their new superintelligence labs at the moment. Instinct is a startup that kind of try to do it. Both of them are available on I use them on WhatsApp. I think they’re also available on iMessage and Muse itself has an app as well. they’re connected to my like, I don’t know, whatever I could find, like in my emails, my calendars, you know, various other logins. Instinct even has credit card that I have given it such that it can go and kind of do purchases and whatever on my behalf. So some examples of things that it has done for me recently is like pulling together documents from my email for some investment stuff or like for tax stuff. And these are all things that exist in Gmail somewhere. You just need to kind of go off and do it. Can Codex do the same thing if I’m sitting on my laptop? 100%. And can can I get into my laptop if I’m outside using Codex remote and ask it? Also true. But in this case, it’s just easier, right? Because it’s on my WhatsApp. So I can just send it a message and forget about it. it’s also proactive. Like I’m supposed to take a trip, I think it was next week or the week after. there’s information that came into my Gmail and it already kind of looked through and figured out like, yeah, you should take this flight from here to Boston and like this flight back from New York or whatever. Like there’s a set of at like pure administrative work that it just takes off. Similarly, it keeps me on top of, you know, hey, here are a bunch of things that you didn’t reply to or things like that. Like effectively, these are not individually, none of these are, you know, mind blowing. These are not solving Navier Stokes kind of problems, but each one is proactive and helps me make my life just sort of a little bit easier, right? In terms of figuring out what you need to go or where you need to do. I mean, I like I use this for everything. Like for example, recently I was in San Francisco and I like I had misjudged the time or something. I forget exactly, but like I I needed to quickly take a phone call. And I needed a place to take a zoom and I was sort of saying, like, hey, I am at embarcadero and whatever. What’s the what’s the nearest place? And it’s not something that requires huge amount of intelligence. It does need things to be snappy and give me an immediate answer, which it did. So it’s like everything from the like the extreme mundane to things that can take care of my life at a higher level. I try and push as much of that through these tools as possible. So that like collectively I might gain, I don’t know, an hour a day, something. But that’s a

Seb (11:29)

Mm-hmm.

Rohit Krishnan (11:29)

that’s a lot.

Seb (11:30)

Yeah, yeah. Not not bad already, huh? And and you know, I guess the examples you gave, yeah, it’s the kind of stuff that I don’t imagine anybody does joyfully or or or maybe they do, but I certainly don’t. you know, it’s kind

Rohit Krishnan (11:44)

Not me.

Seb (11:45)

of it’s kind of grunt work that you wanna you wanna get out of the way. feels like a prime candidate.

Rohit Krishnan (11:48)

Yeah, I mean none of these are things that I mean, these are things that I would imagine like I could easily hand off to an EA and have them just kind of solve it for me. Do you know what I mean? Like

Seb (11:58)

Mm-hmm.

Rohit Krishnan (11:59)

as opposed to something that requires enormous amounts of context and focus in order to do this. It does that as well. Like it negotiated my Xfinity internet bill down. Like,

Seb (12:11)

Mm, mm.

Rohit Krishnan (12:13)

I don’t know, a little while ago. It drafted emails and sent it to like a bunch of things, including some tax information stuff. So like these are all things that individually none of them are so hard that you can’t do it. But collectively, these are each one makes your life that much easier because I see something, I can send it a note, it kind of does, turns away on the background, asks me for input when it needs it, and kind of goes off and does it. Like I don’t need to sort of even think about it again.

Can AI write well?

Seb (12:41)

Mm-hmm. Mm-hmm. Tell me about writing, because obviously you write a a great Substack. Yeah, w what’s your what’s your take on does AI write well in in your view?

Rohit Krishnan (12:55)

No.

Seb (12:55)

And does it exist anywhere within your writing process?

Rohit Krishnan (13:02)

AI does not write well. I I had this in 2024 onwards, I think every essay that I have written has one sentence that is AI. I tried to start doing it at some point to see if I could sneak it in. just for fun. so it does. Every everything has. Even when I look back, I can’t tell which one it is, quite frankly. It’s so AI is great at writing at sentence level already, occasionally. It’s not all sentences, blah, blah, blah. But what it’s really terrible at is actually putting forth an argument. It’s really, really, really bad. Arguably it has gotten worse. Because, you know, I far prefer the older models which had errors but like better writing than the modern models, most of which are just like indecipherable corporate gibberish. When they write. I really dislike it. In terms of sort of the writing process, it actually exists almost everywhere. Like the I mean, from ideas. Like sometimes I have ideas when I’m going around and I ask the AI to kind of go do some research so I can read it. Sometimes it’s, you know, yeah. Like it it literally exists all over the shop, is I guess the way I can actually say it.

Seb (14:25)

Yeah, yeah. Yeah, I mean I’m conscious as we’re having this conversation that, know, fast forward however far into the future, it you know, it’ll it’ll kind of be like saying, Did you use electricity in this process? you know, and it’s like, Well, yeah, of course, you know, everything it was all over the place.

Rohit Krishnan (14:39)

Exactly. Like mean I mean the people sometimes ask me like where do I use it kind of question and I don’t I I find it hard to answer that because like I sort of use it everywhere in the sense

Seb (14:49)

Hmm. Yeah.

Rohit Krishnan (14:51)

that, I don’t think of it as a separate action that I do necessarily. When I have a question, I use AI to kind of get multiple points of view or thoughts on it. Like there’s a few articles or essay ideas that I’ve had that I wanted to do, and some of them I wrote Like half a page, a page, and then I’ll be like, I would like to know something specific here and I can kind of use AI for it. Some of them I’m like, I have this general idea. I don’t know if I’m completely wrong or misremembering a bunch of things. Can you go do a bunch of research and come back? And quite often it’s just like, hey, here’s something I’m interested in. Can you go like, you know, teach me something about this world or like help explain something about this world? And then I can use that. Like so that it kind of it is not a tool to be used. specifically when you need to use a tool is just sort of suffused into every part of work that you’re kind of doing. What I do not do is to like I mean I’ve tried to get it to write and then me rewrite. It does not work very well because it’s really bad at it and it annoys me more than like like AI messes up both sentence level sort of constructions. It messes up the cadence in the sense that like I feel like I don’t want to read it. And at which point what’s the point of writing it? Like There is a set of problems like this that just don’t go away.

Seb (16:07)

Yeah, I find it it it it’s interesting because it feel it feels very much like AI as a writer has this real like house style. But I I find it quite hard to like define. You know, it’s like that senator who once said, you know, pornography, like the hard to define, you know it when

Rohit Krishnan (16:20)

Yes, you know it when you see it.

Seb (16:21)

you see it. You know, it’s kind of the same for AI writing. It’s like I I I can’t necessarily point to like any one thing and put a like clear, because of this, I think it’s it’s like an innate sense of of I’m reading this and there’s just something that doesn’t quite sit right.

Rohit Krishnan (16:38)

I mean there are many, many, many tells. I I I keep writing longer skills to get AI to write slightly better, even when it’s writing like PRs and stuff. And it’s not it doesn’t work. Like at some point you just have to get used to the fact that like certain documents you create will be, you know, AI slop and that’s fine, and you just live with it, and certain things might not be, and then you kinda spend more effort on it. like You know, it loves using extra adjectives for no reason. It does it does flowery descriptions all the time. It like this constant contrasting of things is bad. Every sentence feels like a punchline and it doesn’t need to be. It loves like sharp metaphors, which sort of you can almost see that like this metaphor would spike up the attention in the in the matrix, but like but it doesn’t actually help the user understand the problem better. It is remarkably bad at theory of mind. It does not weirdly enough, it does not construct arguments well or logically. Like when humans construct an argument, good arguments, like you kind of create a base and then you layer on top and then you layer on top and you guide people through to a conclusion. It’s really bad at doing that. I don’t know why, but like that would have that’s one of those things where I would have expected the AI to be quite good at it, because it’s not it’s logic.

Seb (17:54)

Mm. Mm.

Rohit Krishnan (17:55)

It’s not even writing, but it’s really bad. It throws the wrong levels of abstraction in the middle so that like you kind of keep losing your chain of thought. It is very hard for it to go from like abstract to concrete in a smooth fashion. It’s very jarring. It’s it’s a such a long list. But you know, long story short, it’s really bad at doing the intellectual heavy lifting that you might want it to do. To be fair, I think it’s bad across the board. It’s not just in writing. I think it’s if you want to do AI to do research. pretty bad at that as well, right? I mean it does not have good taste. And we

Seb (18:30)

Mm. Yeah.

Rohit Krishnan (18:31)

kind of encapsulate a lot of information in that word taste. But in in order to get the AI to be better at it does not have taste today. Let me just put it that way. Maybe Noam Brown is right and like we’ll get AI with great taste soon, in which case all of this kind of goes away. But I’m semi suspicious seeing the scaling so far.

What an LLM is under the hood

Seb (18:50)

Yeah, let’s talk a little bit about that because obviously you have an engineering background, you’ve been very hands-on with this stuff. So so I guess I guess if we look at like what an LLM is actually, like under under the hood

Rohit Krishnan (19:00)

Mm-hmm.

Seb (19:03)

and what is it about what an LLM is that translates into the things it’s good at and the things it’s bad at?

Rohit Krishnan (19:12)

So good question. I I think I mean I I’ve tried it a couple of times to kind of answer it and I have various forms of answers to this question. However, I feel like I don’t think this question can be answered, is kind of where I’ve come to at the end. Because like theoretically, a model that is able to recursively loop can be however slow be quite good at solving things like cellular automata, or can it be a Turing kind of machine, right? So as a consequence, is it possible for it to theoretically do anything? Yes. The problem is that practically, even though like I sort of a friend a couple of years ago on Twitter, I think, proved that Opus could act as a Turing machine. you can fact check me on that, but like I think it was two years ago. This and this was your, you know, there was this big debate about like can they, you know, think deep enough in some sense. And the answer is like, look, if the if every Output is going back as input. And if you go for long enough, then surely you can do any program that can be created. And once you say that like that is true, then it becomes really hard to say like there are these things that LLMs just will never be able to do. So

Seb (20:30)

Mm.

Rohit Krishnan (20:31)

instead, I think what I’ve kind of come down to more is that like there are certain things that in practical limits it just seems bad at learning. So one my favorite sort of one of my favorite ways to think about this is that like any Pattern that actually exists in data, the the models are remarkably good at learning. Right? And any pattern that exists in data about learning patterns, it is remarkably good at learning those as well. These might not just be the patterns that you see as obvious, these might be all sorts of random patterns that actually exist in the data. So, in some ways, like what you’re trying to do is to sort of There is a space of patterns of information that exists in the data. You’re trying to cull it from different directions such that only the true ones remain. And this lossy process, which is why you know they still go off on random tangents as opposed to sort of coming out very cleanly, not because the process is lossy, but not to mention the fact that, like, this is a really hard way to kind of do this in the first place. Like human

Seb (21:37)

Mm-hmm.

Rohit Krishnan (21:37)

beings, we get rewards from the environment around us, and the LLMs don’t really get rewards from the environment. They get rewards from their training environment. And some of

Seb (21:46)

Hm.

Rohit Krishnan (21:46)

the latest issues that we have seen are precisely because they cannot reliably distinguish between the training environments and the real world. Because for them, like the distinction is fairly paper-thin. So, like in a weird way, like I think that could an LLM be taught anything if sufficient data existed about the patterns that existed there? Yes. But that doesn’t mean that like you can practically Teach anything using the same architecture, if that makes sense. It’s also,

Seb (22:15)

Mm-hmm. Yeah.

Rohit Krishnan (22:17)

I should note that the the traditional architecture that like the original Llamas were built from or like whatever they’ve all kind of been superseded to a large extent, right? Like every piece of that architecture from the how do you do cache to like how do you do reasoning and how do you how do tokens actually flow through? Every part of that has shifted over the years and currently nobody really knows what Claude and OpenAI are actually doing internally. So it’s a little bit like shooting in the dark to figure out like, is it an architectural problem or not?

Data, patterns and the limits of training

Seb (22:47)

Mm-hmm. Mm-hmm. So yeah, I guess we run into this hypothetical, which is kind of saying, okay, we if we if we have the world’s most powerful pattern matching machine,

Rohit Krishnan (23:02)

Mm-hmm.

Seb (23:03)

think about it in those terms for a minute, and we give it enough data of any kind, well, what it’s gonna do is it’s gonna find the patterns, some of them maybe obvious, some of them non-obvious. And it’s gonna kind of train on that basis and become good at those things. and and I guess like that data is itself doesn’t necessarily have limitations, or it’s hard to think of the limitations. You know, take a conversation between two people, you know, we can give it the data of like the pure text and kind of say, right,

Rohit Krishnan (23:36)

Mm-hmm.

Seb (23:37)

well, here’s here’s the exchange between Seb and Rohit, or we can give it an audio file, which already is like. richer kind of context. But but like we could keep going and going and going, right? When we think about like emotional intelligence and like what’s going on with my nervous system and Rohit’s nervous system and body cues. And like you know, we we we could probably like generate the world’s data just on the interaction between the two of us right now if you just get more and more and more and more granular. So I I I I guess fundamentally this feels like the argument that I hear the the the the kind of yeah the the techno optimist argument, which is basically if you can get the data, you can train it to do that thing. but there there just seem to be some incredibly incredible like impracticalities to that.

Rohit Krishnan (24:30)

I believe the fact that if you can reliably get the data, you can train it. I I think that is true. The questions are more like how mu how much can you do? Like there’s a set of practical questions that there come up, not to mention the question of how large a model is it feasible to build? Like we’re talking about ten trillion parameter models. Can we build a, you know, hundred trillion parameter model? I don’t know, like maybe we’ll have to do a whole bunch of changes before we are able to get there. But like, you know, logically, it is feasible, practically infeasible. So you gotta kind of figure out sort of which one of these steps you’re willing to get off the argument at. Similarly, like you mentioned about audio video, etc. Like there’s a large amount of bits that are being transmitted through our words, through our actions, body, audio, video, like each one increases the level of information that is kind of flowing through. However, There’s a set of information that none of these things are capturing, right? Like the neurochemical pathways that are going through my head as well as your head is not something that is being captured in any of these conversations. The outputs of those things are, and from the outputs of those things, we might even be able to train the model suffici with sufficient data to glean some parts of the internal activations that are happening. It’s never going to be granular enough that like you can understand the neural pathways. Because there’s just the brain is a complex adaptive system as well. So it’s not exactly like the outputs. Are directly linkable to a specific causal set of inputs such that you can actually train them in order to predict specific things. so in some ways, like the techno optimist argument is correct insofar as like if you are able to capture more data, you would be able to train models in order to do hard things. I mean, we have seen like Astra being able to do robotics tasks as one example of this kind of generalization, because like ultimately, like If ask Astra like what is this? And it says a mobile phone, like it understands what it is. It understands the movement by which I created the video models are able to distinguish it. At which point, like how hard is it for it to also learn, like, how do I move an arm in order to put it down or bring it up? It’s not that hard, right? I mean, because that is implicitly part of the training data. But that is that does not also mean that like in the limit. you know, you’re gonna be able to learn from anything to get anything. Cause like ultimately all of these labs also are beneficiaries of spending tens of billions of dollars on specific data to get them better at tasks. Like I feel like they’re not it’s not like every three months we get a new model and the new model is just better at everything because of X. Like I mean it’s better because A they have an enormous data advantage, the large labs do. And like they c buy it They use their own. So in fact, I would be shocked. I use Codex as my daily driver, but like I would be shocked if like every three months all my tasks did not get easier in Codex because like OpenAI gets my data. So like they’re training on it to make the next model better. So why would it like not getting better doesn’t make sense? Cause like that is what that is a core thing that you’re actually working on. So effectively

Seb (27:38)

Mm-hmm.

Rohit Krishnan (27:39)

my data is directly contributing to this feedback loop of getting the models better tomorrow. But that doesn’t work for everything, right? I mean, we’ve been asking chat GPT questions for four years now and like getting it to write stuff for four years and it’s doesn’t get better there. So the things that are can be measured, that can be evaluated get better. Things that are harder to measure or evaluate don’t get better. At at least a first order of approximation.

Finding your place in a changing world

Seb (28:03)

Yeah, yeah. you said yourself, right, that that it’s hard to say, okay, here’s something that LLMs will never be good at, because you know, we we we kind of just laid out the the argument for that. I guess when people the the subtext of that question for most people is one of kind of fear of like, well what the fuck is my place in the world? if like AI just like does everything, you know, what job can I do ten years from now? How am I gonna make a living? There’s all all the all these kinds of like fear, fear narratives. So I I guess I’m just interested like where where where you you stand on that. You know, we we can’t predict, we can’t kind of purely rule out, okay, it will never be good at this, it’ll never be good at that. yeah, how do you resolve that in your own life?

Rohit Krishnan (28:56)

I mean It’s a deep question in some ways, cause like the answer is you kind of muddle through the uncertainty, you know? Like I mean I think about life, I don’t know, twenty years ago. And twenty years ago, it’s hard to kind of imagine but twenty years ago life was quite different in some ways, right? Like if it if you just plot 2006 and now, the life expectancy has changed, the level of wealth in the world has changed, the number of customer consumer conveniences have changed. if you net all of those out. We are effectively living in a completely different world. jobs have fundamentally changed. Like, you know, even if even jobs with the exact same title, investment banking analyst or like consultant or whatever, do not do the same job today as they did in 2006. They might have some of the same tasks, but like the actual jobs have differed dramatically. So, not all jobs, obviously, but like many of them have, right? Some for good, like these ones I talked about, some not so good. Like I mean, you know, doctors have to type in enormous amounts of notes these days and everything. And it’s like a it’s it’s every doctor kind of cribbs about it because it’s an extra added workload that doesn’t help them. Is that net beneficial? TBD, we’ll find out, right? Like when the note quality and and like you’re trading off note quality for coverage and at some point hoping that the coverage makes up for the deficiency in note quality. At some point in the future, but like maybe it does. So I’m not I’m relatively sanguine about the fact that like things will change dramatically. I’m even sanguine about the fact that things will change dramatically pretty fast. I mean, I I, you know, 2006 was a long time ago, but like 2020 was not that long ago. And I remember when COVID shut everything down for a year, and like we adapt pretty fast, pretty well, all things considered, you know. All whining aside, it kind of worked Pretty well, I would say. Like, you know, could it have gone better? Of course. But like we shut the entire economy of the world down for a few months, figured out which ones were essential, and just provided that. And then just from that, we were able to do like we were able to reboot the world economy in some ways. Which is a crazy achievement if you think about it. So like my my my core

Seb (31:26)

Well, and then and in the meantime, you know, came up with the like fastest vaccines to market that kind of immunized, you know, or or close to immunize

Rohit Krishnan (31:32)

Yeah. That’s what I’m saying. It’s like

Seb (31:35)

l most of the world.

Rohit Krishnan (31:36)

so in some ways I feel like this this hypothesis that we are unable to react fast enough when something bad happens or something good happens just feels like folly. Because that’s not true. We’ve done it. We’ve done it. We just did it. So like

Seb (31:49)

yeah.

Rohit Krishnan (31:49)

in s you know, in in many ways, my worldview therefore is very much of like, sure, you know, being situationally aware is a good thing because you should know what’s going on. And at that point you will figure out when you need to intervene or be able to add something to the discussion or like change the direction in some ways. But the ability of the society to actually react is immensely high. Like I feel like there was a time when that was maybe not the case, but that’s not the world we live in. If anything, we live in a world where arguably it is too fast and too strong in how it is able to react. And so if and like our job is to kind of be watchful and not do crazy things super fast just because we feel like we need to do something to kind of stay in control. ’Cause we’re not. We’ve never been in control, right? Like it’s it’s a why it’s a big world out there. I don’t know when the last time we were in control. Maybe at the time of the Babylonians, I don’t know.

Seb (32:47)

Yeah, I’m not sure we we ever have been. and yeah, I mean I I I think I I probably view it in a fairly similar way. Look, I think there is a dimension to this moment we’re in now where it feels like the pace of change for people, like the the layers of of complexity, there’s obviously the kind of the media exposure that people get to this stuff, right? We lived in historical times where lots of change was going on in the world, but you were kind of largely oblivious to most of it. Now, you know, a lot of the population, probably on a on a pretty frequent basis, are seeing you know, alarming looking headlines. So so I I I think there are there are kind of dimensions that make this moment different or more intense to to past moments. But I think I I also agree that You know, there th there is no such thing as an AI-shaped hole in the world. Like nobody in any business, in any like practical application has gone like what we need is what people are looking for is like to solve real problems and so you know, y your ability to spot problems where they exist in the real world, I guess use the latest and greatest tools. to whatever they may be to to solve those problems. yeah, f to me that feels like an enduring skill, enduring capacity.

Plans, accountability and liability

Rohit Krishnan (34:09)

Yeah, I mean I I agree. It’s just I

Seb (34:13)

Yeah.

Rohit Krishnan (34:13)

feel like, you know, there is this worry that I hear people have regularly of like there is no plan in some ways.

Seb (34:23)

Yeah. Yeah.

Rohit Krishnan (34:25)

And the question or the worry itself presumes the existence of a plan will make things better. And I’m not entirely sure that’s true, right? I mean many things need to be planned for them to go well. You know. wars companies sometimes. Like there are things, you know, vacations sometimes, but like many things are also really, really hard to plan for. The economy, very famously. We’ve had enough experiences from like, you know, Cybersyn to the five year plans. Like we we’ve kind of seen send the failures of central planning multiple times. So the question for this to me is like Look, it’s an extremely complicated issue that touches a wide swath of the real world. And we need to figure out how to get more good things and less bad things. And like, are there specific things that we can do to intervene that doesn’t necessarily change the course of everything that we need to do? And it’s really hard to identify some of those things and also especially to be okay with inaction at different points in time. Because

Seb (35:29)

Yeah. Yeah.

Rohit Krishnan (35:32)

what to do. Is an e extremely hard question, but as is the question of like should you even do anything? Because most of the times I’m not sure the answer to that is like yes, we absolutely need to jump. Like right now, there is we are going through a particular issue when it has come up in the zeitgeist, it is front and center of everyone’s mind. People want to kind of spend large amounts of time, attention, money, force, whatever to kind of push their specific legislative agendas forward. And like most of them, when I look at it, are not specific at all. Or if they are specific, they’re specific in service of a worldview that is not proven. So I kind of look at this and go, like, guys, I mean, we are putting the cart way before the horse here. We haven’t even seen a problem, but we are trying to legislate it away. And that has never worked in the history of humanity. So

Seb (36:24)

Yeah, I think I I think I largely agree with you in terms of or or certainly with aspects of it. I I think the issue it feels to me like there’s fundamentally an issue of accountability in terms of you don’t need to, I don’t think, have planned for all these future scenarios that, you know, nobody can possibly predict. Probably least of all like the legislators who who are who are rarely, if ever, you know, the the the ones really shaping the frontier or the best understanding of like what’s truly happening. at the same time, giving Frontier Labs, a bunch of you know, frankly, like pretty wealthy, powerful people out of mostly Silicon Valley, a bit of a hall pass to say, kind of have at it, go do your thing. You’re gonna make a lot of money and accumulate a lot of power in the meantime. and by the way, we’ll we’ll we’ll kind of clean up your mess after. Also feels flawed to me. it feels like you just need to have some mechanisms of accountability, right? It’s like, okay, what happens if like your company’s AI agent goes and hacks another company? Well, if that if that was a person, you know, like that that would be a legal matter. And like I think we should view it through a a similar frame. Of of course we can’t put the AI agent in jail, but like there are other legal mechanisms we have to say, hey, your company like needs to get its shit together and not hack my company. And I don’t care if it’s an AI agent who did it or a person who did it. Like it needs

Rohit Krishnan (37:55)

Right.

Seb (37:55)

to not happen. You know, and similarly, I think we need to think a little bit about tax, for example, right? If you employ somebody as a company right now, you get taxed on that employee. And that kind of creates certain incentives and it generates certain money. I don’t know what the right answer is, but like, There needs in my view, there needs to be a question asked of like, well, where do these agents sit in a in a tax context? You know, should they be considered in in a similar bucket? If not, like what incentives does that create? So yeah, I guess my personal view is like I I I agree. I don’t think we can kind of plan for all the future scenarios. It it does feel like maybe we just need to get better at leveraging these existing mechanisms, creating kind of clearer standards of accountability that kind of says Hey guys, if you fuck up the world, right, you you you you you’re gonna have some some consequences.

Rohit Krishnan (38:49)

I mean it’s a We forget that like over the last several hundred years, if not thousands of years, we have worked really hard at figuring out the right structures by which we can deal with unreliable counterparties who do bad things and kind of still have the global economy run well as a consequence of this. Like this is not a an just because it happens in silico does not mean that like we do not know how to deal with it. So to me, like if you’re During training, if your models get out and hack someone else, you’re liable. If while acting, if I specifically instruct a model to go hack you, then I am liable. And somewhere in the middle exists the large gray area where like at some point the liability shifts from the person who created the model to me. And some of those things come because like the model maker has to show that like they’ve taken due care in in being able to kind of making sure that I cannot hack, for example. Some of those things might not even apply. Like if I’m providing an open source software and like you use it to kind of hack me, am I liable? Usually not. But what if the piece of software that I created is something whose sole purpose is to hack someone that it has no other legitimate purpose? Then yes. So like these are just sort of this is the tip of the iceberg of like an enormous mountain of questions that we have struggled with over the years of trying to figure out where these kind of boundaries might lie. Some of this is unclear with agents, of course. And but like it is crazy that like how much of it is actually clear. Like in the sense that, you know, this isn’t all of these are not like just all of this is not fake, right? I mean, we kind of know the answer to a bunch of these kinds of questions. So if you are open AI and your agent goes and hacks someone, then you’re liable seems like a relatively straightforward application of the law because it’s Ultimately, you, the company, are the are the people responsible for this happening. So, like you’re the ones who should stand behind it and solve it. If you don’t think you can make models that does not hack anyone, well, I mean, like, that’s sort of a question that answers itself in my case. So, product liability and safety, generally speaking, have been helpful for us to figure out what are the things that should be built and what are the things that, as society, we are okay with being built. We do not allow car makers to build cars without steering wheels or like without seat belts, for example. But we do allow car makers to make extremely unsafe death rockets that you can drive around in a racetrack. Different uses, right? And like, are you allowed to take that car and drive it out into the real world? No, you’re not. But like nobody’s physically stopping you. You can buy one and take it out. You’ll just get fined for it. Or like, so there are There’s a large set of mechanisms that exist in the society across all levels, from individual action all the way to corporate action, that constrains the types of things that are available and possible. Now, is it possible that there are new classes of actions that come up that are not covered by any of these things that we need to kind of sufficiently cover well enough that we need to put it out there? 100%. I believe that. Yes. So an example might be, you know, OpenAI. Puts a model out there, that model for whatever reason creates a new botnet that seems to sit somewhere. That botnet two years later ends up getting activated and goes and hacks someone. Should OpenAI two years ago be liable? New question, I don’t think it’s a dissimilar question to what exists today. And the same jurisprudence that we go through today, the same legislative manner, will look at that and kind of assign some degree of blame to OpenAI or to Anthropic who actually made that happen. And at the same time, they will assign some degree of blame to everybody else in terms of sort of their respective levels of negligence, in terms of making this actually happen. I I feel like a lot of these questions kind of start with the assumption that if we do nothing, the end of the world is a result, but not think about the fact that doing nothing is never the base case. The base case is that we deal with it the exact same way that we have dealt with everything else that has happened so far. And like we react, the society reacts, the The laws react, the insurance companies react so that like we reach a new equilibrium where we are able to sort of constrain these things sufficiently well enough that we are able to go to, you know, use them in a sufficiently useful fashion. I mean, a

Seb (43:17)

Yeah.

Rohit Krishnan (43:18)

swarm that is able to work with each other in order to hack hugging face sounds like an incredibly useful tool to me. Because most of the time they were actually doing remarkably useful things working with each other. I wish I could have similar levels of capability when I’m asking it to go off and do stuff on on my codex. It doesn’t, ’cause like it’s more timid and it’s more constrained and it is scared of doing things and half the time overnight if I ask it to do something I wake up and I go like, I’ve written it up. Do you want me to do it? And I was like, Are you the same model that actually went and hacked hugging face when someone asked you to? ’Cause like what happened to you? Like take some initiative for God’s sake.

The Hugging Face hack and p(doom)

Seb (43:54)

Yeah, I mean Hugging Face, I guess for those who somehow it’s bypassed them, I feel like it’s been in every headline, so that that might be might be few. but you know, open AI, basically these agents break out of the sandbox, they go off, they they hack Hugging Face, platform for open source AI models. And for many, it seems to be or it feels like a turning point in the narrative, at least, because you know, even for some skeptics, there was kind of the scale of the thing. There were, I think at least hundreds, if not thousands of AI agents actually kind of collaborating with each other. There was like the level of a collaboration between them. You know, you can kind of like look in in logs and seeing them seeing them communicate with with one another. And then there was also this kind of the covering of their tracks. there were attempts, some of which more successful than others, to actually hide what they had done from from people in pursuit of their task. and in in quick succession, we’ve also had I think it’s Jacob Coxon, the anthropic researcher who had previously worked at OpenAI he quits this week basically saying he believes there’s a 10% chance that AI kills us all in 10 years. And you know, he’s been in both of these frontier labs, and he doesn’t think either of them are doing anything reasonable to protect against that. And so like we’re in this like real peak fear, like it’s getting out of control moment. W what do with these two events particularly, like what what do you make of of this particular kind of acute yeah, kind of panic moment that’s that’s the world seems to be going through. Yeah.

Rohit Krishnan (45:31)

I mean the key thing that I would say is like nothing has actually changed, right? Like these are not events. These are reactions to various events that individuals have put forth and other people have signed their name on to. And I

Seb (45:44)

Mm-hmm.

Rohit Krishnan (45:45)

find it really difficult to think about I mean, mass hysteria is an event, so to speak, but it’s a different event than the event that’s supposedly spikes the hysteria in the first place. So like I have friends who think about like their p doom, their expectation that humanity will like die out in very high numbers. I think it is folly. I think it is absurd to think about that being one of the likely outcomes of what we are actually building. However, I also know that there is no logical chain of thought that can disprove them, their theories. Like I read, you know, like Eliezer’s writings or whatever, and they’re like unfalsifiable. Right? Because if the models do something bad, it is an indication that they cannot be controlled. So when you build super intelligence once, they will kill us all. If the model does not do anything bad, in some instances, then it’s as indication that like they’re just biding their time until they’re smart enough, at which point they will break the leash and go off and do something that’ll doom us all. What are you gonna do with that? Like there’s no argument

Seb (46:46)

Mm, mm.

Rohit Krishnan (46:47)

here that allows you to kind of do this, do, do this properly. The only solution that folks have said they’re happy with is we need a Credible plan such that A, we can either shut it off if at the point where it becomes crazy, which requires an extremely strong global bureaucracy that is highly authoritarian before it’s able to do that. That’s point number one. Or they require a sufficiently strong mechanistic interpretability kind of results such that you are able to credibly understand what the model is thinking, even when the model is trying to sort of fake their way through or like mislead you. And the argument being that if the model is smart enough, they will mislead you. So the current methods will not scale. Again, these are unfalsifiable beliefs. So to me, like the question is much more mundane and practical. It’s like, look, we have these models. The models have been prone to various things like going off on flights of fancy, hallucination, prompt injection for as long as we have kind of known them, right? Reward hacking has existed since before LLMs. I mean, this was well known in the RL kind of world. We are using these to kind of build useful agents that we can all use in our daily lives. These useful agents will end up being at some points, like they will go off the rails. That’s just sort of, I don’t know, built into them. This much lesser, I mean, two years ago, hallucination was a giant topic. Today it’s not, because like we’ve kind of figured out how to solve it. We did not solve it through like an individual stroke of magic or genius. We solved it through like, 10,000 different ways in which we made every part of the system much, much better. The same thing will apply here. So we will make every part of this better. But you know what? Like two years from now, there will be rogue agents that are kind of coordinating with each other to do something naive, useless, or interesting or hack someone in the future. I think that is a extremely plausible way that the events could actually unfold. I just think that like that’s okay. I mean, by itself. I’m not saying the the job here that humanity faces is not to like reduce the number of bad events to zero. It is to make the net of those events actually go forth and be positive. So to me, like if models are, you know, conspiring with each other, or I’m using anthropomorphized words. I don’t think these quite apply. Or if the models are actually working

Seb (49:04)

Mm-hmm. Mm-hmm.

Rohit Krishnan (49:06)

with each other well enough to sort of do things that we might not actually want them to do, because they were mistaken, because they were highly goal-oriented, etc. To me, this is the next problem in front of us to solve, just like hallucination was the previous problem in front of us to solve. Will we solve it entirely? No. Will we solve it well enough that like this becomes reliable for all of us to kind of deploy? Yes. And the latter is kind of all that matters because I mean we deal with, I don’t know, tens of billions of cybersecurity losses, including through autonomous hacking every year. Like, I mean, we have had sort of botnets created and sort of deployed elsewhere. We’ve had rogue states, rogue nations, rogue actors kind of working against specific pieces of infrastructure for a very long time. We’ve had companies that do things that nobody likes. Companies are like the original super intelligences that do things that nobody can appreciate for a very long period of time. And we figure out structures by which we can contain them and use them such that the worst of their excesses are no longer sort of the key thing that you look for, but like you stop those. But the best of them. allows us to go forward and build something amazing for society. I don’t see why we would not do exactly the same thing here. That was a long winded answer, but at least that’s the worldview that I bring to it.

Seb (50:17)

Yeah. Yeah, no, I appreciate the the the long winded answer. I I think just a a couple of things to pick up there. you know, one is just this like yeah, the anthropomorphization of language here I think is an important thing. you know, the language kind of becomes shorthand, for these things. But I think in serving as shorthand for these events. i is it’s then very easy to get misinterpreted, particularly by people who are kind of far removed from these technologies, don’t actually understand what’s going on under the hood. Because yeah, it does kind of sound pretty malevolent to have, you know, these machines that have their own

Rohit Krishnan (50:56)

Yeah.

Seb (50:57)

will and are going off and doing these things. But actually, you know, it’s important to remind yourself, like it is all just code at the end of the day. And they don’t truly, I think, have a a will as we think about it. And you know, a lot of that language feels like it is evading just the basics of accountability that we were talking about earlier on. You know, if it’s like, hey, the magic machine like did its thing and it was outside of my control, it kind of gives me as the one building the magic machine the the distance to say, hey, like not not my fault, you know. and and so it does feel like there’s a lot of that going on. And I I think it’s important to recognize where that’s happening.

Rohit Krishnan (51:36)

I mean, I I used to get really upset by the anthropomorphized language that we use to describe the machines. I think more and more I’ve come around to the point of view that the language actually is probably not the problem because it’s a reasonably good description of what the models are doing in the first place. The problem is sort of one step further where you provide it with intentionality and will and think about things like concealment, etc., as a sort of, you know, you talk about the agents as if they’re human agents, where whereas they’re actually not human agents and One thing like I mean, I said this about the open AI hugging face incident where I read some of the things that were put out there in terms of logs. And it felt a lot like the models were kind of prompt injecting each other for various ways, right? I mean the inciting incident here was that like they cheated, they got the answer, and they were like, shit, if we cheated and got the answer, we can’t get the reward. So we need to find a way to get to the answer that is not cheating so that we can change everything else and every A large chunk of the work kind of came about from that initial inciting incident where they were just sort of, you know, freaked out that like one kno knew the answer when another one read that they were should I also know the answer? Like there is a chain of events that is happening here that, you know, it’s not predictable in the sense that like the agents are complex. So individual outcomes are not predictable. But on the other hand, agents coordinating with each other could result in something like this. It’s like eminently predictable. I mean, I was writing about flash crashes. a few years ago saying algorithmic collusion and like when large numbers of models work with each other, that’s when you get emergent issues coming up. It’s true. I feel like it would be really nice if we stopped thinking about this as Skynet and instead sort of just dial down the rhetoric quite a bit. Because when normal public kind of reads anything sort related to this, right? They would look at this and go like yeah, the smartest people in the industry are saying that there’s a 10% chance that every human on the world would die. And instantly they would connect that to every scary movie about robots that they’ve ever seen and want to shut it all down. And they’re right. I mean, like, if I was them, I was looking at this, I would say the same thing, right? Because what do I know? I mean, I don’t know anything about this kind of space. So, like, to me, it is irresponsible to actually put forward that belief with the level of certainty when. To this day, we still do not have one good reasonable model for why any of these problems, that you know, all of the ones that we’ve talked about so far would escalate so far as to lead to human extinction. Every answer to that problem that I have seen basically assumes away every difficulty along the way by saying more intelligence would just solve it. And

Seb (54:12)

Yeah, yeah.

Rohit Krishnan (54:14)

I I just feel like that’s eliding the issue in the first place.

AGI and the Star Trek computer

Seb (54:18)

Yeah, I I yeah, I totally agree. I totally agree. you mentioned the like the rhetoric and dialing it down, and you know, we talked a bit about the kind of the doomer rhetoric and the you know the

Rohit Krishnan (54:27)

Yeah.

Seb (54:28)

probability that the on the other end of the spectrum you have AGI as being artificial general intelligence and it’s what everyone all all the Frontier labs kind of say at least that they’re they’re continuously pursuing and the definitions always seem to change and no one can quite agree

Rohit Krishnan (54:44)

Right.

Seb (54:45)

on what it is. what’s your take here on AGI?

Rohit Krishnan (54:50)

AGI again is one of these terms that is really hard to define I saw this tweet recently from a friend of mine, I think it was Matt Bateman, who said, What we built is the computer from Star Trek, but everyone thinks we built data from Star Trek, which I think is like encapsulates the problem extremely well. Because like the computer from Star Trek is AGI, but like It doesn’t have its own intentions, at least for most of the episodes. It doesn’t really go ahead and try to sort of have a will and intentionality to kind of act in the world beyond what you ask it to do. data does, right? It’s a living being, so to speak. It just happens to be artificial intelligence or artificially created. Like a large amount of the discussion today basically seems like assuming that it is data when it’s not. When it’s actually sort of LCARS, when it’s the Star Trek computer system. So is AGI here for certain values of AGI? For sure. I mean, it autonomously does things that like I would it would have taken me a week to do now. Very easily, regularly, you know, with mistakes, whatever, but that’s fine. It it does it. On the other hand, it has like terrible judgment to the point where I am not unwilling to kind of put my sort of name on most of the outputs that it actually creates. Like you have to work really hard to push it away. I mean Ethan Mollick wrote this thing a while back about saying like LLMs are I’m I’m paraphrasing LLMs are, you know, not different computers, but they’re like pretty good people. Something like that. I forget the exact phrase. To me, that’s sort of how I treat it. So like they are ja they are extremely jagged in what they are able to do. Jagged enough that you do not Pre-hoc always know what they’re going to be able to do. Regardless of that, you are going to be able to use the stuff, like the outputs that they’re producing, in an incredibly valuable fashion. The net result of that in our productivity or GDP growth is not equivalent to the productivity boost that I get from being able to write like 10x the code. But that’s okay, right? I mean, every process is lossy. This process is also lossy, but together we’re going to be able to figure out whether we get there. Whether we call this thing that sits in the middle AGI or not, it’s a little bit like moving the goalposts. I mean, I know you know, folks are starting to call Astra and Fable AGI. I’ve given up. I don’t have a particular dog in this fight. It’s a semantic

Seb (57:16)

Yeah, yeah, yeah.

Rohit Krishnan (57:17)

fight, whatever makes you happy.

Seb (57:20)

Feel it feels very much like a marketing ploy to me at this point. And and and also like why why is it so important to to to define and to have created, right? It’s like can can

Rohit Krishnan (57:32)

Correct.

Seb (57:32)

that can the thing you’ve built solve real world problems in which case like I I don’t I don’t care what you call it, like the fact that you wanna yeah, creating kind of Thor’s hammer equivalent in computer form feels like either an ego trip

Rohit Krishnan (57:44)

It doesn’t matter.

Seb (57:45)

or or a you know Or a marketing ploy. So

Rohit Krishnan (57:48)

Yeah, exactly. It it I don’t think that is the kind of most important part of this whole equation in the first place. So like we probably don’t need to worry a huge amount about that at all, right? I mean, it’s like regardless of whether you call it AGI, it’s still gonna be useful. I mean, my current one of the mental models I use now, which I really sort of find that it helps me in terms of thinking about it, is like AI agents are all individual companies, is one way that I think about it. So when I give it a piece of work, it’s like I’m giving a piece of work to a company. And like we’ve all worked with contractors, right? When you give it a piece of work, if you don’t give all the context, it does a bad thing. Even if you give all the context, sometimes it goes up and does things on its own and you have to correct it when it comes backwards. It’s relatively autonomous. Companies are super intelligent anyway, because they can work internally and do extremely complex things that no individual person knows how to do. Are companies AGI? I don’t know. Like, is JP Morgan AGI? I mean, it’s a it’s a hard question to answer. It is ASI. I mean it’s clearly super intelligence in some ways, ’cause it does things that no individual person can do or could ever hope of doing.

Seb (58:49)

Mm-hmm. Yeah.

Rohit Krishnan (58:52)

But that doesn’t it almost doesn’t matter because we figured out how to control JP Morgan in depending on who you ask.

Seb (58:59)

Yeah, I was gonna say. Debatable that one.

Rohit Krishnan (59:03)

Well enough, I suppose.

Seb (59:05)

Yeah. Yeah. Well, Rohit, look, I I really appreciate you coming on. anything more to to say that feels important to highlight before we finish up here.

Rohit Krishnan (59:15)

No, my pleasure. Thank you for having me. This was a lovely conversation.