Wharton professor and author Ethan Mollick joins Sana to explore how we’ve moved from chatty AI to an agentic era where systems can run multi‑hour tasks and reshape how organizations work. He explains his idea of the “jagged frontier” of AI abilities, why leaders must embrace experimentation and weirdness rather than treat it as just another IT project, and how new interfaces, organizational design, and human agency will determine whether AI becomes a tool for genuine learning, innovation, and better work, or just a way to create more work slop.
Watch the full episode above or read the transcript of their conversation below.
“Nobody knows anything” and the three eras of AI
Nobody knows anything, right? Like I spend my time talking to the AI labs. I've like very famous people. I talk to CEOs all the time and nobody knows anything.
I wanted to kick off with the fact that it's been, it's been almost exactly a year since you spoke to our founder, Joel. And obviously a lot has changed in the past year. So where are we in AI right now?
So you can kind of think about three eras of AI broadly. We talked to what I'd call the second era. The first era was machine learning writ large, right? Everybody was doing, you know, data analysis and you had your beautiful data lakes and you had your data scientists and you were doing predictive modeling and all that's still valuable, but that was sort of what AI meant prior to 2022. Then we had our Chat GPT moment. And the last time we talked, we were sort of still in the era that I will grandiosely call after my own book title, co-intelligence, where you'd prompt an AI back and forth to get answers. And I think we have entered in the last few months, especially what we call the agendic era, where it's less about working back and forth with the AI through a chatbot interface, but more about assigning it to do long running tasks, starting to think about what it means to have organizations built around AI. And I think that that has been the biggest change.
And could you tell us a bit more about why you think the co-intelligence era is over?
From co‑intelligence to agents
So I think that there is still a need for people in the loop, and there's lots of really interesting things to talk about, about how we incorporate people into AI. I think we still have to be centered around people, but the idea that every interaction is going to be you making requests, the AI tells you something, you do it in the world, you're like, I don't quite know how to do this. Can you make it easier for me? Maybe give me the code for this. That back and forth that was sort of the driver of like, let's talk to a chatbot or find this email, do it better, was based on the idea that AI couldn't do anything, right? Like it had to be closely watched and made lots of mistakes. There were lots of errors. It couldn't take action in the world. Now, hallucination rates have dropped. They're not zero, but they've dropped. Models can do multiple hours of work independently. That work is judged by independent people to be as high quality as experts work often. And that changes how you do things.
Has anything surprised you about what's gone on in the past year?
So I think there have been a couple of inflection points that you couldn't predict exactly when they happened that have occurred. I think a lot of us knew, who are close to the space, knew the agentic moment was gonna happen. I think it was somewhat of a surprise it happened as quickly as it did and that the impact was as large as it was, right? So it went from AI as toy to, oh my God, cloud code is here and now we have to change how we do all coding and all work. And I think that has been a relatively rapid change. I've also been somewhat surprised in the last year about how much there was a pivot in sort of early fall, even before that with companies going from a conversation about is AI worth doing to that being answered and being like, how do we use AI? I also think that just in a general purpose, I think the continued acceleration has been a little surprising. I think that we were all expecting AI to keep getting better but there really has not been a lot of slowdown of exponential growth, which I think is interesting.
Why do you think it's interesting?
So everything is downstream of AI abilities, right? The jaggedness of those abilities, the fact that it's good at some stuff and bad at some stuff is a, I think it tells us what humans do, what our work should look like, where AI might fail, why it's problematic to integrate in. But as AI keeps getting better, the growing aspects of that frontier are what sort of determine how useful it is. So the fact that we've gone from, you can pick any scale that you want, right? The ability of AI to do long running computer science tasks has gone up from 20 minutes or a half hour a year ago to four hours, six hours. I mean, that's a fairly large increase and there's no particular reason to expect that this would keep going. But when you talk to the labs, they don't see any barrier in sight to continue development and they seem to be right so far.
That's a good bridge then maybe to talk about the jagged frontier. You coined the term jagged frontier a few years ago and you still believe that the frontier is jagged. We're at this stage where AI models are good enough to win gold in math Olympiads but it was only a couple of weeks ago that OpenAI was able to say that ChatGPT could now correctly determine that there were three Rs in the letter strawberry. So I'd be curious to know what's on your personal jagged frontier? Where have you been surprised at how much you're still superior to the models and what have you permanently delegated?
The jagged frontier: strengths, weaknesses, and personal micro‑frontiers
So the interesting thing with the jagged frontier- right for those who don't know- and it was my co-authors and I too, so I don't want to take all the credit- was- is that it is that as good at some stuff you want to expect, bad at some stuff you wouldn't expect. But that's changing over time, right. So one of the things you point out was that counting the hours and strawberry or the math Olympiad- those were both weak spots of AI that got closed in the last year because of the advent of reasoning models. So reasoning models, starting with a one preview, turned out to be really good at these tasks a I were bad at. So we had this sudden leap or change in the shape of the frontier. Most of the time we're just in the frontier, expand outwards, right, so it's not massive leaps in one area another, they just keep getting better. Or you track coding right. It's been sort of steady and steady but exponential increase in ability. But there's still areas that are weak. So famously, at least for me, long-form fiction writing. AI is terrible. Long-form fiction writing. It writes in cliches, it's it's all the same kind of voice. It has trouble with actual plot points for a variety of reasons. It can't plot out things well enough, it's it's trained to be too nice, so there's nothing bad that ever happens. The characters in interesting ways, lots of issues with that.
But then there's the I kind of micro frontier right things. That it's good or bad at, that you only know if you know your job really well, because there's no department and open AI that's saying: here's, let's map what AI is good or bad at. In my job as a supply chain manager at a German auto manufacturing company or as a, you know, as a designer in a in you know, podcast production company, nobody knows any of these answers. So I spent a lot of time on my own kind of micro frontier. I found that AI models. One thing I've been tracing is: how good are they writing academic papers? The latest models I can actually point in a directory full of messy files I've gathered over the last decade and it could write a pretty good PhD level thesis about that topic. Right, it still doesn't have a lot of good imagination about what a good topic would be, but it executes really well on it. So there's still an imagination piece kind of missing, but that's getting better too. So the shape of the frontier is evolving but I think people have to pay attention about their own frontier and sort of the global one.
Yeah, that makes that makes a lot of sense. And I'd be curious to know then, if you say that AI is still full of cliches when it's writing fiction, but let's imagine that it does get better and maybe you would find it harder to notice the difference—do you care about whether you're reading fiction that is written by AI or written by a human?
Human vs. AI‑generated art and authenticity
So I think that will be a very big question for a lot of people, right, and I think that we're gonna see two forces at play. One is a push towards artisanal stuff, right, but now you don't want everything artisanal. I don't want artisanal code, right, like, I'd like the code written so it actually executes the way I want it to. But I think you're gonna see pushes for authenticity. But also, if you like reading the piece, you like reading the piece, right. So part of what is interesting about AI writing, just in a very specific way, is that it often depends on you putting the effort in as a reader. So GPT 5.5, especially the GPT series, has been famous for these very stretched metaphors, right, like you know, like our conversation is like a- you know, a gap-toothed smile and that doesn't have any meaning, but we can assign meaning to it and that can feel meaningful to us. So the question is, are we okay with interpret our own interpretation meaning, or we're gonna ask the AI to be have a super interpretation of this stuff? Are we gonna want humans to write these things? I don't really know the answer.
That, yeah, and would you personally pay a premium for art that you knew was human‑made?
I mean, I do it all the time anyway, right, I mean, I can get lots of digital art if I want to get it, and I do pay attention to those kind of things. But I think, again, it's gonna be kind of a mixed bag in some ways, like if you get personalized entertainment that's written for you or a story that you've never seen before, maybe that's good enough, right, because you're not gonna commission artists to do that work. So I think it's gonna be like everything else. So there's always a push and pull an arc between mechanical reproduction and and the human element, and we'll we'll have some sort of tension and compromise.
Hmm, I want to talk now. I want to talk about wizards. Yes, you have a thesis that working with AI is like working with wizards. Can you explain why?
AI as wizards
So it was a way to try and talk about agents and the fact that agents are somewhat inexplicable. Actually, all AI is. There's this whole field of AI interpretability, which is, can we tell what the AI is doing? And you sort of can, but you also can't really write. Like you assign a task and it just does the task. Why is it doing it that way? Nobody knows, and I mean this becomes a real problem when you talk about things like how do we market AI? Well, if we don't know how it makes decisions, I can't tell you how to influence its marketing. We actually have some experiments that we've been doing at Wharton where we've finding like almost everything changes the way the AI makes recommendations. Like you put things in different order, or you say this article is written in reddit versus the New York Times- unpredictable ways you get outcome changes right. So not being able to know how the AI thinks—not that we know how humans think that well- makes it hard to know what's gonna happen. So in some ways, you're just assigning things to the AI and then magic happens and we don't have a really good answer for what that magic is. So the co-intelligence going back and forth, the wizard thing sort of we give the AI the you know, we make a spell and then we see what comes out.
Yeah, and you've, you've described wizards as being, like you said, almost unverifiable, impressive and opaque. Do you think that those characteristics are a temporary feature of the current architectures or are they a permanent one?
They're at least somewhat permanent. I mean, they're permanent even it, because even if we had the whole reproducible chain, people wouldn't be using it anyway. Right, like people don't use the reproducible chain of you know, they don't verify as much as they should in a case, but also it's just a nature of LLM's, and, in fact, we might be in a temporary place where interpretability is higher, because the chain of thought that AI's use is in plain English or, you know, Chinese, depending which model you use, and so there's at least some ability to get inside to that. There's no reason that has to stay in a human interpretable form, and so I think that there's an inexplicable miss to the whole process that is gonna be very hard to eliminate. It doesn't seem to be an area where progress is particularly fast.
And how does that make you feel, personally, that we would be developing technology that we don't fully understand? And, of course, AI is now increasingly writing code for itself to improve itself, and so we might end up in a situation where we've created AI that is creating new AI in maybe even languages that we wouldn't understand.
Recursive self‑improvement, takeoff risk, and anthropomorphizing AIs
Yeah, so there's really kind of two branches here. One of them is the big picture, right, recursive self-improvement. The idea is, make AI is better and we enter a weirder world very quickly. This takeoff concept is often called right. So once AI start improving themselves, is there some sort of ever rapidly increasing sort of exponential curve that we can't understand and I think that we don't really know? Like the co-founder of anthropic was, just a couple days ago- at least the time we're recording this- posted that he thought that there was a 40 to 60 percent chance that we'd have AI scientists doing AI work and improving AI by 2028. I think the number one so like this is on everybody's mind and I don't think we know what that world looks like. I can get very weird very quickly. As long as the AI stays jagged, it may not be quite as tremendous as people think, because it's still weak in some areas and better than others.
But then there's the personal aspect right, of working with these systems and I think to some extent you have to kind of commit the cardinal sin of AI right away, which is anthropomorphization, which is the idea that treating the AI like a person. I mean, first of all, these systems want to act like people, right, that's what they do and they're most effective when you treat them like people and your mental model sort of fits best right. So it's obscure. I don't really know what it's thinking, but I don't know what most people are thinking. I mean, despite the fact that I study people for a living and, you know, work with lots of different people, we don't really know what's going on in people's brains. We don't really know how to make the decisions they do, and when you work with these systems—a lot like a clod or a chat GPT- you start to learn it's foibles, like I know. Certain things will like set clot off and it'll get really anxious and then we'll restart things anxious in quotes. So, even though you have to remember these things, aren't people treating them as sort of obscure people with their own sets of foibles and responsibilities and things that worry them and things they're good at, can be a really useful model, and when you do that, in some ways, the fact that they're opaque feels like less of a big deal. That may not be good, though it's interesting.
You're talking about how opaque humans are, and it's interesting thinking about that in the context of enterprise AI as well. Obviously, we're trying to get to a stage where agents are doing doing work and in order for them to do work, they need to actually understand how work gets done. Unless you're in some kind of perfectly documented organization, there's still a lot of contextual information that lives in people's heads. So how might that be a limiting factor to what ai agents would actually be able to do inside organizations?
Organizational context, implicit knowledge, and limits of agents at work
So I think it's worth stepping back a little first and saying organizations are machines for taking imperfect, stochastic actors who are misaligned and making them work in the same direction. Right, that's literally what we do with people. Organizations are superhuman abilities and machines for taking human input and sort of figuring out how to make it work properly. So it's not like it's a hard problem, like if the AI is wrong some of the time, guess what people are wrong some of the time. We can build organizations that combine AI and people to check that. We haven't built it yet, but we could. The bigger question you're asking, I think, is an interesting one, which is how does it figure out the context which to operate? Because when you know i'm a business professor, business school professor and like we know that a huge amount of the context which people work in is never written down anywhere, right, forget a documentation, it's all implicit. You understand what people are rewarded for and what they're punished for. In some organizations, right, getting the hardest job is a reward. In some cases it's punishment- tells you a lot about our organization, but no one's going to spell that out anywhere, right? Should you be friends with people at work? Should you not be? All these little choices are important choices in organizations and they're not specified.
That said, the AI for work stuff tends to be pretty good at picking out context once it has enough information. So if it has the history of project, it often does a lot better than if it just has instructions for how to execute the project. Some of this will also be people making decisions, right? We will need to document things we didn't document before. Another set of stuff is just starting from scratch. Like, it's not clear that you want to translate processes that exist in a human organization and just have the AI implement those things ad hoc, right? So I think there's a lot of tension across this whole boundary here. I think people are underestimating how much the agents can step in from context to understand what's going on. Given the jaggedness, though, it's not everything, but a lot of things. But then also, how much we don't want that to be the answer a lot of the time.
And have you seen any good examples of organizations that are quite advanced in AI or AI agent implementation, figuring out how to get those loops and those handoffs between humans, agents, and systems?
Extreme example: StrongDM’s “software dark factory”
Well, let's start with a really extreme example. There is a company called StrongDM. That's a small software security company. And they decided to build this software dark factory. So a dark factory is an idea for robotics that if the entire factory is operated by robots, we don't need lights. So it's a dark factory because there's no humans in it. So they built a software dark factory. What you feed into it is specifications, not even specifications, basically, roadmaps for where you want your product to go. And they have two rules. No human can write code, and no human can look at code. Everything is written by AIs, and there's adversarial AIs that test the code. And those adversarial AIs spontaneously built fake Slack and fake Gmail and fake Salesforce to test these security and authentication models. And then all that comes out is finished code, and you just approve or don't approve shipping. And they've shipped products to customers without a human ever touching any part of the code base.
There are other rules. You have to spend $1,000 in tokens per person per day, which is also quite expensive. But that's an example of a really radical version of agent work, right? We take the humans out of the loop entirely. I think that's the wrong way to go, ultimately, because I think you actually want people to be brought into the process, either as sources of variance, right? I don't want Claude's decision-making on everything. I wanna use humans so that I don't just do Claude-based work. Even if Claude's work is good, it's not the work I always wanna see because you wanna make important decisions because you wanna make interesting decisions. But that's an example of a really radical approach. And I think the sort of stepwise thing, which is like, okay, we'll just code more, is probably not gonna be the answer. Maybe not the full weight of the dark factory, but there's somewhere in between that I think a lot of organizations will have to end up.
I think there's an interesting bridge there to talking about wizardry, again, and the paradox that you've identified, which is that every time we cede a task or hand over a task to an agent and we're not doing that task ourselves, we are then losing the opportunity to actually build the expertise that we would need in order to form the judgment that is required to effectively assess the AI's output. So I'd be curious to know how do we resolve that paradox?
Experts, apprenticeship collapse, and reconstructing learning
Yeah, now is the time for experts, right? If you're an expert, you're in great shape because you can look at the output of an AI, know whether it's good or bad, right? Like, it's a great time to be an expert because you can be hyperproductive, right? Finally, you can delegate to machines that will do the work for you. You can assess the results. It's like, it's a reason why I think so many experts descended to AI psychosis when they started using agents the first time. Like, I don't know if you've experienced this yourself, but the first time you use cloud code, people kind of go away for the weekend and they come back with this insane output that is often very useful, but like, it's like, I figured it all out, right? Because they've had this image in their head. They haven't had people to implement it. So experts have their own value. How do we create new experts becomes a real problem, right? Because that requires trial and error. It requires mentorship. It requires doing and failing at tasks that get increasingly hard as you move forward. And AI shortcuts all that. But it's not really AI that shortcuts it. It's our choices as people.
So I think we were talking before about art, right? And that you might want human art. I think in the same way, we're gonna want human effort and struggle and companies are gonna have to be okay with tolerating a learning process. And they're not built for that. We've been very lucky for the last 4,000 years, which is apprenticeship has worked out well for everybody. As a middle manager, you get somebody who is going to do the work for you that you don't want to do, and they're going to do it pretty cheaply. And you're going to be able to assess whether good or bad without spending a lot of money or a lot of effort. And the junior person gets to learn the ropes, because even if your manager is a bad manager, you're still learning what's there, what isn't. You're learning the unwritten rules, which we've talked about before. And you're getting assessed on whether you're doing a good or bad job. And that all broke this last summer, because every junior employee, junior hire, would be much better off using AI. It's faster, and the results are going to be better than doing it themselves. And every middle manager would rather have the AI do the work, because the humans are fallible, and sometimes they complain, or they don't show up for work, and you don't want to deal with that. So that break has already happened. And the question is how we reconstruct it. There's a bunch of ways forward, but they're going to take real effort for parts of an organization that aren't used to doing that. We're used to getting the benefits of learning while doing, and we're going to have to separate those now.
There's lots to unpack there. I think your comment about agency reminds me of something that you had actually said to Joel when the two of you met last year, which was that we as humans, we get to make choices. We have agency over what we're doing. And it sounds to me like you're still very bullish on the idea that we do. Who is it that gets to make choices, though, in practice? If we think about individuals in organizations, perhaps in more junior roles that have less influence, what is the kind of agency that they actually get to exert?
Agency, incentives, and organizational design
So that's the organizational design question. I mean, increasingly, the decisions leaders make in organizations matter, because you get agency if you are given it, in some ways, in organizations. I mean, you decide not to use AI, but in an organization where all your compatriots are using AI secretly, you're going to fail in that kind of outcome. So we have to incentivize people to be able to exercise this kind of agency and choice in the right kind of way. And that isn't something that happens naturally, because you still have agency, right? You're making choices, but you'd be foolish to make the choice that I'm not going to use AI, and I'm going to try and struggle through this on my own, because that would hurt you. So we have to set up incentives to incentivize that the right kind of way. We can do this in schools, right? Because I can say, I'm going to test you without AI at the end of the semester, and 80% of your grade is this test, or this blue book assignment that we do in the room, or the essay you write, or the role-playing exercise we have. And that will incentivize you to not just use AI for learning, because you're going to need to perform in this test. We can do the same kind of thing in organizations, but we can't pretend that people can just use AI for everything and still learn all the skills they want to learn without any help from the organization overall.
The bigger question of agency is, I mean, part of the issue is these systems are seductive, right? So we found this in education, which is that the AI wants to give you the answer to the problem. It's a helpful assistant. And it turns out you don't learn very well when people just give you answers. So adding friction. I think people are worrying too much about removing friction. Adding friction at the right points might be really important.
So you've talked about how AI wants to be helpful. It wants to give you answers. That's not necessarily helpful in an education context. What other things do we need to do to actually motivate children to learn in schools?
AI, motivation, and education: tutors vs. shortcuts
So I think children aren't that motivated to learn. I mean, you're motivated in the area you intrinsically care about, right? So you're a kid who loves math. You're a kid who loves reading. I mean, that's always been the easy part, right? And people always complain, why can't all learning be like the learning I like? It's like, because not everyone likes everything, and intrinsic motivation kind of fails at some point. So we rely a lot on extrinsic motivation. You have to take the test. We think, as a society, you need to learn civics, and you need to learn math, and you need to learn your country's literature, or whatever the outcomes are. And we enforce that, right? Like, if you don't get good grades, you don't advance. These things matter for getting into college or whatever comes afterwards, or an apprenticeship program. And so we build the incentives we need to do these things. Given that we have control of people in the education system, education is sort of a solvable problem. We have some really good evidence now that AI tutors are good for learning. The same team, the research team out of Penn, actually, that found that AI hurt learning if you just let students use AI and say, use it to learn. Students thought they learned. They didn't learn. But when you build an AI tutor, you actually get very large impact and improvement in outcome.
So I mean, the end state looks like something we've already known from pedagogy research to do, which is, outside of class, you'll have AI tutors. And I don't know about classes in Europe, but in the US, increasingly, students are assigned this kind of work, where they're asked to watch a Khan Academy video or something outside of class. And then in class, it'll be discussion, active learning. We'll actually try things. We'll assess people in class, too. You sit down, you write an essay in class. So we'll get there. But the road is always long in education, and everything is messy, and there's lots of competing forces. But I think we can do that in the classroom setting. To me, the work environment is where it gets really weird, because we're not used to formal assessment. What happens is your boss sort of says, you did a good job. You did a bad job. But if your boss is impressed by your clod output, are you really doing a good job? Have you learned anything? We may actually have to have formal assessment inside of actual organizations, as well.
What would that look like in practice?
Formal assessment at work, beyond productivity metrics
So, that may look something more like testing. Like, you have a big, there's a big presentation you give in a room about some sort of topic and people grill you about it and you're judged based on that. It may literally be like the kinds of tests that you do for certification in the US for, you know, insurance or, you know, certification or CPA certification where you have to take some sort of professional test. It might be carving up the kind of thing we talked about before where we're incentivizing people to have some non-AI time and we're judging them based on the quality of their non-AI time rather than AI output. There's a general problem, which is if we incentivize productivity as the only goal, we end up in a really bad place with AI quite quickly.
What would be the other goals that we would need to be incentivizing for?
So this is back to the organizational design problem, right? If you incentivize productivity, you are incentivizing work slop. Like you don't want a hundred times more PowerPoint, right? Like that's what productivity looks like. Let's just do more of the thing. Even in coding, right? Like we're having this big problem right now. Agile was, you know, technique developed, I think in 2002 and it's a lot of how software development works. It's not waterfall based. Everyone's doing this. And now you have this element of the Agile experience that is a hundred times or ten times more productive. What do you do with that? Unless you change every other part of the process, you don't gain out of having somebody stand up in every, in every, you know, you know, Agile meeting and say, you know, Claude did my work today. It'll do my work tomorrow. No blockers and sit down. Like that's not a helpful way of moving forward. So you have to design the processes around these sets of things, right? So what do we want humans to do? What do I want them to do? How do we break down the old barriers that we had and create new ones? So a lot is on leaders' heads at this point, actually.
It's interesting. You're talking about leaders there. Everyone I guess is in a situation where no one has used AI tools enough to really know what the framework or the model should be. So how, how do leaders figure their way through that?
Leadership, Lab, and Crowd
So a couple of things. One is you put, you hit the nail on the head, which is nobody knows anything, right? Like I spend my time talking to the AI labs. I've like very famous people. I talk to CEOs all the time and nobody knows anything, right? Like we're all making this up as we go along. So anyone who's like, we have the playbook, they're lying to you. There's no playbook, right? We're figuring it out. On one hand, that's terrifying. On the other, it's great because that means if you create your own playbook, there's actually a source of advantage for you in that. So the model I use, and I think I talked about this last time as well, was leadership, lab, and crowd. You need three things, right? You need a leader team that is thinking about direction, incentives, what you want to do. You need the crowd. Those are the ones who are discovering use cases. Give people in your organization tools. Incentivize them to expose what they're using AI for because otherwise they'll hide that use and then build a lab. You need a group of people working 24-7 and thinking about AI use cases for your organization. And they're going to harvest ideas from the crowd. They're going to take ideas from leadership. They're going to actually build things and not just talk about it. Because you have to invent, right? And the biggest advantage, going back to experience, is experienced people in your organization will know the shape of the jagged frontier very quickly because they'll be able to use these systems, see what's good or bad at. They have every incentive to tell, you know, if you can build the right tools to tell you what they're doing and for you to scale it up, if you can reward them for doing that. And you have the sort of basis to build from that other people don't.
Do you think Leadership Lab and Crowd applies in the same way to, let's say, a smaller 50-100 person startup as it would to a 10,000 person org?
Absolutely. I mean, I think that the size of the components change, but like two things are absolutely clear, right? Which is leadership doesn't change. Like, you need to have a direction for AI. The C-level has to care about it. And they have to care about it, by the way, not just because they're setting incentives and deciding what the shape of the organization looks like, but also because they're doing other stuff as well, right? They have to have an imagination about where things are going. They have to get a sense of how they personally are using it. And you know, they're actually the most experienced people in the organization. So there's lots of reasons leaders need to be in on this. And the crowd needs to do this because the leaders aren't going to invent on their own. You need everyone using these tools to gain any advantage from it. And I think everybody needs a lab, right? Now, that lab might be part-time kind of work. Maybe if you're a 50-person organization, that might be the CEO spending two days a week thinking about AI stuff. I think one of the things I worry a lot about is cognitive load is already very high on all these sets of people. So they set up times like, I'll learn AI this weekend, or I'll have dedicated AI time at some point, or the consultants will come in and solve our AI problem. And they can help you, but they're not going to be the solution to the problem.
So, the only solution is then that combination of bottom-up and top-down and continuous experimentation. I think that there's no other way forward other than, you know, trying things out, right? And by the way, that also means failure. So, one of the things that I worry about a lot is organizations are not used to organizational experimentation. They do product experimentation all the time, like A-B tests, this didn't work, we're too ambitious here. They do marketing experimentation all the time. But they don't do organizational experimentation, right? How do I take my, you know, what happens if I take my engineering team and disperse it? So, I have one engineer working with one salesperson and one subject matter expert. What does that look like? What if I give them an impossible task that you have two weeks to replicate our code base that we spent three years building? That might fail, that might succeed, but you're going to learn something as a result. So, discipline experimentation is also important. I mean, there is a J-curve of productivity, famous for any technology. At first, your productivity drops as you learn how to use it, then it goes up afterwards. You have to ride through the J-curve and a lot of organizations don't have the willingness to do that.
I think, actually, that could be a good bridge to talking about weirdness. You recently wrote an article for The Economist titled, IT departments are where AI goes to die, in which you argue that IT's biggest failure is essentially failing to recognize that AI is inherently weird. Can you unpack the argument?
IT departments and resisting the urge to “de‑weird” AI
Yeah, I've realized I've made enemies with IT departments everywhere and my computer stopped working. So, I want to say that title was not my choice, right? The Economist made the title. But the point is actually real, which is, it's not just IT, legal, other things, they're risk reduction organizations, right? So, there's two things. They see AI as a source of risk, which it is, and their job is to make sure that you're not exposed to risk. And that's their incentive. So, I mean, IT departments would really be happy if no one had keyboards, like would solve a lot of problems for organizations, right? So, that's not where you're going to get your R&D breakthroughs from. And the other thing is, there's a desire to normalize this technology. I mean, we're talking about playbooks. People desperately want to bring in somebody who's just going to give them the standard playbook, right? So they can just implement AI the way everyone else is implementing AI. And it's just this quest for this stuff. And the easy ways to implement it are all ways that make AI just a fuzzy processor in a sequence of things. Great. Now, we can finally search across all of our documents in natural language. We've solved the natural language processing problem. It's such an unambitious way to view these tools. So, de-weirding them makes it easier for us as organizations, as leaders, to put this in a box or a bucket or to not worry about it. But this is an inherently super strange technology. I mean, in the course of three years, we've gone from like nobody knowing this stuff existed to we can get PhD-level math work out of these things. And it writes a pretty good video game. And it can teach your kids things. And it does art. And like, you know, reasonably good strategic judgment and beats ethicists in ethical recommendations. Like, how do you deal with a system like that? Well, we can just decide it's just a fuzzy language processor. Or we can embrace the weirdness and move forward. I really worry that the natural desire and the exhaustion with the pace of things forces people to think about this and try and think about this as just another piece of software. And it is in some ways, but in most ways, it's not.
In practice, how does one actually optimize for weirdness? It would be a pretty unusual KPI to have inside of an organization, not something that we're used to. What does it look like when you've optimized for it?
So it is not optimizing for weirdness. It's recognizing weirdness, right? KPIs are the biggest enemy at this point, right? I think like they force you into very bad paths in the experimentation phase, right? You can't KPI, like the very nature of saying we need a 10% improvement constrains the kind of use cases that you see, right? And so KPIs are a really dangerous outcome because KPIs are basically tracking past behavior and saying we want more or less of this. When you have radical breaks, you need some room for radicalness. Doesn't mean you don't have to get rid of all your KPIs, but there's some room for radicalness. And we know how to do, I mean, breakthrough ideas, right? Ways that radically change how we work. Those are weirder uses of AI. And so that's where allowing experimentation failure starts to happen, right? So I want to see, you know, I want to see a new organizational form come up with, you know, do this. I want to see a new product line. I want to see us reenter a market that we left because we couldn't compete and try and, you know, compete with it, like big picture stuff, not small.
Enterprise AI is obviously expensive. So I wonder whether the weirdness angle puts CIOs in a bit of a tough spot or ID departments. They are on the hook, obviously, to be able to show to their organizations that they're getting some kind of value from all of that investment. Does that mean that they have to write a completely different kind of business case or try to convince other people inside the organization that we need to eradicate these kinds of kPIs all together? How? How do they navigate this tension?
Tokens, costs, and new kinds of KPIs around innovation
Well, we're seeing companies first of all. I think just if this is the CAO arguing about token spend one way or another without other leadership buying in, you're already kind of in a tough place. Part of this is actually avoiding some of those problems. Like I mean, for a little while, Meta had a token leaderboard about how many tokens you were burning. That's obviously not the right approach, right? I mean you can think about AI expense like organizations there's- you know, when we talk at senior level- people like yourself in an organization. You, you're very competent, your time is very expensive, so there's an entire part of an organization whose job is to filter stuff so that you get to make interesting decisions that are impactful and you use people with the right salary level and right effort level to solve problems in different ways. Part of organizational experimentation also is about thinking about which models we use when. What tasks require us to use a lot of tokens versus not- and it moves pet like as soon as token burners or KPI. You're in trouble, right, and so part of that is about realizing that part of this is a reinvention of the organization, which is expensive but actually leads to lower costs in the long term, not just and not necessarily from headcount reduction, from just thinking about token optimization.
Second piece of this is you do want different kinds of KPIs. Like everybody should be in charge of innovation to some extent. I want to see new product development or new organizational form or something radically changed. I want to see it before and after, and I'm seeing organizations do this right. Like this is not an unusual case. It's hard to write that, and if you're going through the normal writing process, you're in trouble. It's kind of why the sea level needs to be all in on this to some extent.
What are some concrete examples you've seen of organizations that are successfully recognizing the weirdness and thriving because of it?
So we're in early days, right. I mean, one of the things that I think people are- especially technologists are- too ambitious about is organizations are slow, right. Even if you had all-hands emergency board meeting about AI that takes three months to schedule, right, and then two months afterwards for the follow-up meetings, and then, like, organizations don't move that quickly. Some do, but most do not, and that's why urgency is kind of important, but seeing a lot of really useful kinds of experiments and changing direction—right. So let me think about what once I could talk about publicly. So, like Colgate, Palmolive, the people who make toothpaste stuff, they have a really cool AI lab that they've set up that works directly with the Sutter sea level, and they're doing things like having the AI auto generate ideas from, you know, conferences and build prototypes more quickly, right, so everything becomes quicker. And how do I put things? How do you put things out in a more rapid way? There's a lot of organizations that are doing that have found really fast changes in, like legacy software implementation. So they've gone from like a 10-year plan to update their cobalt infrastructure to we're doing this at six months one way or another, right, so setting very high, almost impossible goals. And there's a lot of other really cool stuff that I don't know if I'm allowed to talk about, because I talked to lots of companies about it.
You said in January or March that the bottleneck is no longer AI capability, it's the interface. What would you then expect to see or want to see from companies like sauna that are innovating on the interface layer.
Interfaces as the new bottleneck and beyond chat‑first
So the way to think about AI labs, right, as I think people think of them as sort of omniscient. They're young organizations that are good at one thing, which is building LLMs, and the second thing that they were good at then is coding. So it is no wonder that the interfaces for coders are really good from the air. I'm sure at least we've got perfect, at least imaginative, in that way, because they understand this stuff. They're all coders themselves, so they're building their tools, they're using their tools. What they don't understand is any other job in the universe, right, and so they're trying. Right, like it turns out, lLMs are unreasonably effective tools for so many things that you don't have to be that good, like you don't have to have built code to do marketing. It's cloud code, it'll just do marketing. Codex does a really good job of strategic advice, right, like, and the things give medical advice have no idea about any of these topics. The problem, then, is not the capability of the systems, it's that the interfaces that people use to understand them, that to do the work, to think about how work gets done, are completely lagging. What the capabilities of the models are, right, so we coding. It looks great but, like you, were starting to see some elements for how design might look in the AI world, but vast amounts of work are undone. You know, unthought of right, this like there are 14 times more managers than our coders and there's no management interface for these things, right? So how are you supposed to manage a task using AI? There's no real idea you're assigned the work. The AI just does stuff. It's impossible to figure out what it does. There's no audibility. You can't decide how it approaches the work. The interfaces can solve a lot of these problems, but they require experience with what the world actually looks like and a lot of these companies are missing that.
Yeah, we have been thinking about this a lot, about how to move Sana away from being a chat-first interface, first and foremost, and how would you design an interface that can allow for humans, systems, and agents to hand off to each other in an effective way? I'd be curious to know, though, if you could wave a magic wand and change anything about how you interact with AI today, what would you change?
The systems are so much more capable of being helpful to us than we're letting them be. Like, we just had this whole conversation about context, and yet, type something into a window and look up a document. I mean, they have video capability, they have voice capability, they can create images, they can see images, they can open documents. They should be working with you on things. You should be co-controlling your mouse. You should be thinking about having documents pre-prepared for you that come up. I mean, I've been doing this kind of thing, where I have the AI literally look through my emails and sort of prep me on anything I need to do. It even writes email drafts, which I've not had the guts to send yet, so everything you get is still human, at least for now. But, I mean, this failure of imagination comes in large part from when chatbots were the way forward, and it isn't anymore, right? Agents delegate sub-agents to do stuff. Why isn't all my work being checked? Why aren't I getting feedback dynamically as I go? And I think part of the reason why, like, OpenClaw, which took off, and probably a lot of people watching this are familiar with, part of the reason why it took off was it showed a new mode of AI as personal assistant. And there was obviously a huge demand for that, too. Just like we saw AI as coding assistant. We're going to see this in lots of other areas. And I think being really, like, making it disappear is in some ways what you're aiming for.
And obviously, we were just talking originally about bottlenecks. And the first thing that we talked about there was the bottleneck of the interface. But what about other more human-related bottlenecks? Everyone's obviously talking about taste, and saying that maybe taste is going to be the new bottleneck. Taste is still what is uniquely human to us. Do you buy that argument, or do you think we are self-soothing until the AI gets good enough to have good taste of its own?
Human skills: deep knowledge, wide knowledge, taste, and agency
So, I think that there are two ways of viewing this stuff. I actually tend to say the four things that I think about most as sort of the elements that you want to train yourself in are deep knowledge, expertise in an area, we've talked about that before. Wide knowledge, like, I think it's time for the humanities major again. Like, knowing many things is really useful. These systems can do anything. So, having them do many different things is interesting. And then I think taste is important, but not necessarily like a competitive advantage. It's just your taste is a differentiator, right? And then agency, just doing stuff, right? So, not necessarily surprising. I think, though, that there's also a danger of bright-line arguments, which is to say only humans can exercise judgment. I hear that a lot. That's obviously already wrong, because if you give an agent a seven-hour task, it's exercising judgment all over the place, right? And so, AIs can obviously do judgment for some value of judgment that we can discuss. And so, I think I worry about the bright-line arguments that humans can only do this. I think we can articulate a world where taste matters because you don't want everything to read like, you know, let me sit with this for a bit. This is a load-bearing part of the argument. It's not X, it's Y. Like, clawed writing drives me insane at this point, right? Even though it's not bad writing. And I crave variation, so that's where taste might matter. But that's quite different than saying only humans have taste, right? And so, I think we're going to have to think about those aspects. I think the bottlenecks are often about the complexity of tasks on a jagged frontier. And there are some areas that AI can do well, and some it can do badly. And the bottlenecks become the things AI can do badly, or only does badly, is where human efforts need it the most.
Two years from now, do we still have white-collar jobs?
Jobs, the O‑ring model, and chaotic transitions
Obviously. I mean, we can have a super-intelligent machine. We might already have them in some ways, but we have super-intelligent machines, and very few people's work will change the next day because it's still jagged. And it's just even jagged in the physical world. Like, somebody is attending the meeting. Someone's going to something. There's tons of unwritten material. You can't just drop in. The smartest person you can imagine, if you drop them into a job, and even 1,000 of them into a job, you're still going to have a lot of trouble making things operate, right? So there is obviously some big-picture issue for two years from now. But let's talk about five, seven years from now, which is actually when all the AI labs are saying jobs get destroyed. There's sort of an emerging economic consensus. We don't know the answer, but the consensus is around something called the O-ring model of jobs, which is grimly named after the Space Shuttle Challenger disaster, where thousands of parts worked well in the Challenger, but the O-ring failed, right? In an O-ring theory of jobs, we do many different tasks. And some of those tasks are really important in a way that is not just 10% better, 20% worse. But if they go badly, the entire job falls apart. The entire work stream falls apart. And what happens is those become the, to use a clod term, the load-bearing pieces of the entire job organization.
And by the way, we can see this happening, right? In the early, even before the pandemic, but certainly during it, there was a huge demand for software developers, right? And if you look at majors in the US, everyone switched to software development. And even before AI, software development hiring started to drop. And everyone responded to this by switching their majors and hiring became less. And what you see happen now with the rise of cloud code is it's not about writing as much code as it was before. It's about being a manager of software engineers, about thinking about where things need to go, what tests to write, what's something good or bad enough. So the nature of the O-ring changed, right? From can I write good code that isn't spaghetti code to can I be a good software engineer and think in this higher level? And that might be something the AIs do next. Then there'll be another stop in this. So these bottlenecks are gonna be ever-changing inside jobs in the economy where people like suddenly AI does this well, but now the next tension becomes it doesn't do this well. And then the other thing is competitive advantage. It turns out that tokens are really expensive. AI can do many jobs, but you don't want to do all those jobs. Humans are sometimes cheaper or better than the AI at doing particular work. And we see for a long time, as long as we're token constrained, that there's more work to do than there is necessarily AI even to do it. So I think we're gonna see a lot of weirdness happen. There's going to be a lot of chaos and confusion. The big picture view people always take is the industrial revolution, which all three of our industrial revolutions have worked out well in the end. People got more jobs in the end than they started with, but they all kind of suck to live through. Charles Dickens is all about how much the industrial revolution sucked. And so I think we're in for a chaotic time. And I think being broad based will help you in a place where this chaos is occurring.
So I want to drill into what you said about management and being broad. I know you recently wrote a piece about management as an AI superpower. On the flip side, Silicon Valley is really proclaiming the resurgence of the IC, the death of middle management. I'd be curious to know how you would defend that argument rather than your own, or do you think that they are actually more complementary than I might be thinking?
Management as AI superpower, role collapse, and builders
So I think there's a difference between big M management and small M management. So small M management being a superpower is the better you are at managing work, the better you are using AI. If you can define what you want, all the things managers do, right? So just to give you a bunch of TLAs, three letter acronyms, right? If you're good at PRDs, if you're good at SOPs, any of the RFPs, any of the stuff that we do to give commands to people to get something done, AI is increasingly able to just take those commands and run with it. So one form of superpower is being good at assigning work, understanding where the risks are, where the downsides are, assessing output quickly, having the expertise to do that. That small M management makes you good at AI right now. Then there's sort of the big M management question, which is how do you manage organizations or think about these kind of issues in a bigger sense? And I think there is some question about whether you have management layers collapse, right? Can the CEO just manage more things because a lot of the middle management layer is done by AI? Or does this increase the span of middle managers? One of the trends that I've been seeing in companies has been collapsing the role of product manager and coder together into a single builder title, for example. And everybody just does the same work. We did this experiment at Procter & Gamble where we took 776 employees, had them work either individually or in teams of two, cross-functional teams of two, business and technical people. And this was using GPT-4. So really obsolete system at this point, not agentic at all. But individuals using AI performed as well as teams of two not using AI. And what was more interesting was people's job difference started to collapse. So if you were a coder before, if you weren't using AI, you came with coding ideas. If you're a business person, you came with business ideas. Once you use AI, you execute ideas that look like both ideas together. So one of the big issues is going to be this kind of role collapse. Like what does a manager do? I think management's gonna be a very different role than it was before.
Does that mean then that functions and sub-functions might also collapse? I mean, AI agents are getting good, increasingly good at working cross-functionally as well. Could you foresee the collapse of a marketing department, an IT department, a sales department, for example, and reorganizing teams more around the problems that they're trying to solve?
I mean, there's a hundred ways we can imagine reorganizing things. So first of all, we are in the very early days of actually organizing agents well. It looks great, but then when you actually start to look at them outside of coding, the agents forget things. They lose things. As they work cross-functionally, things sort of get messy. I think a lot of that's solvable problems. Better skills, better interfaces between the agents. There's a lot of things we could do to make that better. Better human checking. But I think that the real question is why are we organized this way? Well, the first org chart was invented in 1855 for New York and New York Railroad, and we've kept org charts roughly the same ever since. You really have two choices. Is it a divisional structure, which is focused on sort of the customers and outputs, or is it a functional structure based on what we do? Then you've got the matrices and other godforsaken alternatives, but those are really the two big choices that you have. And those don't have to be the only choices. It could be by taste, right? You could end up saying that we have, that this is our taste selection agent, and this is our agent that does judgment, and this is our agent that does customer simulation, and all of those weigh in, right? So you can start to think about many different ways of organizing. I think small, cross-functional teams are the short-term way to go. And I think building around projects is probably the right way to do this.
I think the question is: when do you need to call in the expert? Is the sort of unknown question, right? So we, you know I've got a three-person project team. There's no lawyer on it. Are we okay with Claude being the lawyer the whole time? Like, when do we call in the lawyer? And is this is a some sort of supporting function, or are the lawyers also building their own products? And again, we talked about leaders are repeatedly throughout here. This is where there's sort of a heavy as the head that wears a crown moment, because all of this falls on leaders. They cannot just write a check to somebody to solve this problem for them. They have to figure out how to do this themselves. They can get support to do it, but it is a hard problem.
So, as far as I understand, you are a parent of two children. Yes, I don't know how old they are, but I would love to know what your philosophy is on AI and parenting.
AI and parenting: supervised use, tutors, and careers
So one's in high school, one's in college, but we've been using AI for a while, so there's a lot of different pieces here. I think you treat it like other forms of technology, which is supervised. Use is always the best way to go when they're younger, right, so young kids like I think there's lots of delightful things you can do with AI if you have the leadership role in it. Right, you're making decisions about what's good or bad for your kid, creating images together. There's a lot of fun games, things like that. For a learning perspective, I think supervised learning use when the kids are, when the kids are younger. Probably the best way to do him is to actually have the AI help you become a better teacher. So take a picture of the problem and explain to me how it explains to my kid, right, as they get older. Using learning modes in the AI systems is really good, right, so you don't want to just use the AI itself, you want to use it in tutor mode. There's some, you know, kid focus ones. Again, I'd monitor use. And then there's a sort of wider picture about like: okay, so what does this mean for careers? And I mean right, like I think there's some degree of like, making bets that plumbing is gonna be the job of the future feels like a waste. So you let your kids pursue what you think they should pursue. You teach to be flexible. Wide knowledge, deep knowledge, you know. Give them agency. I don't think things change that much, and so I don't. I think, compared to other technologies, I suspect that outside of using it for cheating- where there's real danger, or uses an education, where you're fooling yourself, you're learning because the AI is explaining stuff to you but you're not doing it yourself. Outside of that use, where there's a lot of like you know being careful about this, I think it works like other technologies, but, you know, is it better, worse than tick-tock or anything else? I think there's less passivity there and there's real value in it, but again, it has to be parent guidance. We're back to agency again.
Ethan. You've- I mean you've- arguably—looked at and researched almost every aspect of AI and what's going on right now. What are the big unanswered questions that you would still like to see more work being done on, or any overlooked questions that you think we need to solve?
Unanswered questions: organizations, “how good and how fast,” and pride
So in the work world, I think we don't know anything about organizations at AI, like almost everything's on looked like. I mean, it's weird that the AI companies have all now building their own consulting arms to do AI deployment right. If the models are so good that they can develop, you know that you think they're gonna show what color jobs, shouldn't they also be able to help you with deploy systems. I think that's a really great example of the jagged frontier work and we don't understand enough about that, the biggest picture. There's only two questions that actually matter a lot, which is how good and how fast, right. So how long this exponential curve continue and at what point? At what point does it ease off and how sharp will it be? Because that determines everything else. Everything we've been talking about today is based on our understanding of where things are today and assumption. The path of the future looks like the past. Last year, when we talked like, I think you know the sauna team and I knew agents were coming, most people didn't. It would have been a very different kind of conversation and I think that things are going to keep shifting in that kind of way. So I think really what we should pay attention to is: what does that curve look like? Is the jagged frontier still closing and we're trying to gas a few months out? Otherwise, you just get stuck on where things are today and extrapolate from this sort of you know point where we are, as opposed to thinking about where we might be going.
You've been at the frontier of this. You're an educator, you're writing for a very, very broad audience. What are you most proud of?
I think trying to make AI more human, and I think I've done that. Trying to do that in education by building tools that help people learn and then they've been adopted in huge cases, but also through the writing and everything else of like we not letting the technologists win. This has to be humans and managers deciding how to use these tools, and policymakers, and we shouldn't view the technology as inevitable.