The assistant is the PI
May 20, 2025
Since they are trained on human writing, LLMs are fundamentally downstream of existing human ideas. This leads people to believe that all these models can do is remix and interpolate, rehashing what we already know. I don’t buy that.
This tension came into focus during a conversation with a scientist friend who studies Biological Oceanography. We ended up going in circles about whether AI can actually generate new knowledge. Not just reorganize or compress, but can it originate something meaningful? Her take was no - you’d still need humans to check its work and deem it valuable. My wife also wasn’t convinced. Her sticking point was the human input: someone has to write the prompt, craft the dataset, spin up the cluster. If we’re the ones setting the stage, how can AI be more than a tool and not us be the ones creating new insights? Those views overweight authorship and process, while undervaluing the outcomes.
When Google researchers trained an AI to learn Go from scratch—just the rules, no prior games from grandmasters—in playing itself, it developed strategies that no human had ever seen before. That’s not mimicry, but concept formation. Yes, Go is bounded. There’s a clear win condition. But the search space is enormous—too large to brute-force optimize. To find those strategies, AlphaGo had to abstract, generalize, and test. It learned principles, and made discoveries. That’s how AI knowledge starts: in sandbox environments where feedback loops are fast and the stakes are legible.
Dario Amodei describes how this knowledge generation process could work at scale in a thought experiment summarized by the phrase “a country of geniuses in a data center.” Thousands of agents, trained on everything we’ve ever written down, running 100x faster than any human scientist, feeding each other new hypotheses, surfacing edge cases, refining arguments. They don’t sleep. They don’t forget. They don’t care about disciplinary boundaries or academic norms. That’s not a metaphor—we’re already building systems like this. Maybe they’ll be inefficient. Maybe they’ll waste water and power and generate junk that you or I can’t find a use for today. Doesn’t matter. Scientists also misallocate resources, and we might have different opinions about the societal value of their discoveries. If even one of the outputs of these AIs is novel, useful, and verifiable, that’s knowledge.
And sure, someone has to hit the power switch. Someone writes the first prompt. But if those agents start generating hypotheses, proposing experiments, and then—whether through simulation or lab robotics—testing them, it’s hard to see how that’s different from what we already call science. In most research settings, humans delegate too. We trust instruments. We write code. Replacing the PI with AI, while keeping the sea of postdocs and the peer review process wouldn’t change the fundamentals. This is just more delegation.
It also matters how these models work. They’re not doing grid search. They’re not stepping through permutations. If they did, and that created useful results you could maybe argue that that’s not science. But autoregressive generation isn’t about optimizing a fixed objective, but exploring what could plausibly come next. Most breakthroughs aren’t inventing from nothing. They’re recombination. Reframing. Taking an idea from one context and seeing how it fits in another. These models, trained across domains and cultures, would be uniquely good at that. They don’t get attached to bad theories. They don’t defend their priors. They can hold way more information in memory than we can, and make connections we surely couldn’t. Occasionally, this process surfaces something we haven’t seen before—but that holds up anyway. Knowledge.
I get the instinct to resist this idea. Scientists are trained to be skeptical. But at some point, we have to separate the infrastructure from the output. If the ideas are useful, novel, and testable—if they move the boundary of what we know—it doesn’t matter who or what wrote the prompt.
Thanks to Hannah Doherty and Shannon Doherty for enduring my talk about AI, and for their feedback on early drafts of this post.
Photo: Ideas, Reflections of Ourselves, by me. Previously posted on CDMX, 2025.
Want to see more articles like this? Sign up below: