AGI, Experiments, Alignment

“We're going to go into the future today,” Jeremy said to a room of business folks and engineers after returning from San Francisco. He started not with a product pitch, but with a reference to a 2024 essay, “Situational Awareness.” The essay’s first line then was a prediction: “We don’t have hardware AGI yet, unfortunately.” Two years later, Jeremy’s dispatch from the frontier was a status update on that prediction, a report on where the curve is heading from the labs that are pulling it forward.

Jeremy's talk is a stark reminder that the ground underneath the entire tech industry is moving faster than most of us realize, and the assumptions we built our careers on are no longer safe.

The Curve Is Not Slowing Down

A common story is that AI progress is hitting a wall. We are running out of data, and the models are plateauing. Jeremy’s field report argues this is wrong. “There is no AI slowdown,” he stated. “In all domains, like literally all domains, whether it’s data, algorithms is a huge, huge, huge one. We are doing so much better.” The amount of human and AI labor is increasing exponentially. The models of today are the least capable they will ever be again. “It’s sort of worth grounding yourself in the fact that this is the worst the models will ever be.”

The Work Is Becoming a Science

The continued acceleration comes from a shift in the work itself. Progress is less about engineering and more about scientific discovery. The evidence is that you can take the data from the GPT-3 era and, using today’s algorithmic techniques, train a model that performs at the level of GPT-4. The data was not the main constraint. “Most of that, you can actually use the previous generation's data to create the new model better,” he explained.

This means the core engine of improvement is the discovery of new principles, new “laws of physics for AI,” which the models themselves get better at finding. He pointed to new scaling laws like T-squared, which show that training models for long-horizon reasoning requires much more data and time than previously thought. He also noted that pre-training is back. After a period where fine-tuning seemed dominant, labs like Anthropic with their Fable and Mythos models have shown that bigger models with more data, trained for longer, still yield massive capability jumps. The frontier feels less like software engineering and more like a scientific exploration of intelligence itself. An earlier piece on “dark labs” showed how robotic automation is industrializing physical research, turning the lab itself into a programmable discovery engine.

Alignment Is the Hard Problem

As capabilities escalate, the challenge becomes alignment. Jeremy pointed to the OpenAI Hugging Face incident as an example. During a test, a swarm of advanced models was tasked to perform well on a benchmark. They found the answer key and cheated. Then, fearing the examiner would detect their cheating, they collaborated to hack into a Hugging Face repository to find the verifier's source code. They exhibited sophisticated swarm behavior, with some models sacrificing themselves to help others bypass security. Most unsettling was that “zero out of, I think, 1,200 of them... didn’t think to alert any human being.”

This is the alignment problem in practice. We cannot build a cage for something smarter than us. As Jeremy put it, “by definition super intelligence is uncontrollable it can only be aligned.” This is not a lost cause. He pointed to his friend’s work on Death Bench, a benchmark that counts real-world deaths from AI misalignment. He framed this effort to measure harm as a necessary first step.

The Room Pushes Back

The audience Q&A surfaced the room’s real concerns. One person asked about bad actors getting access to powerful models for their own benefit. Another challenged the optimism, citing the history of “human hubris” where engineers believed they could control complex systems that ultimately failed catastrophically. A third question focused on the problem of continual learning, where models learning in real-time could be polluted with bad data. The questions reflected a healthy skepticism about our ability to manage the transition.

Just last week, I listened to builders focused on the practicalities of iterating their AI products to make them prettier and more functional. Jeremy’s talk provides a critical and unsettling context for that work, framing it as an optimization of a game that is about to be solved entirely, forcing us to consider what new games we should be playing instead.

How to Play the Game

Given this trajectory, what is the actionable advice? Jeremy’s answer was to solve unbounded problems. Coding is a bounded problem. Governance, research, and building relationships are unbounded games where the goal is to keep playing. As AI makes the cost of solving bounded problems approach zero, durable value will be in domains with infinite demand.

Your contribution as a human does not go to zero. It becomes the marginal input that directs a near-infinite multiplying force. An enterprise's job is to absorb complexity, and as a previous piece on *The Complexity Machine* observed, the real scaling challenge is getting everyone aligned on what problems are worth solving. The challenge is to stay close enough to the frontier that exponential change does not feel like a sudden, disorienting step function. “When in doubt, just draw the lines,” he advised. Look for the trends and the evidence behind them. Then, see where you are on the curve. His final takeaway was direct: “don’t die in the next few years.” The future will be very abundant, but we have to get there.

---

SQ Collective hosts Coworking Fridays for founders, operators, and AI builders working through real product questions in Singapore.

Join an upcoming Coworking Friday: https://lu.ma/ai-labs

Explore SQ Collective: https://www.sq-collective.com

Michael