July 2025 BenGoldhaber.com Newsletter
leaps of faith and AI community theater art this month
We launched the FLF Fellowship! We’ve got 32 technologists, researchers, and builders working on projects in AI for Epistemics and Coordination. It has been very gratifying bringing together this group of people and seeing, as one fellow put it, the emerging scenius.
We’re in week two of the fellowship and I’m optimistic we’re going to see some amazing stuff built. A friend created some art that represented the fellowship, here’s my favorite:
FYI the fellowship is remote friendly, but with lots of folks working from SF, I’ve taken the plunge and will be living in SF August and September. If we haven’t seen each other in years because crossing the Bay Bridge is too psychologically taxing, now is our moment.
Herasight: embryo screening and trait prediction - the team published an in-depth technical report, though I’ll admit I mostly just played with their calculator.
I suspect that the market for genetic screening and selection is very large. I’m guessing a lot of people - if given the option to select embryos for specific characteristics, would publicly react with distaste, but would privately - modulo cost and ease - would use it (see also GLP-1s). It’d be an interesting investment strategy to aim for this class of preference falsification products!
Review: Leap of Faith: An excellent book review on how the ‘decision’ to invade Iraq was really made.
By far the longest section of this book is a detailed blow-by-blow of the actual decision to invade Iraq. I will summarize over 300 pages of densely-spaced text for you: nobody ever actually made a decision to invade Iraq. The system as a whole “decided” in the sense that that was the outcome, but this was emergent behavior, like an ant colony discovering a source of food. At no point did any human being sit down and say, “okay, now we’re going to debate the pros and cons of invading Iraq,” and then make a decision that the benefits outweighed the costs. Instead a vast multitude of people — with different goals, presuppositions, and beliefs about the world — interacted over a period of years, and at the end of it an invasion occurred. At some point there was a phase change. At some point everybody started assuming that somebody must have made a decision that we were really doing this, but in fact nobody had. Does a molecule of water decide to join a flood?
Like the author, the Iraq war was a definitive, “core memory” for me. It’s bizarre looking back, two decades later, and seeing how this two trillion dollar war has been forgotten1. And I agree that for all the ways in which people have profiled “how we got Trump”, it’s under-appreciated that he was the candidate who campaigned on how Iraq was a big mistake.
one big downside of the US government as a whole making the decision to invade Iraq “unconsciously” is that it was never actually debated or discussed in blunt term. This meant no hashing it out in a big room, no arguments sharpening each other, no big list of pros and cons, and, crucially, no big list of risks and how to mitigate them…
The trouble with a corporation or a government is that, considered as a mind, it is a very strangely-shaped mind. Is it hyperintelligent, or is it retarded? Clearly in some ways the former and in others the latter. And the process by which it comes to decisions may be no more random or irrational than the process you and I use, but it’s less familiar to us, and that matters a lot. But everything I’ve just said about collections of people is even more true of the artificial intelligences we’re beginning to create. We are not prepared for the sheer diversity of cognitive architectures that is about to flood the world.
Underwriting Superintelligence: A new addition to the beige-background-single-webpage-essay-on-AI meta, this one makes the case for kicking off an insurance market auditing AI agents and frontier AI companies. Insurance is a cornerstone for private governance solutions, and can respond more flexibly and dynamically than many regulatory bodies (though some version of that is needed to help set a price on mistakes), so I’m generally bullish on companies being able to quickly scale up in this area. Related: I think Cloudflare is going to be one of the cornerstone private governance companies in AI.
More compelling research from Owain - seems like large models can transmit data and preferences to “student models” in unexpected, hidden ways.
Multiple frontier models (DeepMind, OpenAI, Harmonic) achieved IMO Gold performance. A big milestone. Impressive related result was that, through careful prompting and pipeline design, Gemini 2.5 (not-finetuned!) is capable of IMO Gold performance.
New favorite Veo3 prompt collection:
24/7 AI Livestreamer in China is prompt hijacked to engage in a marxist analysis of capitalism in between shilling pork rinds. Related: AI Village competition between AI bots to sell the most t-shirts causes Gemini to have an existential crash-out.
Lessons from a Chimp: AI "Scheming" and the Quest for Ape Language: Chris Summerfield et.al juxtaposes the research practices that some AI safety groups are using to assess whether AI’s are scheming in pursuit of their goals, and those of animal behaviorists in the ‘70s when many, mistakenly, thought primates had far greater capability to communicate with humans:
We argue that many of the research practices adopted thus far are not sufficiently rigorous to allow strong claims either way about whether current AI systems can ‘scheme’. We illustrate this by analogy with research conducted in an earlier era, that was designed to test a research question that was similar in spirit: can (non-human) apes learn language…
Several factors conspired to mislead an entire field.
There was a cycle of hype around an astonishing hypothesis. People were entranced by the idea that – in a sort of real-world version of Dr. Doolittle – we would actually be able to talk to animals…
Finally, and perhaps most importantly, there was a failure to articulate an adequate theory of the phenomenon under study, with evaluations of success defaulting instead to a sort of informal, ‘know-it-when-you-see it’ criterion
It does seem important that, while we should absolutely be exploring whether AIs are scheming, we should be very careful to avoid clever hans self-delusion. I’d like to see more adversarial collaborations between skeptics and proponents of AI scheming.
A useful retrospective from one of the developers who participated in the METR study that showed a downgrade in developer performance when using AI tools - he suggests this is in part due to unsophisticated use of LLMs and distract themselves from the hard intellectual labor:
Literally any dev can attest to the satisfaction from finally debugging a thorny issue. LLMs are a big dopamine shortcut button that may one-shot your problem. Do you keep pressing the button that has a 1% chance of fixing everything? It's a lot more enjoyable than the grueling alternative, at least to me.
AI will not suddenly lead to an Alzheimer’s Cure: An analysis of all the ways in which health care research has bottlenecks which are not immediately solvable with more intelligence. Great example of how innovation is an o-ring process where many components and parts need to work to get big effects.
Still, I think this underweights what happens in a broad intelligence uplift, where we’ll see many n-th order effects. Your team of medical researchers is now reduced to just an AI, so you have many more people than that are free to be directed and piloted by the AI to do wet-work experiments (because civilization might not have robotics scaled at that point), you see more human trials run more effectively because medical labs are better at statistical design and recruitment of participants, etc. etc.
#good-content
The Tainted Cup: An excellent fantasy novel, drawing on a Sherlock and Watson dynamic set in a bio-punk medieval landscape. One of my favorite fiction books I’ve read this year.
Or maybe this is just what getting older feels like.