November 2024 BenGoldhaber.com Newsletter
Cautious optimism about the future and a longing for the Shogunate
The end of the year approaches! December’s a nice month because you can feel things start to slow down - people are more reflective, they stop responding to your work emails, and you get to spend time with your family to try to fix their IT issues.
Historically, I’ve recommended things like the 8,760 Hours guide for conducting year end reviews, but technology marches onwards - here’s a prompt for working with Claude to set some 2025 goals:
Act as a supportive year-end reflection coach. Help me honestly reflect on my past year’s achievements and lessons learned, then guide me to identify, stack-rank, and refine my top goals for the coming year. Conduct a pre-mortem on each goal to anticipate challenges, and help me create a clear, realistic action plan that positions me for an exceptional, 10/10 year.
#links
Good general advice:
Anything that hasn’t been a hell yes has been a mistake. Directionally correct, though likely not entirely true (proof by counterexample: Leroy Jenkins).
Often the difficulty in writing well is a failure to anticipate that it should take a long time; by up-and-comer Jeff Bezos.
My favorite essay of the month was from Michael Nielsen, on how to be a wise optimist about science and technology.
From 2011 through 2015 much of my work focused on artificial intelligence. But in the second half of the 2010s I began doubting the wisdom of such work, despite the enormous creative opportunity. For alongside AI's promise is a vast and, to me, all-too-plausible downside: that the genie of science and technology, having granted humanity many mostly wonderful wishes, may grant us the traditional deadly final wish.
I think one reason Michael’s writing so resonates with me is it describes my own journey in thinking about the world and the type of impact I want to have. I fondly look back on 22 year old Ben and the *certainty* with which he believed that markets + technology meant we would all live forever in perfect happiness, and the only pre-requisite was we smash the state and unchain progress.1
Alas, everything feels more complex now, and Michael’s essay captures that tension while still embracing a positive worldview, determined to orient and work towards towards hopeful scenarios. A few selected quotes:
If you go to certain parties in San Francisco, you meet hundreds (and eventually thousands) of people obsessed by AGI. They talk about it constantly, entertain theories about it, start companies working toward it, and generally orient their lives around it. For some of those people it is one of the most important entities in their lives. And what makes this striking is that AGI doesn't exist. They are organizing their lives around an entity which is currently imaginary; it merely exists as a shared social reality. The most interesting hyper-entities often require both tremendous design imagination and tremendous depth of scientific understanding to conceive. But once they've been imagined, people can become invested in bringing them into existence. Crucially, they can become shared visions. That makes hyper-entities important co-ordination mechanisms
While I don't know how to imagine good futures, I do know that hyper-entities are a piece of the future we have experience in supplying. They're sufficiently narrow that they can be reasoned about in detail, and can be the focus of large-scale co-ordination effort.
What, then, of the question of the title: how to be a wise optimist about science and technology? My condensed, provisional, personal answers: Do work on non-market safety.
If xrisk is real, then denying xrisk is a foolish optimism, like someone responding to their cancer diagnosis with denial and a failure to act. Rather, the foundation for a wise optimism is to accept the xrisk exists, and then to ask: "how can we actively and imaginatively respond?"
Crux question; do we aim for the precautionary world or the posthuman world? In the precautionary world, we slow down radically, as a way of limiting destabilizing technologies such as ASI, synthetic biology, and nuclear weapons, while enabling liberal values, abundance, and creative possibility to flourish as best possible. In the posthuman world we aspire toward a loving plurality of posthumanities (humans included!). Both paths have major dangers. Both paths will require tremendous imagination for success.
#Prediction Markets
From Prediction Markets to Info Finance: Prediction markets have gotten a lot of positive press this past year with the rise of Polymarket, and Vitalik describes next steps in deploying market mechanisms for decision making.
every time you need to make a decision, you set up a prediction market on what outcome the costly mechanism would make on the decision if it was called. You let the prediction market run, and put in a small amount of money to subsidize market makers. This gives you a credibly neutral fast and cheap "distilled version" of your original highly trustworthy but highly costly mechanism.
The key is the balance between market and non-market: the market is the "engine", and some other non-financialized trustworthy mechanism is the "steering wheel".
It looks like this experiment will be run on a new blockchain prediction platform Futarchy
I’m excited about this because AI has already gotten very good at making predictions, and you can deploy AI predictors much much more cheaply than you can good human forecasters. So, if you’ve got a thousand boring questions - “How quickly can my team achieve SOC2 compliance if we blah blah blah” you might actually get good predictions on this with AI, while human only markets would never achieve meaningful trade volume.
Related: The French whale who bet millions on Trump to win on polymarket commissioned his own polls using the neighbor method. SSC remains skeptical that the ‘true odds’ weren’t closer to 50-50.
#AI
Lots of reports this month of people getting “one shot” by the new Claude; aka feeling a deep friendliness and affection for the new model that makes you want to celebrate our new AI overlords. While it’s not yet my best friend, the new model is… in ways I can’t quite put my finger on... way better. Over the past few weeks I’ve asked it for help writing, on thinking through hard emotional problems, and on how to best carve a turkey, just to name a few.
The new custom styles feature (example) makes the writing style much more my tone. Note: This email is still written by me, because on certain tasks I guess I’m a luddite.
Davidad suggests some of us should avoid becoming infected with love for Claude; it’s too late for me, save yourself.
Google Project Zero Automated Zero day discovery of exploit in SQLite, a very widely deployed database open source engine. Evidence that the thousands of developers working in open source have in fact not caught all the bugs that a trained language model might.
Three Sketches of ASL-4 Safety Case Components: Anthropic blog post sketching out what safety cases for powerful AI capabilities might look like.
Some of these commitments take the form of affirmative safety cases, which are structured arguments that the system is safe to deploy in a given environment. Unfortunately, it is not yet obvious how to make a safety case to rule out certain threats that arise once AIs have sophisticated strategic abilities. The goal of this post is to present some candidates for what such a safety case might look like.
Note: I think safety cases is one of the best frames for thinking about ‘AI Safety’ in the current commercial regime of widespread AI tool deployment. The central argument in AI Safety, in my opinion, is the degree to which the risk landscape contains a few predictable risks which we can anticipate now and build safety cases around, or whether more powerful intelligences will bring novel risks that we can’t build safety cases for/recover from.
Deepseek, a Chinese LLM company, released their latest model which uses chain of thought similar to the best publicly released OpenAI model and achieves similar if not better benchmarks.
Agent Based Modeling advances:
Simulate millions of agents, grounded in the census, fast and on cheap commodity hardware
I worked at HASH, an ABM company. We got early access to GPT 2, and experimented with it for codegen, but generating realistic agents is a more practical future direction. I do think it was a miss we didn’t focus more on generating agents that use LLMs inspired traits. Related - experimenting with a multi-agent civilization.
METR releases paper estimating how well AI can automate R&D:
The best Claude and o1-preview agents do substantially better than humans given 2 hours, but human experts improve performance at a much quicker rate over time.
Gwern weighs in on the unclear value of creating a new Manhattan project and asks important questions about what happens even if you win an arms race:
I predicted two years ago we’d see large sectors disrupted by AI; so far I think I’ve missed on that prediction, but here’s a report of local news graphics editors as a profession being disemployed by generative AI (note: I think this is probably too small to count for what I was imagining, but it could be indicative of other similar labor segments)
OpenAI Email Archives from Musk v. Altman: takeaway - power players are good at writing clear direct emails.
#good-content
Shogun: So good - a beautiful, artfully plotted season of television about the feudal Japan. Like early game of thrones level good, paired with an appreciation for my-god just how violent and harsh human history has been.
Arcane Season 2: Also excellent - not quite Season 1 Arcane good, but a satisfactory conclusion of the plot.
xoxo,
Ben
to be clear, this still 100% applies to the FDA. FDA Delenda Est.