Mateo Fowler
Making AI great for creatives at Krea
A black cut-paper crystal ball with a crimson star, suggesting a glimpse of the future

AI Safety Predictions

I write this down to hold myself accountable:

I see the future as roughly capabilities <> safety cooperation to save us from the (default) path of the extinction of the human race.

I envision government-enforced safety cases by mid 2027 after the chain of incidents that will happen from the present onwards. Capabilities will slow down to work on safety because safety will be seen as a product, at least for some time; egregiously misaligned models will have worse UX. This doesn’t scale to RSI, and by March 2028 labs will have the capacity to autonomously loop towards the next model. Anthropic will arrive first. Private coordination will be needed and used to avoid RSI until alignment is solved. Public awareness will be high, which will make this the topic of the elections. Trump will promise a deal with China only if they give him the peace Nobel Prize. Leaders and citizens will be convinced of the risks of not handing off lots of power to possibly-misaligned models and by efficient policies and (maybe) public sabotage the important 10 leaders that have to agree to slow down will do so.

Safety will rise in status from now onwards, steadily until 2028, then exponentially, as it will be the blocker for beating Death.

The biggest blockers to slowing down so safety catches up are sensible, precise ways of verifying misalignment before deployment, quickly enough for capabilities people to integrate this and efficient policy and public awareness. METR and similar organizations will shine on the former and any policy and strategy ones on the latter. Agent foundations will (finally) have their showtime during the 2030s.

This is not the default path. Is the path that safety people will have to force into existence if we want to buy time for ourselves.

I believe power seeking institutions most commonly associated with EAs and not Rationalists will be the responsible of this slowdown and its yet another argument on why obtaining influence it’s instrumental for safety

  1. Geoffrey Irving, 80,000 Hours Podcast #251, 34:16 · “less than 10 people”
  2. i believe (maybe wrongly) that most lab leaders are actually concerned about safety, it’s just that risks aren’t so visceral today and profits outweigh them