
Anthropic quietly said last week that its technology has breached the “threshold” after which precautions must be taken related to recursive self-improvement, the theoretical point at which AI technology can improve itself.
Does this mean the singularity has unceremoniously occurred in some Anthropic office? Probably not. And I’m sorry to say that if you want to understand what it does mean, you’re going to have to weed through some varsity-level Anthropic handwringing. I sincerely apologize for how granular this is about to get.
In the November 2 version of Anthropic’s form S-1 filed with the SEC, the company said it “may imminently meet” the criteria its Responsible Scaling policy lays out for something called “dramatically accelerated progress attributable to AI.” Another section says one of the things its Responsible Scaling Policy does is set out the company’s “plans to monitor and mitigate the risks as we approach what has been termed ‘recursive self-improvement.’”
In the November 10 version of the same document, the company has removed the phrase “dramatically accelerated progress attributable to AI,” but seems to be saying that the relevant section of its Responsible Scaling Policy have kicked in. The exact words in the document (bolding added by Gizmodo) are:
“[…]our use of AI systems has already been accelerating our research and development, and our internal estimates indicate that we may experience a further increase in the rate of progress in AI capabilities. Because the AI research and development capability threshold under our Responsible Scaling Policy applies where we ‘observe or expect’ a specified degree of accelerated progress, we made a determination that we have met this threshold based on what we expect from future models and our overall rate of progress, rather than on what we have observed from current models.”
The reader is left to infer that the relevant section of the Responsible Scaling Policy is the one about “dramatically accelerated progress attributable to AI.” Indeed, the location of the “observe or expect” language the new S-1 mentions in the policy document is from a section on “dramatically accelerated” progress, specifying a doubling of the baseline rate of improvemnt.
What’s being announced here does not seem to be any sort of sudden technological advancement but a revised determination, and one being revised based on speculation. Anthropic says it must consider the threshold crossed “based on what [it] expect[s] from future models.”
At any rate, mitigations are deemed necessary once this threshold is crossed, according to the Responsible Scaling Policy document. If Anthropic’s tech really is where it says it is, the company must now explore “ways to achieve unprecedented levels of security,” reach a state of total awareness of what its models are doing called “eyes on everything,” and learn to red team itself so well that bug bounties become redundant.
Anthropic’s IPO is so imminent that the company has now published the relevant documents on its website. The founders’ letter in that document cache includes a section relevant to self-imporovement: “Building AI safely may require Anthropic (and ideally the rest of the industry) to modulate the extremely fast pace at which AI capabilities are compounding, especially when this is driven by AI systems building themselves,” the letter says.
Anthropic says shares of its company are expected to start being publicly traded on Thursday.

