🔍 Read the full analysis: Are Engineers Measuring AI The Wrong Way? Amazon’s Bets Offer A Clue on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Amazon published engineering guidance on Oct. 27 warning that token counts and AI adoption can be misleading goals when they are treated as measures of performance. The memo’s focus on customer and business outcomes comes as Amazon’s Nova models have been scaled back or retired and its Anthropic investment has risen sharply in value on paper.
Amazon’s senior engineering leadership told software engineers on Oct. 27 not to treat AI usage, token counts or code volume as measures of success, arguing that teams should be judged by results for customers, the business and colleagues. The guidance lands as the company’s own Nova model lineup has been scaled back and its investment in Anthropic has gained substantial reported value, illustrating why activity metrics may not capture the outcome of an AI bet.
The memo, published on Amazon’s website, introduces a revised role guideline for software engineers. It was written by the company’s most senior engineers and presented by Terk Terkowitz, a vice president and distinguished engineer in Amazon’s stores business. The document says long-term outcomes are what matter, while acknowledging that they can be slow and difficult to measure.
Some teams have used inputs such as token usage, AI tool adoption, code changes and reviews as stand-ins. The memo calls these “deeply flawed metrics” when used as targets, warning they can increase costs and complexity. It does not say to discard the measures: it says they currently “correlate well” with important outputs and can help leaders identify where closer examination is needed.
The guidance directs managers to look at both low and high AI adoption. Teams near the 10th percentile may reveal reasons tools are not useful or are difficult to adopt; teams near the 90th percentile should be checked to see whether heavier use is producing better work. Engineers are also asked to record cases where AI does not help, while leaders should use the tools on their own production work and explain where they do and do not add value.
Six months at Amazon
From token leaderboards to an outcomes memo
Amazon spent 2026 learning the same lesson twice: once with how its engineers use AI, once with its own AI models.
- May 2026Token leaderboards backfireFinancial Times (via The Decoder): staff give an internal agent tool pointless tasks to inflate token counts. Reported, anonymous sources.
- Late May 2026KiroRank “deprecated”Amazon spokesperson to Business Insider. SVP Dave Treadwell: “Please don’t use AI just for the sake of using AI.”
- 24 Jul 2026San Francisco AGI Lab site closesConfirmed by Amazon to GeekWire. Frontier model research continues under Pieter Abbeel.
- 28 Jul 2026Most flagship Nova models wound downBusiness Insider, people familiar: Premier, Omni, Reel, Canvas to “keep the lights on”. Amazon: models “remain one of the most important things we’re working on”.
- 14 / 30 Sep 2026Nova Sonic v1, Nova Reel, Nova Canvas reach end of lifeAWS Bedrock model lifecycle page. Requests to them now fail.
- 26 Oct 2026Anthropic files its S-1Amazon notes: $6.2 billion principal, 12% ownership cap that Amazon can waive. Anthropic commits at least $110 billion to Amazon through April 2036.
- 27 Oct 2026The memo: outcomes, not tokenmaxxingUsage and throughput metrics are “deeply flawed” as targets. “Tokenmaxxing is at odds with frugality.”
- 29 Oct 2026Amazon’s third-quarter resultsAnnounced by Amazon. Not yet published when this article went live.
Sources: Amazon engineering memo (27 Oct 2026); Business Insider (29 May and 28 Jul 2026); GeekWire (24 Jul 2026); The Decoder (12 May 2026, summarising the Financial Times); AWS Bedrock model lifecycle (read 28 Oct 2026); Anthropic S-1 (26 Oct 2026).
Powered by Thorsten Meyer AI
Why Usage Is Not Performance
The memo addresses a measurement problem facing companies adopting AI: more activity is not automatically more impact. Counting tokens or tool use is easy, but those counts do not establish whether customers were better served, work improved or costs fell. If an input becomes a target, employees may have reason to increase it without improving the result.
Amazon’s own guidance takes a more qualified position than a blanket rejection of AI metrics. It says adoption measures can help identify patterns, while cautioning against using them to score people. That distinction matters to engineers and managers whose performance systems may reward visible use rather than useful work. The memo also leaves room for teams to conclude that current tools do not help particular tasks.
The company’s model decisions provide a separate, imperfect illustration of the outcome question. Its Nova products and Anthropic investment are not directly comparable: one is an internal model program and the other a stake in an external company. Still, their contrasting trajectories underline why usage, investment and product results require different scorecards.
Products Worth Considering
Logitech MX Vertical Wireless Ergonomic Mouse – Graphite
Apple MacBook Air 15-inch Laptop with M5 Chip, 16GB Memory, 512GB SSD, and Liquid Retina Display
LG UltraGear 27G640A-B 27-Inch QHD IPS Gaming Monitor, 300Hz, 1ms, G-SYNC Compatible, FreeSync Premium, DisplayHDR 400
As an affiliate, we earn on qualifying purchases.
Nova and Anthropic Bets
Amazon developed its Nova family of AI models and sells models through Amazon Web Services. Business Insider reported in July that most flagship Nova models had been moved to a “keep the lights on” status. AWS lifecycle information lists Nova Canvas, Nova Reel and the first Nova Sonic as retired. The available source material does not establish that every Nova model or service has been discontinued.
Amazon has also invested in Anthropic, the maker of Claude, which the company does not control. The source reports that Amazon put about $18 billion into Anthropic and that an Amazon filing valued its holdings at about $190 billion on June 30. Those figures are a reported investment amount and a paper valuation, not a realized cash return. The source also says gains tied to the stake in the second quarter exceeded Amazon’s operating income for that period.
The new memo follows earlier scrutiny of how Amazon staff measured AI use. The Financial Times reported in May that some employees assigned unnecessary jobs to an internal agent tool to raise token counts. Those accounts were anonymous and are not independently confirmed by the memo. At the end of May, Amazon confirmed to Business Insider that an employee-created leaderboard called KiroRank had been deprecated, saying it was informal and not intended to encourage AI use for its own sake.
““Tokenmaxxing is at odds with frugality.””
— Amazon engineering memo, published Oct. 27
What the Metrics Cannot Show
The memo does not publish data showing how much AI use has improved engineering output, reduced costs or affected customer outcomes across Amazon. It says adoption metrics correlate with outputs “currently,” but does not specify the underlying measures, the size of the correlation or how it varies across teams. The guidance also does not describe a replacement numerical score or a company-wide method for evaluating AI’s effect.
The source’s account of token-inflating behavior relies on anonymous reporting; Amazon’s confirmation that KiroRank was deprecated does not independently confirm every reported use of the tool. The reported $190 billion Anthropic holding is a paper valuation as of June 30, not proof that Amazon has sold its stake or realized that amount. The long-term status and performance of individual Nova products also remain distinct from the broader model program.
Amazon’s Results Due Oct. 29
Amazon is scheduled to report its third-quarter results on Oct. 29, one day after the memo’s publication. Those results may provide updated financial information, but the source does not establish that Amazon will disclose engineering productivity measures or a new framework for tracking AI outcomes.
For now, the published guideline sets out the company’s stated approach: use adoption data to spot questions, examine both heavy and light users, and judge work by its effects rather than by tool volume. Whether managers apply that approach consistently—and whether Amazon publishes evidence linking AI use to customer or business results—remains to be seen.
Key Questions
What did Amazon tell its engineers about AI metrics?
Amazon’s Oct. 27 guidance says token counts, AI adoption, code changes and reviews can be misleading when used as performance targets. It says teams should focus on customer, business and team outcomes, while using input measures to identify areas for investigation.
Is Amazon telling engineers to stop using AI?
No. The memo does not tell staff to stop using AI. It says adoption metrics currently correlate with important outputs, but warns against treating the amount of use as success by itself. It also says engineers should document when current tools do not help their work.
What happened to Amazon’s Nova models?
Business Insider reported that most flagship Nova models were moved to “keep the lights on” status. AWS lifecycle information lists Nova Canvas, Nova Reel and the first Nova Sonic as retired. The available information does not say that every Nova model or service has ended.
How much is Amazon’s Anthropic stake worth?
The source reports that an Amazon filing valued its Anthropic holdings at about $190 billion on June 30, after an investment reported at about $18 billion. The valuation is on paper; it is not the same as a realized cash return.
When will Amazon report its next results?
Amazon is scheduled to report its third-quarter results on Oct. 29, 2026. It is not yet clear whether the report will include details about AI-related engineering productivity or the company’s approach to measuring it.
Source: ThorstenMeyerAI.com
