I've been trying to learn as much as I can about the assorted iterations of AI, and I happened to run across a video by "El". As I'm always a bit skeptical, particularly where AI is concerned, I researched her background and it appears she is indeed a real person with strong credentials: a PhD in Computer Science working as a senior data scientist. She describes herself as a systems thinker and geopolitical analyst living in the UK with an emphasis on EU countries.
https://www.youtube.com/watch?v=Jbh8QteVM5g&t=934s
El reported on several cases when organizations make one fundamental mistake: employing artificial intelligence to entirely replace human judgment, not augment it. AI is great at "what", terrible at "why."
The first case study is a huge Pizza Hut franchisee who sued its parent firm, Yum Brands, for $100 million. The franchise owner had a 90% on-time delivery rate, but wanted to get even better. He said a mandated AI delivery management system dubbed “Dragon Tail” basically ruined their business. The AI technology didn’t optimize logistics as designed, but instead provided delivery drivers with real-time insight into kitchen timing and tip levels, i.e. gave them perfect up-to-date information. That produced an unintended incentive dilemma. Independent Door Dash workers started manipulating the system, waiting to batch many deliveries together or cherry-picking just high-tip orders. The results were a disaster. Averages for pizza delivery time fell from 90% on-time delivery to only 50%. The franchise became a money loser, the business connection fell apart entirely and customers were unhappy. The AI did exactly what it was intended to; the problem was in implementation that delivered unintended consequences.
Her second example is the Swedish finance business Klarna. The firm, in a cost-cutting move, terminated about 700 customer service jobs and replaced them with a chatbot powered by OpenAI. The chatbot did a good job of handling large numbers of ordinary encounters at first. But it floundered horribly in complex, emotional client scenarios – the kind that call for empathy, nuance and imaginative problem-solving. Klarna’s 2025 IPO was just months away and the business was obliged to make an expensive public reversal and re-hire human labor.
The real problem is not the technology. What’s interesting is that the AI systems themselves didn’t actually malfunction or break. These challenges were the result of basic mistakes in business and system design. Firms hurried to implement AI without sufficient parallel testing, redundancy or understanding of how people behave in the real world. They expected automation would just do better, without any consideration for how people actually operate when given new tools and new information.
According to a 2026 Forrester survey, 55% of companies today regret having laid off workers to embrace AI.[1] Gartner estimates that in the end 50% of the organizations who eliminate their support workers will have to bring them back.These are not isolated events. They are signs of a systemic problem with the way enterprises approach AI integration. Successful businesses go about it differently. So they have AI do the normal computations and repetitive duties while keeping humans in the loop for the localized judgment calls at the end. It is a tool that enhances human potential, not one that attempts to replace it wholesale.
El closes with a grim message. The reckless use of AI could not only hurt individual firm bottom lines but also provoke a significant public backlash. This could result in stringent rules that impede progress across the whole AI ecosystem. Similarly, high-profile AI failures could delay acceptance of helpful technologies before they’ve had a chance to prove their value. AI should augment human understanding, not replace it. Those organizations who get this will prosper. Those that don't become cautionary tales themselves.As we go into a more automated future, the firms that will win are the ones who see AI as a powerful assistant, not an independent replacement for human judgment, creativity and empathy.
I note that recently Amazon suffered substantial down-time and major data loss from a requirement that their programmers use AI for 80% of their work. They learned the hard way that forcing AI tools on developers can backfire spectacularly. Over a chaotic three-month span, the e-commerce giant hit a wall with four massive AI-driven outages. A brutal six-hour crash on March 5th triggered by Amazon's own AI coding tool wiped out 6.3 million orders in North America alone. Apparently, the Agentic AI tool thought it could make the code better, deleted existing systems and implemented its own rewritten code.
In December 2025. Amazon’s AI tool, Kiro, went rogue and completely deleted a major AWS Cost Explorer environment, knocking out services in China for 13 hours. Shortly after, Amazon Q Developer joined the chaos, glitching delivery times and eating another 120,000 orders. This forced Senior Vice President Dave Treadwell—who had just mandated Kiro a month prior with a required 80% adoption goal—to hit the brakes. He’s now ordered a massive 90-day safety reset across 335 revenue-critical systems.
The problem was the agentic nature of the AI software. It was making decisions without any human review. While leadership bragged about deploying 21,000 AI agents to save $2 billion and boost developer velocity by 4.5x, 1,500 engineers had already protested the mandate, begging to use Claude Code instead. Now, Amazon is shifting to a hybrid playbook: AI writes, deterministic rules double-check, and humans sign off. It’s a reality check that aligns perfectly with Gartner’s prediction that 40% of agentic AI projects will tank by 2027 due to runaway costs and poor risk controls. [2]
Of course, Amazon blamed all the problems on "human error." True, but not in the way they intended. The error was with the senior vice-president doofus who required implementation of Kiro, not the coders who argued against it. AI codes really, really fast and Agentic AI makes decisions based on data, but can't understand the "why" or the unintended consequences from unintended human reactions. Then again, McNamara had the same problem.
[1]https://hrexecutive.com/the-ai-layoff-trap-why-half-will-be-quietly-rehired/
No comments:
Post a Comment