Everyone was confident about where AI was going, which was impressive because despite being in the top 99.9Xth percentile of earliest accounts created on the OpenAI platform, I was still testing and discovering the frontier of ChatGPT’s capabilities. (At the time it was mediocre wedding toasts and confident-sounding legal citations.)

The topic of trust in AI then came up. Everyone agreed trust was VERY important, which was true, but it was also the moment I realized the word trustworthy in the context of AI is not useful. As the conversation continued, it became clear that people were not actually talking about the same thing. Some people meant transparency. Could we understand how the AI reached its answer? Others meant accuracy. Did it get the answer right? Others meant calibration. When the AI sounded confident, was that confidence deserved? Still others meant alignment. Did the system behave according to human goals, or, as one person wondered, were the robots coming for us all?

Everyone was talking past each other, and all of them were using the word trustworthy. That was when I realized “trust” had become a suitcase word, and we needed to stop using the word trust on our team.

Subscribe now

Suitcase Words

Marvin Minsky, one of the foundational figures in artificial intelligence, used the term “suitcase word” for words that carry many meanings packed inside them. In his book The Emotion Machine, from all the way back in 2006, Minsky used the idea for overloaded concepts, including:

• consciousness

• emotion

• intelligence

• etc.

These are words that sound like they have a singular definition, but every person packs their own meanings, processes, assumptions, and arguments inside them. The suitcase can then be used to bludgeon your ideological opponents.

One of my favorite descriptions of this phenomenon is from a book I read when I was 12 years old:

“When I use a word,” Humpty Dumpty said, in rather a scornful tone, “it means just what I choose it to mean—neither more nor less.”

“The question is,” said Alice, “whether you can make words mean so many different things.”

“The question is,” said Humpty Dumpty, “which is to be master—that’s all.”

— Through the Looking-Glass, and What Alice Found There, by Lewis Carroll

In the roundtable meeting, trust sounded like a “very important” and “shared concern,” but it was really a suitcase. Everyone had packed their own meaning inside it, zipped it shut, and started debating assuming everyone was holding the same bag.

Trust is a tricky concept. A system can be accurate but poorly calibrated. It can be transparent but inaccurate. It can be aligned with company policy but miserable for users. What are the conditions for “trustworthy” and are they dependent on the context?

This is why we use the TACA framework of trust: Transparency, Accuracy, Calibration, and Alignment. Once you unpack trust into TACA, meetings become a little less inspirational and a lot more useful, which is a trade-off I am personally willing to make.

TACA Replaces Trust

Trust is easier to achieve once you unpack the suitcase

TACA separates the internal dimensions of AI trust into four components:

• Transparency: Can users sufficiently understand how the system reached its answer?

• Accuracy: Does the system produce correct or useful outputs according to the objective?

• Calibration: Does the system’s expressed confidence match how likely it is to be right?

• Alignment: Does the system behave according to the goals, constraints, and values of the people who create, use, or are affected by it?

Improving one can leave another unchanged, and in some cases improving one can make another worse.

A typical AI trust conversation can sound something like this:

• Product Manager: “Users don’t trust the AI system. We’re losing clients.”

• AI Engineer: “The system is 99% accurate. It’s better than it has ever been.”

• Product Manager: “We’re still losing customers. The system isn’t trustworthy.”

• AI Engineer: “But look at the data! They’re all up and to the right!”

The engineer may be right that accuracy improved. The product manager may be right that they’re losing users. This combination is possible because the two of them are not talking about the same component of trust.

The PM may be talking about calibration and the system might give high-confidence answers in situations where uncertainty would be more appropriate. Or the PM may be talking about alignment and the system might be optimizing for a company policy that technically makes sense but feels adversarial to users. Or maybe the problem is transparency and users may not understand where an answer came from, so they refuse to rely on it even when it is correct.

Now compare that to the same conversation using TACA:

• Product Manager: “The AI isn’t sufficiently calibrated for our users. It expresses high confidence on complex, high-stakes topics where users should be cautious.”

• AI Engineer: “That makes sense. The latest model improved accuracy, but calibration may have dropped. We can test confidence language, escalation thresholds, and uncertainty signaling.”

The team can now investigate a specific failure mode and decide whether the use case requires better calibration, where the system should hedge, when it should escalate, and whether a small reduction in raw accuracy is worth a better user experience.

In other words, TACA turns the argument from “is it trustworthy?” into “which part of trust is failing, how much does that part matter here, and what are we willing to trade to improve it?”

Although this post won’t cover the full depth of TACA (consult our book), below are a few key attributes to keep in mind for each.

Transparency – Can users see enough to judge the answer?

Transparency, along with any other element of TACA, may not be required for your use case. If your users take a picture of a bird and the AI correctly determines its species 99% of the time, do you or your users truly care how it came to the answer?

In other contexts, transparency is essential. This is especially the case when the stakes are high, the consequences are public, or the environment is regulated. If an AI system recommends a hiring decision, medical action, insurance denial, legal interpretation, or significant financial move, “the model said so” is not going to carry the room, nor should it. A transparent system gives stakeholders something to inspect, challenge, verify, and improve.

This is also where many teams confuse explanation with performance. A clear explanation does not make an answer correct. Transparency is not trust by itself, but it does provide visibility needed to evaluate whether trust is warranted.

Accuracy – Did it get the right answer, and what does “right” mean?

Accuracy sounds like the simplest part of trust. Did the model get the answer right? Wonderful. Put it on the dashboard and give it a nice green color.

Unfortunately, accuracy is only simple when the cost of every error is the same. “The model is 99% accurate” is comforting until you learn after the fact that in the remaining 1% your CFO just bet the company’s future.

Accuracy is the correspondence between the AI system’s outputs and the intended objective, but the intended objective has to be defined. Different users often care about different failures. A model that is highly precise on average but occasionally misses wildly may be preferred by someone managing huge workflows and seeking upside, but it would be rejected out of hand by someone else managing risk.

A system that performs well across a broad benchmark may still be unacceptable if its errors cluster around sensitive, expensive, or legally exposed cases.

A bird-species classifier can often be measured with straightforward correctness: did it identify the bird in the photo? Even then, the answer may depend on whether confusing two common sparrows matters less than missing an endangered species.

The technical term for this is that the loss function does not match the utility function. In plainer language, the metric the team can easily measure is not always the outcome the user actually cares about. This is why teams need to define accuracy around the use case, the user, and the cost of different errors. Otherwise they optimize what is simple to report rather than what actually improves the outcome.

Calibration – Does the system know when it might be wrong?

Calibration is the alignment between a model’s expressed confidence and the actual likelihood that the output is correct. A well-calibrated system that says something is 70% likely should be right about 70% of the time in comparable situations. When it sounds uncertain, uncertainty should be appropriate. When it sounds confident, that confidence should be earned.

This matters because users adjust their reliance based on perceived certainty, whether the system gives a numeric probability or just uses confident language. Humans naturally convert words into rough probabilities. “Possible” feels different from “often.” “Almost always” feels different from “sometimes.” “Certain” feels like it should have brought a lawyer.

My toddler nephew once explained to me with the utmost confidence how income taxes worked. A poorly calibrated AI acts similarly.

One of my favorite papers is Mosteller and Youtz’s 1990 paper “Quantifying Probabilistic Expressions.” People interpret words as probability signals, even when the speaker, or model, is using them loosely. Below is a snapshot of their research showing how people translated expressions to probabilities.

How well does it match your probabilities? Are you a “often = 80%” kind of person or closer to 60%?

That table is useful because it shows how much meaning humans attach to ordinary uncertainty language. If an AI system says it is “certain” but is right 80% of the time, users will experience that as failure even if the underlying accuracy is objectively impressive.

Calibration becomes especially important in high-stakes or high-ambiguity domains. A legal AI assistant that presents a precedent even though the case was overturned years earlier is dangerous in a different way than a system that says, “This appears relevant, but I have low confidence and recommend verification.”

Alignment: Does the system behave according to the right values and constraints?

Alignment is the extent to which an AI system’s behavior reflects the expectations, goals, constraints, and values of the people who create, use, or are affected by it.

This sounds straightforward until you remember stakeholders include users, executives, regulators, lawyers, customers, and at least one person whose explicit stated purpose is “but how can we make this REALLY go viral?”

Alignment is also not the same as giving the user whatever they ask for. A child-facing AI application should not behave like an unrestricted general assistant. A car dealership AI assistant can refuse to help a random user write a browser extension that shuffles cat videos. A medical assistant or financial assistant must obey constraints that go beyond immediate user preference.

Alignment is where the implicit values become explicit. What does the AI prioritize when goals conflict? Does it favor user autonomy, company efficiency, regulatory caution, safety, growth, retention, speed, politeness, or escalation avoidance? Which users are protected? Which risks are tolerated? Which trade-offs are documented, and which ones are accidentally embedded because nobody wanted to have a difficult meeting?

The bias can never be eliminated, but it can be chosen, documented, tested, and managed. In this respect, AI systems resemble the humans who design and deploy them, which is either comforting or upsetting, depending on your experience.

Alignment is ultimately a negotiation between technical performance and multiple sets of human values, and it changes as strategies, markets, laws, and cultures change. A system can be transparent, accurate, and calibrated, yet still rejected if users believe it is serving the wrong master.

Quality is the weighted combination of TACA

Once trust is unpacked, AI quality becomes easier to discuss. Quality is not simply a universal knob that teams can turn from 7 to 8. Quality is the weighted combination of Transparency, Accuracy, Calibration, and Alignment for a specific use case.

Not every AI system needs the same trust, or “TACA,” profile.

A bird classifier may care mostly about accuracy. Users want to know whether the photo shows a Shoebill Stork or a different bird that looks just as judgmental. Transparency may be helpful, but for many users it is secondary. Calibration may matter if the system shows confidence or offers multiple possible species. Alignment matters less, unless the system is being used for conservation, hunting restrictions, or ecological reporting.

Your AI model has failed to impress the Shoebill Stork

In our work we disaggregate the portion of trust that matters. In our day-to-day Accuracy is, as expected, the most common, but the other elements also factor in as needed. For example, one of our company’s values is “partner obsessed,” where we do not take an action that will benefit the company to the detriment of a brand partner. This alignment ethos is also embedded in how AI is designed.

Without TACA, “trustworthy” becomes a suitcase word packed with conflicting meanings. Unpack the suitcase early. Otherwise the team will spend the next six months arguing about trust while carrying different luggage.

To learn more about TACA and other principles to bring AI to your tech and company, read our book Architected Intelligence: Principles for Building AI-First Organizations and Technologies.

Red Lobster provides us with a plausible theory of the next phase of AI economics:

unlimited consumption is a sound strategy right up until the moment finance double checks the unit economics.

This is not an article about shrimp. Mostly.

Subscribe now

For a while, AI has been living through its Endless Shrimp phase. MORE context. MORE tool use. MORE agents. MORE evals. MORE “let’s just use the best model because the stakes are high” (where “the stakes” means a Slack summary that three people will skim during lunch). Always MOOOOORE!!

Consider the AI model releases as four distinct waves:

• The first wave of serious frontier models showed what might be possible, but the pricing and limited intelligence were painful enough that ambitious prototypes remained prototypes.

• The second wave of models (GPT-4, Claude 3 Opus, o1, and especially GPT-4.5) were impressive, expensive, and frequently worth testing because the capabilities were moving faster and total costs still flew under the budget radar.

• The third wave of models produced higher levels of intelligence at a 90% discount or more. This was the moment when the collective freaked out with better models at lower costs. Everyone freaked out, and the narrative became “intelligence would keep getting cheaper until it became too cheap to meter.”

• The fourth and current wave of models are steadily more intelligent, but at an increasing price. Additionally, those third wave models are being deprecated, effectively increasing the price, even as adoption and token usage is rising. Quantity is rising rapidly at the same moment price is increasing.

However, the third wave narrative was slow to die. Finance departments across the world are slowly waking up to the fourth wave of increasingly expensive models at higher consumption.

The above chart is a little busy, but it’s useful because it shows each of the waves. The latest models have been trending more expensive at the frontier. The new Mythos model simply extends the extrapolation above. Smarter and more expensive.

If you want to simply take two points in time, here’s another way of visualizing the jump from the second to the fourth wave:

Tokens Are Not Value

But organizations are looking for ROI on valuable work, and not leaderboard points or token maxxing. For the first time, “dumber” models are more than sufficient to get the job done at very high levels of accuracy, especially now that evals, harnesses, and controls are in place.

This is why it’s so hard to project total spend --

As developer and user AI expertise expands, token usage increases, but more can be accomplished with less expensive models.

The old days (a few months ago):

• Use most expensive model

• “Implement this feature / create this marketing copy / Do the thing!”

• “Make up your own tests and make sure to pass them”

• “Make no mistakes”

The new days:

• Select best model for use case (sometimes far in advance of it actually executing)

• [After five iterative rounds of planning and prototyping with the team alongside an AI that is wholly aware of the environment, codebase, and infrastructure] “Execute this detailed plan.”

• “You must pass the very extensive tests that already exist. Use these repeatable AI skills that we’ve been refining over months. The tests will block you from submitting and you can’t change them, so you’ll need to pass them.”

Hiding underneath the pricing and cost debate is translation to value. The rate per token matters, but cost per job matters more.

The Macro of Computing Story: Prices are Going Up

Remember “Extended Thinking” from Claude’s models? Turn it on and the model will spend longer verifying its results every time.

Those were the days. Now it’s “Extended thinking.” If it seems like it’s worth the trouble AND Anthropic has enough compute, it will spin longer on it. Model degradations were becoming palpable for users.

Somehow Anthropic’s savior for compute became… Elon Musk? Yes, Anthropic was against the wall, and Elon Musk and xAI came to the rescue with Colossus’s compute that is sufficient (for now) to relieve the pressure.

OpenAI, Anthropic, and Google have locked up the spare compute for training (creating new models) and inference (allowing us to use them).

Intelligence will become increasingly expensive as compute, memory, electricity, or any of the other essential pieces of the supply chain of AI compute is bottlenecked. Without another breakthrough in model efficiency, which we haven’t seen the likes of for over a year, demand is skyrocketing and will outpace supply.

The Fixed Cost of Expertise

That is where the industry is headed on a macro basis, but how do companies react?

The first serious AI buildout rewarded experimentation. Teams were trying to discover what these systems could do, where they broke, and whether any of the demos survived real world contact. During that phase, token spend was often a small percentage of the total cost because the expensive part was human labor.

• scope the use case,

• design the workflow,

• build retrieval,

• define evals,

• manage stakeholders (so many stakeholders),

• review outputs,

• iterate with beta users ,

• decide whether the whole thing was ready for production or merely marketing

This is the part where “the fixed-cost argument” misses.

Dwarkesh Patel recently discussed AI compute economics with Dylan Patel in an episode focused on the major bottlenecks to scaling AI:

• compute

• memory

• electricity

I’m a big fan of Dwarkesh, but he misses the mark on fixed cost. Here is his argument:

By the way, there’s this interesting economic effect called Alchian-Allen, which is the idea that if you increase the fixed cost of different goods, one of which is higher quality and one which is lower quality, that will make people choose the higher quality good, on the margin.

To give a specific example, suppose the better-tasting apple costs two dollars and the shittier apple costs one dollar. Now suppose you put an import tariff on them. Now it’s $3 versus $2 for a great apple versus a medium apple.

…

So I wonder if applied to AI that would mean that, if GPUs are going to get more expensive, there will be a fixed cost increase in the price of compute. As a result, that will push people to be willing to pay higher margins for slightly better models. Because the calculus is, I’m going to be paying all this money for the compute anyway. I might as well just pay slightly more to make sure it’s the very best model rather than a model that’s slightly worse.

The limitation in Dwarkesh’s argument is that it comes mostly from the lab side and does not consider the fixed costs of those deploying. As expertise increases, the fixed costs associated with launching a product continually decreases. The missing piece is:

Output quality ≠ Model quality

For many companies, the larger fixed cost was the upfront labor required to deploy the system and decide whether the project should exist at all. The model call might have been the cheap part compared with the weeks or months of engineering, evaluation, process design, stakeholder alignment, and the many iterations with users required to make the system useful.

Intelligence is improving, model costs are going up, but teams are getting better at building than ever. (On a related note of increasing AI expertise, you can pick up a copy of our book, Architected Intelligence, today!)

Token Costs as Substitutes for SaaS Subscriptions, Wages, and Contractor Costs

This is also the moment when the Eye of Sauron of finance departments turns toward token spend. Tying each token’s cost to a project will soon become not just normal but mandatory.

At first, finance mostly sees AI as a strategic initiative, an innovation budget, or an executive priority with enough glow and “pizzazz” around it to discourage normal scrutiny. Eventually, AI costs become comparable to, eventually exceeds, some of the major line-item costs of SaaS, vendors, wages, infra, etc. Once that threshold is crossed, a new era arrives.

AI can act as a substitute for SaaS, contractors, vendors, and wages. Once an organization realizes AI costs are BOTH a substitute for other costs and a major budgetary line item, incentives will quickly adapt.

In the new world, a department receives a budget and is then responsible for deploying it however it sees fit. This new realm of responsibility can be both empowering and terrifying. A marketing team will have more autonomy than ever, but accountability will come bundled with it.

With increasing AI expertise, the ROI calculation will become increasingly explicit.

Departments will start asking whether smaller models can do the job. They will ask whether frontier models should be reserved for escalation, review, or high-impact decisions. They will ask whether the entire policy library needs to be injected into context every time someone asks a question that could have been answered by three paragraphs and a citation link. They will ask why an agent used seven tool calls to check whether Tuesday comes after Monday.

This is new, but at least it feels more sustainable than token-maxxing where token volume is the proxy metric for “seriousness around AI adoption.”

So, it’s a race:

• AI adoption continues to rapidly increase and permeate more organizations and departments, increasing demand

• Compute capacity is increasing, but not fast enough, with demand outpacing supply

• AI expertise is increasing, allowing for more selective ROI-based efficiency

While it’s true that “token maxxing endless shrimp” is ending and costs are continuing to increase, with intention and expertise, you can continue to accelerate.

Subscribe now

The original Star Wars trilogy provides us with a plausible theory of the AI cybersecurity endgame:

offense wins, and everyone responds by making technology worse on purpose

That sounds backwards at first. Star Wars has faster-than-light travel, planet-destroying weapons, sentient droids, holograms, and, most importantly, laser swords. It also has astonishingly bad interoperability. Droids physically plug into wall terminals, sensitive Jedi archives require physical retrieval, and critical infrastructure (the center of the plot!) is protected by shields, then blast doors, and then an isolated generator.

Our world, in many ways, is more connected than Star Wars. My watch talks to my phone. My phone talks to my car. My car wants to talk to the cloud for… reasons(?) I do not fully trust. More connection = more technological progress. But, maybe, that only works when connection is mostly safe.

In Star Wars, R2-D2 rolls up to a Death Star terminal, plugs in, chirps for a few seconds, and appears to gain administrative access to the Empire’s most sensitive systems. He bypassed those Imperial CAPTCHAs no problem. Just one brave little astromech and a maintenance port on “dangerously skip permissions mode” is all that’s needed.

Did the Empire build a moon-sized battle station and forget to secure the USB port? Or does the cyber world start to look like Star Wars?

Just like Lord Vader, we also recommend purchasing our book Architected Intelligence, today!

That is the actual issue underneath AI and cybersecurity. We should not ask whether AI helps attackers or defenders. It obviously helps both. The useful question is which side compounds faster.

• More AI means much more code.

• More AI means more cybersecurity.

• More AI means more cybercrime.

So which side wins in the cyber security offense vs. defense battle once AI is added?

(Cyber Crime + Code Quantity) vs. Security

A different offense vs. defense perspective is to examine the “Hacker ROI” of a breach:

Probability of Breach * Expected Value of Breach > Cost of Attack

The optimistic story is AI helps defenders review code, find vulnerabilities, monitor logs, triage alerts, write patches, summarize incidents, and automate responses. A security team with good AI support can move faster and cover more ground. Modern environments are too large, too messy, and too interconnected for humans to inspect by hand.

The pessimistic story is just as easy to see. AI helps attackers generate better phishing, personalize scams, scan more systems, chain together tools, summarize documentation, write malware variants, test hypotheses, and TIRELESSLY keep trying. Cybersecurity is a race between two optimization processes.

Modern AI agents are becoming more R2-like and can keep going longer with less supervision. OpenAI’s just-released GPT-5.5 reached 78.7% on OSWorld-Verified, a benchmark for operating real computer environments. It’s better at using tools, checking its work, and persisting through complex tasks.

Cybersecurity can be like a spacesuit, where a tiny puncture can become catastrophic because the environment outside is hostile. A company can have a solid security team, reasonable policies, and sound execution, but the attack can come from every seam.

Like: … phishing email, fake vendor invoice, compromised SaaS account, stolen session token, malicious browser extension, leaked GitHub secret, over-permissioned service account, vulnerable dependency, package typo-squat, forgotten staging environment, old VPN appliance, contractor laptops, shadow IT tools, “help desk” resets, exposed API keys, “helpful” AI agents with malicious instructions…

After the 1984 IRA bombing of the United Kingdom’s Conservative Party conference in Brighton—which targeted, but failed, to take out the Prime Minister—the IRA issued a statement saying, “Today we were unlucky, but remember we only have to be lucky once. You will have to be lucky always.”

The attacker does not need to understand your company. The attacker needs to “get lucky” on one seam.

Historically, a lot of practical cybersecurity depended on friction to raise the cost of hacking to win the ROI equation. If your security was annoying enough for attackers, you could deter them.

It is the old bear joke. Two people are running from a bear. One stops to put on running shoes. The other says, “You’ll never outrun the bear.” The first says, “I don’t have to outrun the bear. I only have to outrun you.”

“Make yourself harder to attack than the next company” used to be sound advice, but AI changes the cost curve.

Why Is Airgap The End Point of Offense Winning?

Security typically falls along a few dimensions. You gain access to a system because of:

• Something you know (a password or pattern)

• Something you have (a phone, hardware token, keycard, or private key)

• Something you are (a fingerprint, face, or voice)

• Somewhere you’re located, like a trusted device, secure facility, local terminal, office network, or controlled region.

If AI offense obliterates the first three through hacks, social engineering, and other methods, that leaves us with the fourth—our location.

Maybe these plot points don’t sound quite so ridiculous now!

• Instead of relying on plain old galaxy-wide SSO, the Death Star plans in Rogue One are in a physical archive on Scarif, accessed through a mechanical retrieval system, and then can only be transmitted through a physical tower after a military assault.

• The entire Trade Federation droid army’s server was centralized to one orbiting space station, minimizing the interfaces that could infect it, but maximizing the effect of a well aimed laser.

Maybe Not Either / Or

Joe Reis recently posted a useful security-related prediction in his Mythos-related piece. The rough idea was that organizations with access to cutting-edge cyber models could defend themselves, while organizations without those models could not. This is the in-between scenario of offense vs. defense.

However, attackers do not need to beat the best-defended organization directly if they can walk through the weakest connected partner.

Most of us have discovered that much of the value of AI stems from its interconnectedness with applications across an ecosystem. Even an “in-between” scenario, can quickly result in a world of walled gardens, greatly limiting AI’s impact, but perhaps not achieving the full extent of a world of Star Wars-like airgaps.

Practical Advice for Today

With all that being said, we aren’t in a battle among Jedi and Sith. We live in the here and now. What can be done today? Here’s a synthesis of practical best practices we have collected from those in AI security.

• A system is as vulnerable as its weakest component. Strengthen those weak points, create barriers between system, or cut them off entirely.

• If an AI agent has access to a system or piece of information, assume that anyone who can access that AI agent has access to the same.

• Any context that AI pulls from can be used for exploitation. Online “best collections of AI skills and AI connectors” are especially rampant with vulnerabilities.

• Update your applications with recent upstream versions… BUT NOT TOO FAST. Every old vulnerability can be exploited, but every dependency can be supply chain attacked, and that most likely surfaces in the most recent versions, like LiteLLM or Axios or … .

The bottom line: This makes for interesting futurecasting, but the immediate action is to take security seriously today, without going full Star Wars airgapped mode, which could also destroy your organization, just more slowly.

Architected Intelligence: Principles for Building AI-First Organizations and Technologies by Jacob Miller and Jeremy Mumford is available now. Published by Wiley, 2026.

My reason for this new answer is that I’ve come to recognize an evolution in the applied AI industry that has been taking place over the last five years. Perhaps early on it was possible to know everything about leveraging LLMs in a business setting, when the only model that mattered was GPT-3.5 Turbo and RAG was the hot new trend. But today, I find myself painfully aware of the gaps in my knowledge as I look across the vast landscape of generative AI that started from those early seeds. I’ve also come to accept those gaps and take joy in my specialties.

“I don’t know everything about AI” is maybe a risky thing to share with the world, considering I’m trying to persuade people to buy a book about AI from the position of “I am a person that knows a lot about AI”. I certainly spend a lot of time thinking about AI. How it affects my organization. Why it’s powerful. What I can do to help people learn how to use it. But I also have shallow knowledge on areas like building MCP servers or designing advanced knowledge management systems. The truth is, the reach of generative AI has grown so broad that it is impossible for a single person to execute everything you can do with it deeply and effectively in all areas. AI engineers must learn to specialize. I’ve decided that I’m comfortable with the areas I am specialized in, and can accept my shortcomings in the other areas.

This gap between the desire to know everything and the reality of needing to specialize can cause problems if you don’t learn to accept it. You’ll waste time, constantly churning and attempting to consume everything, when in reality you could be diving deeper in any sub-area. Become an expert in knowledge management systems. Establish yourself as a wizard with evals. Build an MCP server or AI friendly CLI. Learn how to deploy autonomous agents. Go deep with local models, or focus on connecting to AI via paid APIs. But it’s dangerous to be a generic jack-of-all-trades. My friend Carlos Kenemy said something wise:

It is no longer enough to be broad OR deep. You must do BOTH to succeed in the age of AI.

• Carlos Kenemy

In the world of recruiting, this is known as “T-shaped”. You search for talent that has both a broad (yet admittedly shallow) understanding of many things, while also excelling in a single area. I feel like at Pattern, we’ve been fortunate in finding many new team members that fit that description.

The key to being a successful T-shaped AI engineer and finding where to go deep is humility. Humility is what allows you to recognize gaps in your knowledge. Having gaps is ok. But you need to know which gaps need filling, and which gaps can be strategically neglected. I could go get a GCP certification for AI. But, perhaps my time would be better spent attaining an AWS certification since I don’t use GCP at my company. Time is short. I can’t do everything. Instead, go deep where you have talent, have passion, and there is need in your current company or in the market.

Here are some potential areas an AI engineer could focus in 2026, many of which have many openings and few people specializing in them:

• Evaluations: Understanding how to build benchmarks, define metrics, run A/B tests, and optimize agents is a powerful skill that not many have. Tools like Langfuse and Opik are quickly becoming a requirement on job descriptions.

• Tools: Tools are the key to AI agents that deliver read value. Plugging in an MCP server you found on the internet is easy, but building MCP servers and handling them at an enterprise scale is much more challenging. How do you handle RBAC, discovery, and definitions? Tools like FastMCP are key.

• Running open-source models on infra: This is a powerful skillset with a wide range of expertise. On one end of the spectrum are lots of new entrants jumping into the field, fulfilling the massive demand from companies that demand on-prem inference for privacy reasons or customization to domain problems. On the other end of the spectrum are people with PhDs in NLP, who have deep technical skills to actually undercut hyperscalers and outperform out-of-the-box models with advanced customization techniques.

• Workflows (no-code): Building AI workflows that can be repeatedly and deterministically run is a powerful technique, often superior to open-ended AI agents. Tools like Zapier, n8n, OpenAI Agents, and more offer non-coders a mirage of the ‘easy’ solution that vanishes once reality hits. Operating these tools at scale is challenging. Having an expert to lead an organization in building these workflows is invaluable.

• Agents: AI running in the cloud, using tool harnesses, solving problems. It’s the holy grail. This speciality sits at the intersection of many different specialities, and can often shapeshift depending on how a company is using the term ‘agent’. Tools like Temporal, CrewAI, LangGraph, AgentCore, AI SDK, and more are beneficial for these specialists.

• AI SDLC: Dario Amodei said 90% of coding would be AI generated by the end of 2025. He was right. In a follow interview, he said the next step is reaching 100% (much harder) and attacking the steps before and after coding: problem definition and solution validation. That transformation is happening as you read this. People with expertise in AI-driven Software Development Life Cycle (AI-SDLC) can revolutionize how companies create software, raising the productivity of entire engineering departments.

• Knowledge Management: The once humble Retrieval Augmented Generation (RAG) technique has evolved into a highly complex discipline known as ‘knowledge management’. This is much more than just vector databases. You’re dealing with the scale of big data and the complexity of unstructured data, trying to handle RBAC for sensitive documents while also maximizing retrieval accuracy and relevancy at the lowest cost point possible. If you feel like you can tackle all of that, then you’re ready to specialize in knowledge management.

• Training and consulting: They say those who can’t do, teach. The reality is that being an expert in how generative AI works and knowing how to train a room of 100 people to use ChatGPT effectively are not the same thing. People who can effectively teach and lead people in AI are sorely needed, and those teachers are typically high skilled experts who find joy in sharing knowledge.

• Observability: Congratulations, you’re using AI! But who’s using it? What’s going to the model? How much does it cost? Questions like these can go unanswered if companies have no cohesive strategy when it comes to AI access. Having a strategy for AI observability is critical. Understanding OpenTelemetry and enterprise resource management are key in this area.

This is not an exhaustive list, and I’d love to hear what other areas of specialization you are noticing. Hopefully, for those searching right now in college and in the middle of their careers, this is helpful. To circle back to the book I referenced (Architected Intelligence), this diversity of specializations is displayed throughout the different chapters of the book and can be a helpful guide.

The bottom line: AI engineering is hot right now, but you’ll be even hotter if you learn to specialize inside of that nebulous title.

We’re now at the throwing Molotov cocktails stage of AI disruption.

You do not have to like Sam Altman, OpenAI, or the trajectory of AI to recognize that attempted violence against a household is an act of civilizational decay. It is also not an isolated moral failure from one maladjusted individual floating in a vacuum. Our hyperboles and rants and echo chambers amplify the noise and temperature and grant permission to the crazies. When everything is framed as apocalyptic and unforgivable, somebody eventually decides extraordinary action is justified. That is how moral panic works in practice.

AI is important, perhaps existentially so. There are legitimate debates around concentration of power, labor displacement, geopolitical risk, erosion of human autonomy, truth, and personal safety, among many others. However, descending into chaos and violence will push us further from constructive pathways. I’ve come to believe more than ever in “disagree and commit.” A team choosing the wrong strategy and executing well is more likely to succeed than one that is fragmented, even if the strategy is right. That applies inside companies and also applies, more loosely, to societies trying to navigate a major technological shift without tearing themselves apart.

The Mechanical Extensions of Mankind

One of the most useful heuristics for technology is to consider it an extension of your biological capabilities: muscles, brain, and nervous system.

A tractor enhances my strength. We even measure it in horsepower! A calculator enhances my mind, and its speed can be measured in FLOPS and MemOps. It outsources my arithmetic and enhances my precision, sometimes at a transcendent scale. Social media and the internet augments my nervous system across humanity. Suddenly I can feel the outrage, panic, joy, humiliation, fear, laughter, tribal loyalty, and moral disgust of strangers. All of this occurs at scale (all of us have access), instantaneously (news travels fast), and perpetually (in any moment I can ingest more content).

Each of these forms of technology can be dangerous. I can avoid exercising my body because cars and tractors and machines do all of the work. My thinking and memory abilities can atrophy if I completely avoid rigorous thinking or ask Google every question. These can be dangerous and challenging, but little has biologically and psychologically prepared us for our overexcited “hypertrophied” nervous systems that began with the telegraph and moved to radio, then television, then internet, then social media, and now potentially AI.

The Crisis of the Hyper-Connected Nervous System

One of the strangest phenomena of our era is that most of us are functionally living with an enlarged nervous system and an underdeveloped social immune system.

We see a video, we get mad. We see a viral post, we get upset. We see a clip stripped of context and our body reacts before our mind has rationally processed it.

Social media lets us feel every slight and threat instantly, but lacks the framework to interpret it intelligently or wisely. For years I’ve joked with others to call the internet “the rage machine.” Although, with short form video, I’d now rename it the “rage and mollification machine.” We love to imagine ourselves as resisting the machine. Usually we are just becoming a cog in a different one. It’s an outrage machine with no brakes, fueled by social media incentives to maximize engagement.

One of the most dangerous things about moral panic is that it makes us feel more virtuous than we are. It gives people a villain to hate, a script to follow, and a flattering story about themselves. That combination creates a dangerous precursor cocktail long before someone throws a literal one.

The Erosion of Perspective

I have lived through so many supposed apocalypses. So many. Nuclear annihilation through mutually assured destruction with the Soviets was the most real of them all, but I also somehow survived y2k, net neutrality, near-term projected 10-20 meter rising ocean levels, and I even played Dungeons & Dragons through the satanic panic. When everything is framed as apocalyptic, the public loses the ability to distinguish between existential stakes and theatrical stakes.

For our day, if your worldview requires every AI leader to be either a savior or a supervillain, your worldview should not be taken seriously. The framing is childish and operationally reductive. The real world is full of flawed and complicated people inside of complex systems of massive consequence, mixed incentives, partial knowledge, institutional pressure, and genuine disagreement.

I don’t worry so much about either disagreement or isolation. You can disagree with others. You can also build your own thing with other people and play by your own rules. Those are often healthy. But hyper-connected fragmentation is a different animal because it creates discord and violence because it combines tribal identity, constant stimulation, public performance, and a general “low-friction at high velocity” escalation.

AI: A Scaffolding for Collective Reasoning?

Humanity built technologies that let millions of people feel and react together before we built technologies to help us reason together.

The internet, smartphones, and social media massively expanded our nervous systems. They connected sensations with reactions to create social contagion across the planet. However, they did not do much to improve collective reasoning. Because the platforms were built to optimize engagement, they often degraded reasoning. The last twenty years have shown our desperate need for antibodies against systems that optimize for engagement.

AI is interesting in part because it might help fill that role. On our previous trajectory, were we doomed to destroy ourselves on the internet rage and mollification machine without AI’s assistance?

To be clear, I do not believe that means AI will magically save us. It could absolutely exacerbate much of this. Tailored manipulation, industrialized slop, automated propaganda, and fake consensus may make the dead internet theory a reality. I see those arguments and some of them are compelling, but for the first time in this broader technological arc, we may have tools that can actually help people REASON together instead of REACTING together. Tools that can summarize arguments, expose contradictions, compare claims, simulate tradeoffs, clarify mechanisms, and lower the cost of careful thinking instead of only lowering the cost of emotional amplification.

While I don’t believe in going full Spock mode with no feelings, biologically, the most viral reactions are threats in order to increase our probability of survival (anxiety, anger, “the outgroup is so stupid”). If we were all reacting to inspirational messages to collaborate better and look for opportunities to serve the people around us, this wouldn’t be an issue, but the infection rate for positivity in us humans is much lower than for negativity.

Share

Though people on X are dunking on “@grok is this true,” I unironically believe this is one of the most positive recent developments to shift away from the previous trajectory of pernicious social media virality. Humanity is combatting a nervous-system overflow with enhanced reasoning at scale for the very first time.

What Can We I Do?

We start with what we have the most influence over, and that is… ourselves! Block the hyperbolic accounts. Click the three dots on whatever platform you are on and mute, show less, unfollow, mute keywords, whatever version exists. It starts with ourselves to stop being infected by the omnipresent overloaded sympathetic nervous system. Also, it lowers reach. If a source reliably makes us dumber, angrier, and more certain than the evidence supports, we have to stop piping it directly into our consciousness.

Looking back on this moment 20 years from now, is the attack on Sam Altman’s home the beginning of a grand destructive conflict, or will it become a small asterisk? If AI becomes the topic flash point for outrage, and then AI itself is used to further fuel it, we are not positioned well.

If reacting together outpaces reasoning together, we will all lose.

Subscribe now

What makes a business succeed? Many frameworks reduce it to some variation of three things:

Market – You are competing somewhere worth winning. It is better to be a mediocre company in a great market than a brilliant company in a dying one.

Strategy – You differentiate in the right way and get the product into customers’ hands before someone else does. You give customers a reason to choose you and a reason to stay.

Execution – You repeatedly deliver higher quality at lower cost. You improve faster than the competition. You turn learning into a system.

Different eras reward different strengths. Buffett built Berkshire largely around market selection and capital allocation. Salesforce built an empire around strategy and workflow entrenchment. Toyota turned execution into a quasi religion.

For the last twenty years, execution has often been the quiet third variable. Many companies succeeded with strong markets and strong strategy while execution ranged from excellent to surprisingly mediocre. High margins, venture capital, network effects, and digital distribution covered a lot of operational inefficiency. The lesson many organizations internalized, even if they never said it explicitly, was that execution mattered, but it did not matter as much as being in the right market at the right time with the right positioning.

And that’s when AI showed up…

AI reduces the cost of doing work. That part is obvious and widely discussed. What is less discussed is that AI also reduces the cost of learning. Teams can build faster, but they can also gather feedback faster, synthesize it faster, analyze failures faster, generate alternatives faster, and iterate faster. The build loop accelerates and the learning loop accelerates. Both sides of the equation improve at the same time.

Because of that, execution should be thought of less as effort or culture and more as a measurable capability:

Execution is the number of high-quality build-and-learn cycles an organization can complete in a given period of time.

Once you think about it this way, the impact of AI becomes clearer. AI increases the number of cycles you can run AND improves what you learn from each cycle. The organizations that benefit most will not necessarily be the ones with the best strategy documents or the most ambitious AI roadmaps. They will be the organizations that can run more high-quality cycles than everyone else.

Many in the venture capital industry have been beating the drum making similar points. Marc Andreessen beats the drum on OODA loops (observe, orient, decide, act). At a recent investor panel, I heard Ethan Choi of Khosla Ventures repeatedly emphasizing that the primary attributes they look for in companies were the founding team’s ability to execute with velocity.

This is where the idea of the gingerbread man moat comes in. The gingerbread man does not win by being bigger or stronger or cuter. Instead, he wins by being difficult to catch. He keeps moving, and when he stops moving, he gets eaten. For many industries right now, that is the situation. Traditional moats still exist. Regulation still matters. Brand still matters. Distribution still matters. Capital intensity still matters. Data still matters. But velocity (execution) is becoming a moat available to far more companies than before.

The companies that can iterate, learn, and improve faster will feel very difficult to compete against because they keep widening the gap. Most organizations are not built for that.

The Recipe for Velocity

When you look closely at teams that execute quickly, a pattern shows up repeatedly. They tend to have more decision makers (without complete decentralization) with real authority operating closer to the work. That sounds simple, but it requires three things to be true at the same time.

Empowered – They are allowed to make decisions and they have the resources to act on those decisions. Authority without resources results in flailing. Resources without authority create frustration. Both authority and resources are required.

Aligned – They are accountable for the outcome, they believe in the direction, and their incentives point in the same direction as the organization. They receive feedback not only on the product, but also on their own decisions and performance so they can improve along with the system they are building.

Informed – They are close to the user, the product, and the technology. They understand the problems and the edge cases. They are NOT operating entirely through dashboards and summaries. They have proximity to reality, and they still bring a magnifying glass.

Empowered, Aligned, Informed. Each is necessary, and none is sufficient alone.

There is also a surprisingly simple way to get a rough sense of whether an organization is structured this way. Look at the meetings. Do not count the number of meetings. Instead, count the number of attendees in the meetings.

Meeting count is a noisy metric. A team can have many short meetings and still move quickly. Or, a team can have collaborative four hour working sessions with three people. The more revealing signal is how many people need to be in the room for a quality decision to happen. When empowerment, alignment, and information are concentrated in the right people, meetings tend to be small. When those things are fragmented across layers and functions, committees appear, either formally or informally. Committees meet because they have to. If decisions require pieces that live in different places (empowerment, alignment, or information), those pieces have to be assembled regularly. Often, they fail in their attempt to assemble each piece. The more fragmented, the more people are necessary and the likelihood increases that a critical component is missed.

Large recurring meetings are often a structural signal that suggests that components are distributed across the organization in a way that slows decision making and iteration. And now with AI, iteration speed is increasingly the game.

To compress this argument to three points:

• AI has tilted the environment back toward execution. Not exclusively toward execution, but enough that execution capability is becoming more important relative to market selection and high-level strategy than it has been in a long time.

• Many organizations optimized for the previous environment where market position, strategy, and access to capital carried more weight. They built structures that were good at planning, coordinating, and managing risk across large organizations. Those are still valuable capabilities, but they do not produce high iteration velocity.

• The organizations that will do well in this environment are the ones that can empower the right people, align them to outcomes, keep them informed and close to reality, and then allow them to run many high-quality build-and-learn cycles.

Many organizations do not have those strengths and are not maniacally pursuing them. Over the next 24 months, that gap will weigh more than the pile of AI-written strategy documents attempting to dig the company out of its hole.

Architected Intelligence: Principles for Building AI-First Organizations and Technologies by Jacob Miller and Jeremy Mumford is available now. Published by Wiley, 2026.

A few years ago, I made the decision to change my title to AI Engineer. At the time, it was a decision that carried risk. Other new and trendy titles like “Prompt Engineer” were in vogue. I felt strongly that AI engineer was the descriptor that was meant for me, and time has proved me right. Prompt Engineer got laughed out as a legitimate title, but AI Engineer has managed to cement itself as a new fixture in companies. LinkedIn’s 2025 Jobs on the Rise report named Artificial Intelligence Engineer as the #1 fastest growing job position.

But what is an AI Engineer? The job report describes it as someone who will “design, develop and implement AI models and algorithms to optimize processes and solve complex problems.” Here’s the problem with that statement. Designing + developing an AI model versus implementing it to solve business problems are two completely different jobs! One of those individuals makes $3 million a year at Anthropic to design the next Claude model, and the other is on the frontlines solving business problems using generative AI as their primary tool of choice. This conflation of roles has caused many folks on the internet to crash out, decrying how “Back in my day, AI engineers actually knew how AI models worked”. And that is a fair point. Most of the roles you see open for AI engineers have skill requirements that look suspiciously like software engineers.

So I want to put my hot take out there: People who build AI models are NOT AI engineers. Call them anything else: ML engineers, data scientists, AI researchers, NLP specialists, MLOps engineers, LLM Ops engineers, LLM engineers, etc. But most AI engineer positions you see open on LinkedIn are not asking you to build an AI model, or even tinker with one. Why? Because 99% of companies believe they can gain more ROI from leveraging existing foundation models rather than hiring experts that actually know how to build and fine-tune LLMs. The assumption is that the foundation models from the Big Three (OpenAI, Anthropic, and Google)1 are so good that there is no point in trying to compete. It’s easier to just plug into what they have and focus on context engineering.

So if AI engineers are not defined by building AI models, what does define them? Is it just calling the OpenAI API? The role of an AI engineer is defined in my mind with the following statement:

Think like a data scientist,

build like a software engineer

Much of the day to day work of an AI engineer looks suspiciously like a software engineer. You are writing application code or backend services that support applications. These are hard jobs, but they aren’t new. The key difference between the AI engineer and the software engineer is that non-deterministic generative AI is a critical component in an AI engineer’s applications and platforms. Why is this significant? Let’s look at the difference between software engineers and data scientists. Software engineers can feel confident that if it works in production on day one, it will likely work in production on day 20. Data scientists do not have that same luxury. The below graph that I shamelessly stole from the internet (thanks Eric) illustrates this difference in confidence.

AI engineers straddle both disciplines. And that’s what makes it an interesting job. You need to know BOTH sides of the coin.

On the software engineer side, you need to:

• Build scalable infrastructure with IaC and CI/CD

• Create codebases that can handle dozens or hundreds of developers working simultaneously

• Navigate micro-service cross-service chaos with modern observability practices, attention to security, and thoughtful API design

• Provide real value to users (the most simple but oft overlooked skill)

On the data science side, you need to:

• Handle non-deterministic behavior

• Use benchmarks and A/B testing

• Identify trade-offs between various approaches

• Understand how AI and data combine to solve previously impossible problems

At the intersection of both of these are skills like:

• Knowing when AI is actually a subpar solution that could be replaced with deterministic code logic

• Staying on top of an avalanche of tech updates that are obscured by marketing and internet hype bros

• Evaluating when to build vs self-host vs buy, and identifying when to re-evaluate and change those decisions in a light-speed industry that constantly evolves

• Diffusing AI knowledge across the broader organization

So to repeat: AI engineers blend the skillset and daily activities of a software engineer with the mindset and caution of a data scientist. They focus on building services that leverage generative AI, understanding that they need to follow the best practices of building production worthy applications while also applying the rigorous standards of testing in the field of data science. While not every job posting looking for an AI engineer will actually match what I’ve described, this is the standard I believe it’s solidifying around, and what every AI engineer should strive for.

Quick plug: A lot of the things an AI engineer needs to know are outlined in the book I just wrote with Jacob Miller, Architected Intelligence. We directly address the skill gap that AI engineers have as they get pulled in from either the software engineering or data science side.

The bottom line: Think like a data scientist, build like a software engineer. That is the role and purpose of an AI engineer. Starting with that mantra will create a powerful foundation that will pay dividends moving forward.

That performance gap is what drove Jeremy Mumford and me to write Architected Intelligence: Principles for Building AI-First Organizations and Technologies, published by Wiley, and now available for purchase.

We are aware of the irony of publishing a physical book about the most transformational technology since the printing press. We have made peace with it.

It takes a certain amount of hubris to write a book. We know the AI community didn’t need one more AI title with a glowing robot head on the cover and a subtitle boldly claiming that we’re “unlocking tomorrow” with a “blueprint of the future.” We wrote Architected Intelligence because over the last few years we kept seeing the same modes and patterns of failure and a recurring single framework for success.

Subscribe now

Before we dive into the patterns of success, we’re certain you recognize the failures. Product teams bolted on AI summaries with a nice ✨emoji and called it innovation, while the technical teams built unscalable but impressive demos destined for the proof-of-concept graveyard. Leadership would then get frustrated as they see the wild progress on social media and make a decision to “shake things up.” However, instead of a new bold “AI strategy,” what these companies really needed was a way to think clearly about value, data, models, trust, and organizational execution as one connected system.

These companies needed a structure to architect with artificial intelligence.

The Chaotic Early Days

“The art of progress is to preserve order amid change and to preserve change amid order.” — Alfred North Whitehead

Think back to when ChatGPT launched on November 30, 2022 and the internet promptly lost its mind in the ensuing months. We were in the trenches at Pattern, navigating a high-growth ecommerce environment attempting to shape billions in revenue while our feet were being swept away in the AI tsunami.

What followed was a strange period (in retrospect) with an odd combination of enormous promise, small context windows, frequent model outages, pricey API calls, and groundbreaking but still limited practical intelligence. The technology was historic, but we all were struggling to turn it into something that mattered and created something valuable. The early constraints turned out to be useful because they forced discipline.

The traditional siloed disciplines were failing the AI transformation. In one corner, software engineering didn’t know how to handle the non-deterministic nature of LLMs, and in the other corner data science was too slow and opaque for the “move fast and break things” pace of generative AI, and knowledge management wasn’t found in any corner at all because the discipline was mostly forgotten in an “agile” world.

We needed a unified theory.

A Framework Born in the Trenches

As we worked with the first wave of generative AI tools, we found ourselves pulled into a recurring set of five questions.

• What exactly are we trying to produce?

• What data is actually needed to make that output valuable?

• What model behavior matters for this use case?

• How do we know whether the system is trustworthy in production?

• What is the path to end the “wave of experiments” and begin to scale?

Those questions eventually solidified into a simple interdisciplinary framework. It drew from software engineering, data science, data engineering, product thinking, and knowledge management. It was useful and gave us a way to build and communicate real AI products and processes. The result was a multidisciplinary approach that we call Architected Intelligence.

We further built out our framework with the builders and thinkers we admired. The Andrej Karpathy and Ethan Mollicks of the world, along with the research posted by the leading AI labs and the individual researchers and engineers posting publicly from within them. We tested these concepts in the messier places where ideas either become useful or get exposed quickly: internal planning, system design, conference talks, consulting engagements, vendor discussions, customer conversations, AI meetups, roadmap fights, and postmortems. That broader feedback loop mattered.

While we received strong encouragement and saw positive initial results, we also received sharp criticisms from smart people, further refining our frameworks and principles. To us, the tipping point was when people more brilliant than us could take our structure, adapt and apply it to their own domain, and suddenly move faster. It helped teams break down what had felt like a giant, foggy AI ambition into concrete parts they could actually reason about, communicate, and build.

That was the moment we started taking the idea of a book more seriously, so we partnered with the Wiley team to bring you a book and lay out this framework for you.

The Five Components of Architected Intelligence

Successful AI products and processes consistently execute on five components, in order:

Output — In a world where execution gets cheaper, judgment compounds. What valuable experience are you designing? What is the exact value that each AI call is responsible for? Every downstream technical decision traces back to this.

Input — Power the system with high-quality, “canon” knowledge and differentiated data delivered in the moment it’s needed. This is the AI system’s fuel. Without unique data, your brilliance will be copied and commoditized.

Model — The model that you are leveraging is the engine and NOT the vehicle. Context engineering, fine-tuning, model selection, and cost-latency-intelligence tradeoffs are where technical decisions get made, but only after you’ve defined the output and prepared the input.

Observability — Trust is the currency of AI adoption, and it’s fragile. You need to know what your system is doing, why it’s doing it, and whether that remains acceptable as conditions change. Traditional software observability and data science evaluation each solve half the problem. You need both.

Enablement — The graveyard of AI is full of great prototypes that never scaled. Enablement is the infrastructure, talent, culture, and process that transforms successful experiments into durable organizational capability.

These five components form the structure of the entire book. Each stands on its own and you can open to the chapter most relevant to your immediate problem and apply it. We believe, however, that the real payoff comes from the full picture when you see and understand how a failure in Input undermines a perfectly engineered Model and how gaps in Observability silently destroy trust in an otherwise valuable product.

A Book Built to Last

We wrote the book we wish we’d had as we built our AI platform, products, and processes. It’s grounded in what successful AI projects have in common, whether they were built on GPT-3.5 or on models that didn’t exist six months ago. The principles that govern how you define valuable output, acquire and curate data, configure models, maintain observability, and build enabling infrastructure don’t change when the underlying models do. That’s the bet we made, and we think it holds.

Who This Is For

The primary audience is technical leaders. This includes:

• engineering leaders responsible for delivering AI projects that actually work,

• technical advisors helping organizations navigate transformation

• technology leaders charting the map between their AI ambitions and their team’s current ability to execute

However, we also wrote it so that product leaders and others working close to AI could follow the arc and develop the core intuition for what quality AI projects require. If your roadmap is reliant on blind trust and crossed fingers and you’re looking for a remedy, this book was written for you.

For the Builders

In the end, this book is for the people who we believe will inherit the AI-first future. The builders. The ones who eschew the hype cycles and simply want to create something beautiful and valuable with this extraordinary mass of matrix algebra, computation, and data that has ushered in a new era.

That’s the spirit behind both this newsletter, The Why Behind AI, and our book Architected Intelligence. We hope this newsletter and this book help you architect yours.

Architected Intelligence: Principles for Building AI-First Organizations and Technologies by Jacob Miller and Jeremy Mumford is available now. Published by Wiley, 2026.

There has been significant recent hype around the latest AI models and the impending disruption across the economy.

Very few, however, walk through the variables that determine the blast radius. Fewer still provide a playbook to integrate the new technology stack across white-collar work. This article does both: what determines the level of disruption AI causes in a given domain, and the critical moves to incorporate it, whatever its degree within your space.

The Landscape

Is everything crashing down, or up, or sideways? Is your job or industry ripe for disruption?

Narrow professions, or segments of professions, have already been disrupted by AI. But software engineering is ground zero for the first major, widespread disruption across a large industry. It provides the lessons to apply everywhere else.

The immediate impact of AI is lowering the cost of doing. This means the competitive bottleneck shifts away from raw execution and toward the disciplines that determine whether execution becomes value:

a) choosing the right outcomes,

b) translating those outcomes into real constraints, and

c) learning faster than the competition through iteration and implementation.

This reframing matters because velocity is increasing, but not in a simple “everyone ships more” way. In a world with tight feedback loops, speed tends to win, especially when aimed at deeper user understanding.

Faster building also changes the economics of experimentation. The median ROI across technology products is negative, while the average ROI is shockingly high because a small number of outliers create enormous value. As AI makes building easier, we expect that pattern to intensify. More teams will run more experiments, more will fail, and the median will fall. But the winners will iterate faster, compound learning sooner, and capture more value, pulling the average higher. Velocity becomes a “fail fast” superpower when paired with disciplined feedback and a clear definition of what good looks like.

The New Roles in an AI-First World

As roles evolve, traditional boundaries blur. The classic separation between functions begins to break down. In software engineering, small teams are already outpacing larger ones and outperforming more credentialed ones.

A comparison that makes this easy to grasp is a restaurant where a robot does the cooking. Someone decides what’s on the menu and what a great meal feels like (Direction), someone designs the kitchen so it can actually produce that meal reliably at scale (Architecture), and someone tastes everything and figures out what needs fixing (Evaluation). While the robot does most of the cooking, the other three roles become more important.

These three roles have become the bottlenecks to creating value.

• Direction: the ability to precisely define a valuable end-to-end experience.

• Architecture: the ability to translate that experience into constraints, interfaces, and data relationships that create reliable systems.

• Evaluation: the ability to measure, debug, and learn through testing, verification, and user feedback.

Crucially, these are NOT necessarily three job titles or even departments. These are three categories of responsibility that will blend within individuals and teams in some companies, but also scale into specialized teams in larger organizations.

Direction

Direction is the discipline of deciding what is worth building and what “good” should feel like when it is built… in detail and within the realm of feasibility of our new, highly capable systems. This is not “an ideas person” waving their hands at science fiction. Understanding the capabilities and the “jagged edges” of AI is a prerequisite to doing Direction well.

The core artifact is a set of vivid user stories detailing walkthroughs of the user experience that are far more specific than most teams produce today. These stories capture the complete experience: how the user discovers the feature, how it behaves under normal use, what happens when something goes wrong, and, crucially, what the user trusts the system to do on their behalf.

Metrics still matter, but they rarely encode the full complexity (or welcome simplicity) of a user experience. Direction includes look and feel, tone and clarity of system responses, workflow structure, and the evidence behind the stories: user research, support tickets, win-loss notes, market signals, and qualitative insight.

A Direction function should be able to answer, with precision: Who is this for? What are they trying to accomplish? What does success feel like? What should the user never experience? When Direction is done well, teams share a definition of a great experience and understand why it creates value.

Architecture

Architecture translates Direction into a reliable system. Once Direction defines the desired experience, Architecture defines the technical realities required to deliver it. This includes: performance requirements like latency, uptime, and graceful degradation; safety requirements around privacy, security, and compliance; and the constraints that keep teams from accidentally breaking things as they move fast.

Architecture consists of the technically demanding job of defining requirements and constraints. AI can propose designs and generate code, but Architecture must recognize where AI remains weak. In software engineering some of its weakest areas include data and infrastructure complexity. These areas still require deeper technical expertise and will continue to do so for at least a couple years.

Architecture should make it easy to build quickly, now and in the future, without causing severe breakages. It specifies what must be true for the system to behave the way Direction intends, even under load, when dependencies fail, or when new features are added. When Architecture executes well, builders can move fast because they are operating inside a set of constraints and interfaces that prevent accidental catastrophes. It becomes much easier to parallelize work across teams (human and AI) because the system contracts are explicit.

Evaluation

We will never “one-shot” perfection. Evaluation provides the feedback loop for continuous iteration. It is how the organization learns whether the system is actually delivering the experience defined in Direction, within the constraints defined by Architecture.

The speed at which our AI restaurant dishes can be expertly tasted is a primary determinant of success.

Besides speed, a mature Evaluation function distinguishes between:

• Bugs that violate expectations

• Quality issues that could be improved

• Feature requests that expand scope

Each requires a different response. That triage discipline is a core element of Evaluation. It becomes a primary source of competitive advantage. “Move fast, but don’t break anything essential” is now a genuine choice.

Variables of AI Disruption for White-Collar Work

This flow of Direction, Architecture, Evaluation will quickly begin to transform most white-collar work. Software engineering is a special case because it lines up almost perfectly with AI’s current capabilities. Other fields will require more adjustment, but the framework still applies.

What determines how fast AI comes for a given domain? We identify four spectrums, in order of importance.

Fidelity to Simulation

In 2020, Google DeepMind’s AlphaFold stunned the scientific world by solving protein structure prediction. In a matter of months, a single team with relatively limited experience in the field solved a critical area of basic research that had stumped biologists for fifty years. The team combined novel AI methods with the ability to run simulations at a scale no human researcher could match.

On the other hand, to date, no AI can predict with high accuracy whether a given protein will survive FDA trials for human use. One step further downstream, and the AI hits a wall because we still lack a sufficiently faithful model of how the human body actually responds to medicine.

The deep question at play is: Can the AI check its own work? Can it simulate reality with enough fidelity to know whether it solved the problem, or whether it should keep trying?

Where the answer is yes, the AI transformation will be swift. Software engineering sits at the far end of this spectrum because an AI can attempt thousands of solutions and simulate the final outcomes with nothing but processing cost. Physics, mathematics, and many engineering disciplines share this property.

Wherever there are high fidelity simulations, AI will dominate faster than most expect.

Reversibility + Feedback Velocity

In The Incredibles, the villain [redacted for spoilers] solves his superhero problem with ruthless… inefficiency. He lures heroes to a remote island, unleashes the latest robot, and lets the fight play out. Robot loses? Upgrade it. Repeat until the robot wins… lethally.

Super villains are often not efficient in their A/B testing

We don’t have the time, resources, or morals to run experiments that way. The relevant question for AI’s potential impact in a domain is: Is this a two-way door?

Jeff Bezos’s two-way door heuristic: if a decision is easily reversible, just try it. For our purposes in AI, we also need tight feedback loops. It needs to be both reversible and visible.

Can we quickly tell if the AI goes wrong, and reverse it with little lasting damage? If yes, AI will move quickly.

Consider an AI proposing changes to a website’s checkout flow: move the cart icon and reposition the purchase button. Rather than debating, run an A/B test. A thousand users get the old experience, a thousand get the new one. Add-to-cart rates tell you everything you need to know within a week.

Where high-fidelity simulation isn’t possible, AI will dominate where there is real behavior, low stakes, fast signals, and easy rollbacks.

Bill O’Reilly also recognizes the importance of AI’s capabilities to perform live experiments.

Verification Speed

If a task falls outside high-fidelity simulation and isn’t quickly reversible, that leaves us with human verification of AI outputs.

A skilled graphic designer can request an AI to swap out a portion of an image. Upon reviewing the output, the designer points out, “The hand has six fingers. Fix it.” Verification is fast because the work is easily observable.

Other tasks are inherently harder to verify. For cancer treatment plans, large-scale data engineering, and complex legal analysis, even experts struggle to evaluate quality at a glance. At a certain point, it takes nearly as long to verify and correct the AI’s work as it does to simply do the task yourself. Here, AI may be most useful as a collaborative suggestion engine (a built-in copilot) rather than a primary executor.

The faster the verification speed, the faster AI permeates that type of work.

Digital Centricity

In August 2015, my friend Rhett bet me that autonomous vehicles would be ubiquitous across the USA within two years. Having done a deep dive in the sector recently, I bet scaled rollout across most major US metropolitan areas would occur by the end of 2025. I still undershot. We both lost.

After 11 years since that bet, autonomous vehicles are… kind of here. Even though they operate in more controlled road environments, it’s a challenging problem to tackle. However, navigating a busy city street on foot is even more challenging, and is far more chaotic than anything in cyberspace. Real-world complexity, combined with safety concerns and regulation, makes physically manipulating objects a far harder challenge than most assume.

The more value a job generates from physical-world interactions, the less AI will affect it. The HVAC technician’s hands-on work is safe. Their marketing, scheduling, accounting, and customer feedback workflows are not.

Software Engineering Catapults into AI-First

Now that we understand how and why AI will take on white-collar work generally, you’ll notice that along every dimension, software engineering is ideal for AI disruption.

We are being catapulted to a new land of AI-first, but we do get to choose which way we face when launched.

The right questions are:

• How reliably can our team transform intent into working systems, receive feedback, and iterate?

• How can AI increase the velocity at which that occurs?

The first question is our performance North Star, while the second question is how we will navigate there.

Let’s begin with a wrong answer. Even in software engineering, there are no AI-shaped holes waiting to be filled. We cannot simply sprinkle “AI dust” over legacy workflows and codebases and expect a step-change in productivity.

The right answer is building an AI-first philosophy that treats AI as a primary user of your systems, and not just a tool someone occasionally consults. We call this the User Agnosticism Tenet: products, documentation, and processes should be designed for ease of use by humans, AIs, or a combination of both. When you embrace this, a codebase becomes legible and adaptable to both the agents doing the day-to-day and the people managing them.

This transformation requires shifts along three axes. The industry is swiftly converging on this framework independently. When many teams converge upon the same “new discovery” completely independently, we consider it additional evidence that the direction is sound.

The Infrastructure (Sandbox)

We must first recognize that code does not leap out of a vacuum and execute in isolation. For an AI to be an effective collaborator, it must understand the “room” it is operating in. This means providing the AI with a cloud infrastructure, security protocols, authorization systems, and testing environments available to it.

An AI-first workspace ensures that the agent knows exactly what resources are at its disposal and, crucially, how to verify its own work. If the AI cannot autonomously spin up a container or run a battery of integration tests to ensure its contribution is stable, it will forever remain a “suggestion engine” and second-class contributor. The infrastructure must be architected so that the AI can navigate the path from “code written” to “code running” with the same level of environmental awareness as a senior systems engineer.

The Codebase (Self-Awareness)

In an AI-first world, teams must invest in creating a navigable map of each codebase. The AI requires signposts and documentation to progressively disclose where it should be working and how it should be working. The goal here is legibility.

In practice, this means embedding lightweight guide files (called agents.md and similar files) throughout the codebase that points AI agents toward correct modules, pull request standards, and known gotchas. More importantly, the hidden assumptions and user experience intent that once lived only in developers’ heads must become explicit, and embedded directly in the code. (Note: At the moment, you can’t shortcut and just have AI spin them all up for you. It doesn’t work.)

The goal here is progressive disclosure: the system should provide the right amount of context exactly when it’s needed. When an AI modifies a feature, it should understand both the how and the why. The implicit becomes explicit, and the explicit is surfaced exactly when it’s needed.

The Issues (Communication)

Even the most sophisticated model will fail if it is fed “slop.” This is the ultimate GIGO (Garbage In, Garbage Out) problem. Telling an AI to “fix the bug where the user sometimes gets an error” and then blaming the model is a flagrant skill issue. If the models improve, but quality and velocity doesn’t, we are obviously the bottleneck.

The “Issue” or work ticket becomes the primary interface for execution. Writing an excellent one requires discipline. If our North Star is, “How reliably can our team transform intent into working systems?” then we need to clarify intent into precise guidance.

We use AI agents that combine codebase context with user intent to workshop issues collaboratively until the issue is refined and sufficiently precise for the AI to successfully tackle the issue independently.

We now have created a virtuous cycle. If an AI can’t solve a well-defined issue, that’s a diagnostic signal: either the codebase is too brittle, or the issue writer needs to sharpen their thinking. Treating issue writing as an AI-enhanced craft trains product managers to translate intent into execution and trains engineers to care about user understanding.

The Result and Next Steps

AI assists you to communicate precisely to the AI system. The system’s own documentation knows both how to push you for further details and execute once the instructions are finalized. The system then simulates and validates the results before implementation.

That’s the new world of software engineering we have been thrown into, but it is beginning to transform human resources, medicine, finance, law, marketing, accounting, etc. etc. Each field will follow a similar arc of Communication, Self-Awareness, and Sandbox. The pace of that change is determined by:

1. Fidelity to simulation

2. Reversibility + feedback velocity

3. Verification speed

4. Digital centricity

So, what do you do?

First: Recognize that the value of many types of expertise has gone up and not down. Depth of expertise rather than years of experience is what to look for and develop, both in yourself and across your teams. Someone with two years of experience, but both of those years are focused within one core dimension, is more impactful than the 10 year generalist. Why?

• Communication and Design: Experts know how to design the process and communicate the nuanced jargon that can command the AI in specific ways.

• Evaluation: Experts know what good looks like and can provide high value feedback.

Second: Know where AI disruption will strike first in your domain. In software engineering, AI lags in infrastructure and data engineering but excels in traditional front and back end tasks. Map your equivalent.

Third: Get comfortable operating inside the new flow personally. It’s uncomfortable. Do it anyway on routine tasks until it’s not. You cannot assess the future state without living in it.

Fourth: Begin adapting the organization. Jobs are bundles of tasks. Teams are bundles of jobs. Organizations are bundles of teams and leaders. When the fundamental tasks change, it ripples upward. Growth and structural change will be required, incorporating, among other items, the key points laid out here.

We’re catapulting forward, but you can choose your landing.

Looking for practical guidance and frameworks to transform organizations and technologies with AI? Architected Intelligence is now available for pre-order!

This was not wrong for its era. Unfortunately, every salesperson read this sentence and collectively decided it was a phenomenal tactic. So they used your name. And then they used it again. And then again. Hi Jacob. Absolutely, Jacob. I completely understand, Jacob. Let me tell you why this is important, Jacob.

If you grew up in the era of cold calls from the 80s through the early 2000s, you developed a very specific reflex. Someone says your name too many times and your shoulders tense up. Your guard goes up. Your brain quietly labels the interaction as manipulation and prepares an exit strategy. The beautiful sound of my name became a defensive trigger.

Because of this, over time, we stopped using people’s names in conversation. It had been corrupted by bad actors. “Social antibodies” formed against our own names. For the bold, feel free to test this by using someone’s name over and over to see what happens.

Gen Alpha, however, does not have these antibodies. They did not grow up fielding dinner-time sales calls from someone reading a script filled with [First Name] fill-in-the-blank. This has created a cultural asymmetry across generations. If you’d like to show concern for people, you can once again repeat their name! (But only for those under the age of 30.) If you want to really bend people’s minds, call someone by their name over and over, but with a mixed audience across age groups.

We are now watching the same phenomenon of social antibodies in writing. Let’s do a case study on one of the most popular AI tells: —

Em dashes are great—truly. They let you add nuance, create rhythm, inject voice, and signal a thoughtful aside. They are a wonderful tool when used… sparingly.

However, through RLHF (for the uninitiated: humans providing feedback to the AI models based on their output) the models realized that raters consistently upvoted responses that looked “smart”—and nothing looks smarter to a tired human rater evaluating their 379th piece of AI generated content in a day than a confident em dash! It’s grammatically correct and humans upvote it. So, AI decides to use them everywhere. “For example” gets replaced with an em dash… “in other words,” em dash. Parenthetical aside? Em dash. You all win an em dash! Em dashes became the punctuation equivalent of a salesperson repeating your name in 1995, except this time I am not wearing a sick Rasta-colored slap bracelet.

Of course, em dashes aren’t the only sign of AI generated writing. That there are no slow burn payoffs is the most striking difference. Other obvious signs are excessive groups of three, short contrastive statements, always moving “from” something and “to” something, hyperbolic generalizations, and excessive colons. These tics and styles are the hallmark of AI writing because it looks good and feels good in small doses, but the dose makes the poison, and we’re all feeling a little sickly. Strike out all those contrastive statements in the AI-generated text and you realize that nothing of value was lost. We need meatier narrative arcs, long payoffs, and rule-breaking.

The writing ends up with the veneer of intelligence without the substance. It sounds thoughtful. It sounds measured. It sounds clear. And it is increasingly difficult to tell the difference between genuine insight and well-formatted slop. So, people adapt. Some writers are now adapting by writing a little poorly on purpose so you know a human was involved. We are the equivalent of hipsters dressing badly to signal irony, except now it is to signal our humanity.

Below I propose a different approach. This is what’s working for me in collaborating with AI, at least at the moment. I believe it also can apply to most forms of writing today.

• Sparring Partner – Fight with AI, ideally on voice mode, but you have to prompt it to be combative. Do I have anything useful to say? The AI pushes back and exposes weak reasoning. It makes for a good rubber duck armed with boxing gloves.

• Sans-AI Building Blocks – Talk or type out your key points. Record a discussion with a friend and generate a transcript or ramble into a microphone. Create as many building blocks as possible to work with: jokes, half-formed ideas, key takeaways, and insights.

• Outliner – Fight with AI on the post’s outline. Instead of “rough points,” get down into the nitty gritty and nail down a detailed play by play.

• Constructor – Take all of your original content and your outline, then tell the AI to take the content to fill your outline and to do so verbatim whenever possible. Tell it explicitly to NOT smooth out the rough edges… yet.

• Finisher – Pass it through one more time to make sure there aren’t obvious factual or grammatical errors. Gemini 3 is especially good at bolding every change it makes without resorting to a different UI. But you’ll have to pry the Millennial ellipses out of my cold dead fingers…

  • From Gemini on the final pass:
    You mention: “Gemini is especially good at bolding every change...”
    Note: Since I am Gemini, and I actually did bold the changes in this conversation, this is a funny meta-layer.

I love writing. I highly recommend it. In my eyes, I gain three benefits from writing. First, it forces clarity of thought. Paul Graham has a phenomenal short essay (go read it!) where he wrote, “writing is thinking.” Second, posting publicly acts as a forcing function to create “stage fright” and push me to think harder. Third, my thoughts might be useful to you. Having grown up in another era, I don’t feel either the possibility or allure of fame or influence, but when a person tells me what I wrote was genuinely helpful, it’s motivating. The older you get, the more gratifying it becomes to help others learn from your mistakes. If this is helpful, forward it to a friend or hit the subscribe button at the end.

And now, to conclude properly:

This is not about correctness—it is about authenticity.

This is not about AI—it is about humanity.

This is not about writing—it is about communication.

We delve deeper. We shift from noise to signal. We embrace nuance, clarity, and intentionality. 🤮🤖😂

Subscribe now

Subscribe now

Google’s last two weeks have been extraordinary. Google is crushing it. This is where we are in the two year cycle:

Nano Banana Pro

Google’s “Nano Banana” was already considered best-in-class at image generation, but the new model surpasses its predecessor. Where’s the unlock?

• Fidelity to Text: It captures text nearly perfectly in standard scenarios.

• Visual References: You can incorporate up to 14 visual references into the image with high fidelity.

• Image Quality: It generates professional-level 4k images.

• Logical Consistency: It has the ability to follow complex instructions; even complicated diagrams can be constructed accurately and consistently.

What are all of those use cases for AI generated images you’ve thought of?

• Custom flashy slides

• Detailed marketing posters

• Illustrations with consistently portrayed characters in a story or comic book

• Any job that usually takes 30 minutes or more in Adobe Photoshop

• Any touch up in Photoshop that takes less than 30 minutes, but doesn’t have to be exactly perfect.

It can do it all.

I turned most of my recent blog post for Pattern into a magazine shot, word-for-word:

Google Begins Swapping to AI Default for Search

Consider the following:

• Google has more users than any of the other large tech companies.

• Google makes insane amounts of money from search, basically acting as a monopoly on advertising search content.

• It is not obvious that 10 years from now, Google Search in its present form will be a viable business.

• There is no business model YET within AI that is as profitable as the current Google Search business model.

Google’s “self-disruption” decision to swap the default mode from Search to AI is one for the ages. They are potentially lighting hundreds of billions of dollars on fire to bet on this transition. Will they do it?

Google just started rolling out default AI mode to some users.

Old default view for Google.com:

New default view for some desktop users on Google.com:

Gemini 3 – Use It Now

Chat GPT 5.1 was released to some fanfare. It looked great in benchmarks, but the quality immediately dropped for all of my [non-coding] use cases. The reasoning would go haywire on me on a relatively frequent basis, including this fun instance where it tried to correct the spelling of “shoos.”

Driven by my frustration with 5.1 and the buzz surrounding Gemini, I came back to Google’s model, and it quickly became my go-to for most use cases. Yes, in a month or two someone else will come out with a breakthrough model and we will all switch again. But if you have access to Gemini Pro, you should start leveraging it today.

Shareable Gems Going Mainstream

Now that Gemini Gems are shareable, one of the easiest corporate AI enablement hacks is to quickly stand up a knowledge management system for a team, department, or company.

Here are the steps:

• Go to Gemini

• Create a Gemini Gem

• Drop in the company documents, guidelines, policies, and best practices you want accessible to the team.

• Share the gem with the team, just as you would a Google Drive folder

For a group of 5 to 150 people, this is the easiest way to stand up a knowledge management system. Larger groups will require more controls and scaling methods to build high-quality infrastructure, but this is the best solution for any small group right now.

Our Parting Thought

Find what’s useful. Discover lasting principles. Be wrong. Adapt.

• Technology adoption happens at viral speeds

• Early adopters thrive within the constraints of nascent technology

• Late adopters lose the talent war

• Late adopters can still win (if they’re willing to pay the price)

A Tale of Two Movies

The first movie is one you may not have heard of: Home on the Range, released in 2004. Home on the Range was made by the venerated Disney Animation Studios, who had made dozens of highly successful hand animated films over several decades. As recently as the early 90s, they had created classics such as Beauty and the Beast and The Lion King, but something started going wrong as they entered the 2000s. By the time Home on the Range released, things had come to a head. A critical and commercial failure, Home on the Range was a sign of the times, a signal that a pivot had happened somewhere in the animation industry and Disney Animation had missed it. That pivot began a mere nine years earlier.

That brings us to our second movie, one I can almost guarantee you’ve watched: Toy Story. Considered one of the greatest films of all time, its a classic that spawned a massive franchise. While the Toy Story IP is considered a bedrock of the Disney media world today, it didn’t start that way. Toy Story wasn’t even made by Disney. It was made by a little studio known as Pixar, a spinoff of Lucasfilm purchased by Steve Jobs in 1986. Pixar wanted to be in the business of creating animated films. There were a few problems though. First, Pixar had never released a full length animated movie before Toy Story. Second, they were going up against Disney, who had built an empire on decades of highly popular and successful animation films. But the third and biggest difference is that Pixar wanted to make a film using only computer generated imagery, or CGI.

No one had ever released a film that was 100% CGI. It was unheard of. Up to this point, the primary method of animating a film was through hand-animation, a laborious process that required hundreds of skilled animators and artists. Pixar’s tiny team was proposing that they could create a compelling, emotional, artistic film using CG, which at the time quite frankly looked awful.

Yet despite the odds and the limitations of the technology, Toy Story was a smash hit. So was A Bug’s Life. And Toy Story 2, and Monster’s Inc., and Finding Nemo, and The Incredibles. Between 1995 and 2004, Pixar released hit after hit, all made 100% with CGI.

Meanwhile, Disney Animation films, still done by hand, began to inexplicably see lower and lower numbers at the box office. Movies like Dinosaur and Treasure Planet received mixed reviews and struggled to reach the high box office numbers of previous films.

Home on the Range was the final nail in the coffin. Disney Animation once again released a hand animated movie, despite the writing on the wall that things were going south. Upon release, critics and audiences reacted negatively. Disney Animation declared that Home on the Range would be their final hand animated film, and that moving forward they would plan on all films being made with CGI. While there have been some exceptions to this, overall CGI has become the near-universal way that animated films are created over the last 20 years. Not only that, CGI is used in nearly every live action film as well, from The Avengers to Avatar to The Lord of the Rings.

Lesson 1: Technology Moves Fast

Nine years. That’s all it took for an industry to go from hand animation to CGI as the default choice. When the ChatGPT moment happened, many individuals made bold predictions that within a decade, practically everyone would become reliant on AI. Skeptics shook their head, believing such a seismic shift in a short period of time was impossible. But history has demonstrated that impactful technologies can rapidly proliferate in an almost pandemic-like manner. Today we’ve discussing animation, but similar tales can be found in the history of cars, airplanes, electricity, railroads, computers, the internet, and more. AI appears to be on that same track.

However, the story of Disney and Pixar has unique relevance for business leaders looking to evaluate the risks of losing the AI race. There are three questions from the Pixar story that have massive implications for AI adoption.

How did Pixar make CGI work despite its limitations?

Why did Disney Animation movies stop working?

How did Disney Animation end up winning by the end?

Let’s answer those three questions with three lessons for AI adopters.

Lesson 2: Early Adopters Work Within the Constraints of Technology

How did Pixar sell CGI movies despite the shortcomings of the technology?

Early CGI was terrible, but Pixar was able to successfully work with those limitations. If CGI looked like plastic on screen, then the solution was to make a movie about plastic toys. Thus, Toy Story was their first movie by necessity. By working within the constraints of technology, Pixar was still able to express artistic creativity and build an emotionally impactful film. The success of the first film laid the foundation for them to slowly expand the ability of CGI. Monster’s Inc. added fur simulations, Finding Nemo added water simulations, and so on. By iteratively building upon the technology and slowly expanding its capabilities, Pixar was able to overcome those early limitations.

Lesson 3: Late Adopters Lose the Talent War

The success of Pixar is only half of the equation. Why did Disney Animation films stop winning?

There is nothing inherently wrong with hand animation. Clearly, decades of success proved that hand animation worked. Movies like The Lion King are still an enjoyable watch to this day. However, Home on the Range is not. What went wrong?

The first and obvious argument is that the efficiency gains from switching to CGI allowed Pixar to serve animated films at a lower cost, at a faster rate, with a greater ability to rapidly iterate and improve scenes through their technological advantage.

Such a sterile view of art through the lens of raw efficiency gains falls short of answering our question though.

I propose a second, more important reason Disney Animation films stopped working.

Disney Animation lost their talent edge due to falling behind the technology curve.

Imagine this. You’re a recent college graduate from the top art program in the nation. Your lifelong ambition is to make animation movies. Growing up, you had watched Disney and assumed a job at Disney Animation Studios was the logical endpoint for the journey you began by studying art and animation.

But as you drew close to graduation, a new kid on the block appears: Pixar. Their movies are a breath of fresh air. Modern humor. Heartwarming stories. But most interestingly, they seem to be forging the path of where animation is going. You have an opportunity to get in on the ground floor and create art with cutting edge technology. Or, you can go work at Disney Animation and painstakingly hand draw background characters twelve times per second of film.

The choice is obvious to us, and it was obvious to talent at the time. The best artists, screenwriters, and actors were all drawn to CGI as the new medium. They saw the writing on the wall. CGI was the future of filmmaking. Not only are modern animated films made with CGI, but so is practically every other movie as well. Whether you are watching horror, comedy, drama, or action, CGI is used to enhance hundreds of scenes across cinema.

By losing the talent war, Disney was left with second tier talent and bureaucracy that refused to take risks and thought making an animated film in the style of a Western was a safe bet.

The same applies to companies today. Employees are not stupid. They see the writing on the wall as well. They want to work at companies that are quick to adopt AI and position themselves as leaders of the new technology wave.

How can large enterprises ensure their employees are positioned to succeed with AI?

The same way Disney did. After announcing they were ditching hand animation, they gave free training courses to all of their hand animators so they could learn CGI if they wanted. Many employees took the courses, re-skilled, and went on to have successful careers. Some employees chose not to, and those animators had to fight over the few remaining opportunities for hand animation.

Lesson 4: Money Can Save Late Adopters

The story doesn’t end in 2004 though. The Mouse always wins.

The truth is, I’ve left out some key details in my storytelling. During this nine year period from 1995 to 2004, Disney funded all of Pixar’s movies, and handled distribution for them as well. A complicated legal relationship between the two meant that Pixar was financially bound to Disney.

And in 2006, Disney bought Pixar outright, finally bringing the circle to a close.

Disney immediately moved the best talent into Disney Animation, where they began working on new animated films built with CGI. Within a few years, Disney Animation was back on top, with films like Tangled and Frozen dominating the box office. Meanwhile, Pixar powered through with a few more hits before slowly experiencing the same talent drain problem Disney had faced earlier. Today, Pixar struggles to make successful movies that are not sequels, with several notable box office duds in recent years. Sony Picture Animations is now the ‘new’ kid on the block, taking risks with new technologies and art styles in movies like Into the Spiderverse and KPop Demon Hunters.

Companies are discovering the same lesson today with AI. Money can atone for the sins of late adoption. Mark Zuckerberg found this out the hard way when Meta began falling behind (see our story on their trillion dollar mistake). Acqui-hires are happening left and right, with big companies snapping up smaller startups to stay competitive. If you’re a big company falling behind, consider injecting your company with new life via an acquisition. If you’re a small startup jostling for attention, consider carefully if you want to go to war with the big players or aim for acquisition.

The bottom line: The race to adopt AI is more intense than ever. Companies recognize the risks of falling behind include not just losing the technology edge, but also the people edge. As organizations evaluate how to move forward, acquisition and talent attraction remain key cornerstones for staying ahead of the competition.

If we continually A/B test food, we eventually end with chicken nuggets.
If we continually A/B test our movies, we eventually end up with p0rngraphy.
If we continually A/B test our music, we eventually end up with a four-second TikTok sound and a dance challenge.

But if we continually A/B test our sense of humor, we eventually end up with a p0rnographic chicken nugget, desperate to go viral, dancing in a TikTok challenge.

A/B testing typically evaluates small and short-term changes. In scenarios where you can steadily improve in one direction with no bumps or disjointed gaps along the way, this works splendidly.

Unfortunately, your life doesn’t work this way. Because your life doesn’t work this way, your choices shouldn’t operate this way. Your AI shouldn’t operate this way either.

Machine learning is not like Donkey Kong where the highest score wins. It’s more like golf, but the ideal score is a zero. However, it’s very easy to get stuck in a “local minimum”.

My beautiful drawing above illustrates that each little stick guy is trying to win with the lowest score possible. It achieves this by walking downhill wherever it finds itself. That’s great if you happen to start in the blue zone. You’ll walk down and achieve the lowest score.

If you’re in the red zone though? You’re toast. No matter how hard you try to walk further downhill you can’t win.

For example, you can interview for a better job in your city. You might find something better than what you have. Or… you can move cities, or states, or countries. A drastic change will land you somewhere else entirely on the curve. These larger changes are not easy to A/B test, making them risky, or at the very least they feel risky.

A/B testing is probing the ground to see which way is downhill. That’s great, but it can only get you so far. If you don’t start in the ideal place though, you’ll need to pair it with a different strategy.

Escaping Your Local Minimum, and Helping Your AI Escape as Well

In machine learning, we fight against local minimum with two standard playbooks:

• Explore wildly different approaches simultaneously

• Continue experimenting, but “take larger steps” that are more drastic

This is great advice for machine learning, but time still moves forward linearly and you can’t clone yourself in simulations (yet), so these approaches can only help so much.

For AI, and I believe this is true for ourselves as well, the key is the combination of knowledge and taste. Phenomenal AI today is influenced by data (knowledge and experience), but directed by taste.

Experts and creators act as curators of the AI system’s “taste” and “feel.” AI system designers shape experience through response quality, interaction patterns, and the intangible sense of fit as humans adapt to it over the long term. This curatorial vision requires both technical fluency and user empathy.

Essentially, expert / creator “vibe checks” capture qualities of tone, fit, and taste that benchmarks alone cannot measure. Taste is an underappreciated talent in the world of subjective AI experiences. Those with vision and taste can create unseen productivity enhancements or beautiful artistic experiences that users didn’t even know they wanted.

The nature of AI is that it magnifies the decisions of the people who wield it. Without taste and expertise you end up with AI slop. With them, the people + AI combination is unstoppable.

I believe this same principle holds true for our lives as well. Have you ever seen someone stuck in their life? Have you ever felt stuck? Taste and expertise pull us out to try something new. Because we’re restricted by our own experiences, the easiest way to drop ourselves into a new zone is either the bold agency to try something new (most common in the young), or tapping into others with taste and expertise to provide insights to push us away from A/B testing and make the jump.

Also, after playing with GPT-5 for two days, I canceled my OpenAI Pro subscription. The only substantive difference between the $20 Plus and the $200 Pro level is the advanced research mode. After running experiments, xAI’s deep research seems to be on par or better than OpenAI’s.

OpenAI now hosts a variety of models, splitting up their infrastructure and increasing both costs and brittleness. With the GPT-5 release, they removed all other models from the app except for the three GPT-5 variants. Plus users are left with smaller context windows and more restrictive thinking models caps than they had in the GPT-4 era, which I’m sure will cause some frustration. From the API side, I expect a wave of model deprecations over the next six months to consolidate their hosting into a more manageable set, keeping cost growth more in check.

Brief GPT-5 Recap:

• As expected, the all-in-one approach delivers an upgrade across most dimensions.

• Pro ($200/month) isn’t really worth it anymore unless you want access to the old context window lengths that were removed, or you prefer its deep research over xAI, Gemini, or Claude.

• In early testing, GPT-5’s coding capabilities are finally competitive with Claude’s coding models. Friends who have used it more extensively say that if you structure your codebase in AI-friendly ways, GPT-5 really shines.

• Rationalizing the model lineup and hiding them under a single “GPT-5” connection should help OpenAI keep operating costs more reasonable.

Overall, it’s a bit of a let down versus the hype, but still a step forward.

What About the Accelerationist Curve?

Global economic growth is precariously perched and praying that AI delivers. Is the accelerationist curve real?

(Derek Thompson post)

A friend called yesterday to discuss the release. He reminded me of a prediction I made a little over 18 months ago: the industry likely had two major step-changes left before plateauing until the next categorical breakthrough. After the multimodal, deep-reasoning, and efficiency gains, we may have reached that point.

(See The Information article here.)

On the other end of the spectrum, while OpenAI is struggling with marginal improvements, Meta drops a blog post out of the blue claiming that AI is now improving AI:

(Article. Excellent basic html choice.)

Where that Leaves Us

Despite the signs of incrementalism, I’m still insanely bullish.

I continue to believe that AGI will be achieved through a more complex architecture combining both LLMs and other learning models. As I’ve mentioned before, I never subscribed to the idea “just train an LLM enough and AGI will arrive.” This type of architectural breakthrough may occur soon or years from now. If soon, it will feel like it was inevitable to many. If years from now, the era of incrementalism will feel inevitable. Both sides appear loud and overconfident.

Regardless of the path of AGI advancement, after the GPT-4 series and the general multi-modality (audio, image, and voice) quality increased significantly, I began repeating a line I still believe:

“Even if AI advancements froze today, we’d have at least 10 years of building to capture the benefit of this technology.”

Like the internet or smart phones, it will take time to discover, build, and iterate the best use cases. Opportunities are everywhere to adapt the current generation of models where they consistently outperform humans, even experts.

Even if the era of AI researchers begins to fade, the era of AI builders has only begun.

First, let’s take a trip down memory lane to understand how single individuals can create or destroy economic value. Steve Ballmer, as CEO of Microsoft, led the company down unproductive paths. The Zune mp3 player was the central symbol of Microsoft’s lethargy. “Like Apple, but worse” was the general sentiment.

In August 2013 Steve Ballmer unexpectedly announced his retirement from Microsoft. How did markets react? Microsoft’s value increased by $18 BILLION dollars. Microsoft hadn’t even announced who would replace him. The market consensus on the difference between Steve Ballmer and an unknown CEO was $18 billion. Satya Nadella eventually stepped in, transformed Microsoft’s strategy, and now the company is worth many times its 2013 valuation.

(Image from Marginal Revolution)

Take any moment between five years ago and two months ago. Imagine you’re a cutting-edge LLM researcher looking to make a lasting impact in AI. Meta reaches out to you to join their AI team. There’s Meta, but what are your other options?

1. OpenAI – Post-November 2022, it’s an obviously attractive choice, but even before then it was a cutting-edge AI startup.

2. Anthropic – Solid, ambitious, and intelligent teams focused on the cutting edge.

3. Google – Slow, bureaucratic giant, but with unreal amounts of talent and resources.

4. xAI – Really smart people attacking problems at an almost maniacal pace.

5. Niche players (e.g., Midjourney, Mistral) – Cool if you’re into their niche and culture.

6. Apple – Jk. Apple doesn’t do AI. They’re praying on the sidelines that AI becomes commoditized.

Besides Apple, you have some interesting options.

How does Meta stack up? You use AI (not Google, you’re not a boomer) to review Meta’s public statements about AI and discover their Chief AI Scientist has some interesting opinions:

• “This AI Pioneer Thinks AI Is Dumber Than a Cat”

• “LLMs can do none of those [reason, plan, etc.] or they can only do them in a very primitive way and they don’t really understand the physical world. They don’t really have persistent memory. They can’t really reason and they certainly can’t plan.”

Meta’s Chief AI Scientist calls LLMs dumb parrots repeatedly and insists they are not the future. As a cutting-edge researcher… you don’t agree. LLMs are an incredible technology with limitations, but still amazing potential. Your own research has shown this.

You, along with most other teams, picked a different horse to bet on.

Meta’s Reaction

Meta eventually found themselves in last place (Apple has been DQ’d). They poured in money and turned to open source (not a bad strategy given their ecosystem alignment), but Meta still lacks a cutting-edge research team.

Eventually, Meta (likely Zuckerberg?) needed to stop losing or give up. They had not a single major model in the top 30 on arena leaderboards.

Meta began throwing amounts of money, once reserved for sports superstars, at everyone else’s top researchers. How did OpenAI feel about that?

(I started with TBPN’s post and had some fun with AI and photoshop. 😆)

For the detailed breakdown on the Meta Superintelligence team, click the image below:

My Takeaways

I am impressed that Meta doubled down instead of giving up. It’s a gutsy move. Also, I don’t completely disagree with Yann LeCun! Ultimate superintelligence likely requires a larger reasoning and reinforcement framework beyond just an LLM. In my opinion, there are too many theoretical weaknesses to reach that point with only language token reasoning. However, I still believe LLMs will be a significant or even majority component of that larger intelligence framework. I believe LeCun got lost in ideal future purity over today’s practicality.

Imagine a world where the Chief AI Scientist is an LLM bull, and Meta attracts top-tier researchers early. Without exaggeration, this could have been worth a trillion dollars to Meta.

Who are the candidates for the greatest economic value destruction by an employee of a company?

Each step is a 1-2 orders of magnitude change:

Yann LeCun >> Steve Ballmer >> Stephen Colbert

Then again, they’re in the arena. Or at least, they’re in much bigger arenas than I am.

There is a lot of context around this viral moment, so please bear with me as I share the story leading up to the event. I think it will clarify a lot of questions around how my team interviews candidates, what we had already done to modify interviews in an AI-saturated world, and my thoughts in the aftermath of the event.

Note: As always, this is 100% handwritten by me, with no AI.

Subscribe now

SOME BACKGROUND

I was fortunate enough to have a new position opened on my team. Seeing the crushing workload of other engineers at my company, I knew I was lucky to be given an opportunity to bring on an additional pair of hands. I also felt proud that I was doing a small part to combat the challenging job environment developers face. I looked forward to helping one individual escape the unpleasant anxiety and stress that unemployment brings.

My excitement was tempered by my knowledge that hiring is a challenging process in 2025. Outdated practices, awkward interactions, and of course AI cheating were all present in my mind as things to be cognizant of as I designed the interview process. I biased towards interviewing locally first, knowing that for most people, the interviewing experience is more pleasant in person. I also knew you couldn’t cheat with AI in person.

I interviewed some amazing people here in Utah. Some of them may be reading this! Unfortunately, none had quite the blend of skills and experience we were looking for. I decided to cast our net wider and started interviewing remote candidates while setting a clear expectation that this was an onsite role and would require relocation to Utah. Many of those remote candidates were also wonderful folk.

RESUME GRINDING

I read through at least 100 resumes during our search. The resumes were a foreshadowing of what was to come. At first they all looked fine. Some good, some weak. But on resume number 60, I noticed something odd. My eyes, now seasoned, noted that this resume felt… off. This guy worked at Nvidia and Uber, and now wanted what was basically a Senior engineer role? This would be a serious step down. That made no sense. Searching for his LinkedIn and Github confirmed my suspicion, as both had no history on them. “This is a fake”, I told my recruiter. He was surprised but not shocked. “We’ve seen this a few times,” he nodded knowingly. A few times!? How many of the resumes in front of me had been fabricated by AI? I ended up spotting one or two more deceptive resumes by the end, which makes me think I definitely missed some fakes.

INTERVIEW STRUCTURE

When tasked with hiring someone, I knew there was no textbook on what our interview structure should look like for an AI team. To what degree should we allow the use of AI in our coding challenge? What kind of experience were we looking for? How could we find out in under an hour what a candidate actually knew? I’ll dive into the nitty gritty of our technical interview soon, but first let’s look at the bigger picture of the entire interview process.

I had originally structured our first two rounds of interviews to focus on experience in the first round, then do a technical interview as a follow up. I thought this would be the better approach, given that experience is harder to fake. However, we found that a lot of candidates were good at sharing their experience, but ended up struggling with the technical interview. I’m proud of those candidates for their honest attempts without cheating their way through it. While they may not have gotten the job, they should sleep well at night knowing they are honest people.
We then decided to switch it up and do the technical interview before the experience interview. This format would have sped things up a lot with previous candidates who got far only to hit a wall with the technical interview, so I was optimistic on this approach. I should also note that after these first two interviews, regardless of order, there is a 3rd round interview that does a DEEP dive into experience and culture fit, along with a lot of reference calls. After that is an intense process of blue teams and red teams that compete to persuade executives to approve or not approve hiring a candidate. There are also screening rounds by our recruiters before I even look at candidates and get them started on the interviews. It’s very hard for anyone to completely fake their way into working at our company.

Ironically, our cheater happened on the first instance of switching to doing the technical interview first. It makes me wonder how it would have gone if we had done the reverse order. Would I have rejected him during that initial experience interview? Let’s now walk through the actual technical interview, and how the cheater performed throughout the process.

PART ONE: TECHNICAL DISCUSSION

The first half of my technical interview is what I like to call technical discussion. I ask a variety of open-ended questions, many of which ask the candidate to describe the pros and cons of two competing or complimentary technical concepts or problems in the world of AI engineering. I’ve found that candidates with experience do well on these questions. They can describe the problems of each approach, yet also the benefits they present. The tradeoffs are nuanced. Good answers come from people who have gotten their hands dirty with both.

Early on, I noted that the candidate seemed to have serious communication skill challenges. His answers were mumbled and rambling, with little to no depth. There was no passion.

“Poor communication skills”, my notes say, “not excited about him.”

It got weirder. For example, I asked him to explain MCP (a communication protocol used to standardize AI tool use), and he described it as a ‘father/child’ relationship. An unusual analogy, to say the least. He also claimed it facilitated multi-agent collaboration. That’s a stretch. While MCP can be used for agent to agent communication, that is not it’s designed purpose.

PART TWO: CODING CHALLENGE

After how poorly part one had gone, I had already mentally marked him as no. However, in good faith I went forward with the 2nd half of the interview. (I’ve always found the concept of cutting someone off in the middle of an interview distasteful. It’s terribly disheartening for someone who puts in a lot of work. Being rejected a day later lets you assume the interviewer found a better candidate rather than just immediately wrote you off.)

The 2nd half of the coding interview is a simple coding challenge involving data manipulation of a JSON object. It does not require any sort of dynamic programming skills or re-ordering tree structures. It reflects one of the most common scenarios we face on a daily basis, which is handling JSON data with Python.

As the role is on our AI team, my AI coding rules are somewhat progressive IMO. Here is the exact text we share with candidates:

You can use tab-complete. You can ask AI for help with Python syntax or specific problems. Do not one-shot the problem with AI. Do not paste the problem into your IDE, or else tab-complete will one shot it. The goal is to see your thinking process.

Over the dozens of interviews I’ve given, I’ve seen a huge range of behavior here. Some candidates forgo using AI entirely. Others heavily rely on it during the interview. I’ve seen some candidates get completely lost following the AI, while others have tab-completed their way to victory. At my current company, engineers are expected to use AI coding on a daily basis, so I feel this challenge reflects our day to day reality. And while many on the internet may opine that restricting wholesale use of AI in this interview is hypocritical, I believe this is a good approximation of the reality of working in large, mature projects, given the 30 minute constraint of the coding challenge. It lets us see both their ability to use AI, and also their ability to understand code. We are not in the business of hiring people who blindly yeet AI generated code into the repo with no understanding of what it means.

THINGS GO SOUTH

This particular candidate read through the rules and problem description, then began sharing his screen. I immediately noticed an odd overlap on top of his screen. The candidate would drag it in and out of view to check what it said. After looking at it a few times, my suspicions were confirmed.

This candidate was using Cluely.

Cluely is a product built by a Columbia student who was kicked out of college after he built an AI interview cheating software that evolved into what is now called Cluely. It openly and brazenly proclaims itself as a ‘cheating’ tool for interviews, tests, and more. Their ads are extremely cringy, and both the VC backed company and founder have attracted significant controversy.

Cluely is supposed to be hidden during zoom calls. It’s supposed to hide itself from the view of the screenshare. Only for this candidate, he was apparently too dumb or too lazy to figure out how to do that.

“Jeremy, that’s some harsh language. Shouldn’t you support people using AI?” Normally, yes, I’m all for AI. Even as my blood pressure raised, I mentally gave him the benefit of the doubt. “This is exactly the kind of people we are looking for,” I told myself, “People who pro-actively use AI.” I assumed (and I hoped!) that he was using it as an assistant, not as a cheating tool. Yet the app was clearly showing a complete solution to my coding challenge, in direct contradiction of the rules he had just read.

So I confronted him. “What’s with the AI app?” I asked. He immediately froze. “Huh?” He feigned confusion. “The app on your screen with AI,” I nodded towards my computer. He made a big show of rotating the view through his different virtual desktops, not realizing that his AI cheating tool was clearly visible. “I…. I’m not using any AI tools!” (I’m cleaning up his grammar, his embarrassed mumbling was not that clear.)

I became angry. Angrier than I’ve been in a long time. “The rules of this coding challenge were clear. You cannot dump the problem into AI. This interview is over. Have a nice day.” I immediately ended the call, fuming. My biggest pet peeve by far is lying. I find it absolutely despicable. Passing off the work of either other humans or AI as your own work is unacceptable in my playbook. If he had owned up to using the AI my reaction would have been very different. Instead, our call ended 25 minutes early and turned into a 9 page rant that you see in front of you.

THOUGHTS IN THE AFTERMATH

A few interesting thoughts came to my mind as I grumbled and stewed over this frustrating experience. I had found myself a victim of the exact thing I had published a paper on in college: the inability of humans to detect the deceptive use of AI. I started thinking on how I could connect my research discoveries with my current experience.

My research had found a few interesting things:

1. Education and being on the lookout improves the ability of younger individuals to detect AI deception.

2. However, older people had the opposite experience. Constantly looking out for AI deception made them worse at spotting AI generated content.

If I began approaching every interview highly skeptical of candidates, would I get better at spotting cheating? Or would I become some pre-occupied with catching cheaters that I’d miss genuinely good candidates? Or worse, falsely accuse them of using AI.

My college research was built on the assumption that AI deception is a game of cat and mouse. Every time AI detection software is updated, the cheating software evolves and outwits it. Undoubtedly, we should work with our recruiting team to update our video conferencing software to add cheating detections, along with AI reviews for incoming resumes. I’ve also been approached by headhunters and external recruiting agencies who offer to solve my problem (for a price). But no matter how many softwares or services I use, I fundamentally believe you cannot fully immunize yourself from cheaters when interviewing remotely.

That’s why part of me wants to just put the hammer down and ban remote interviews. It would simplify the interviewing process. Interviewing candidates in person for onsite roles is so valuable. Much is lost over a two dimensional, half second delayed, head and shoulders view of an individual. Biggest of all, you can’t cheat in person. (At least not as easily. The majority of in person cheating is someone taking lots of bathroom breaks to check their phone.) Luckily, I’m done hiring for the next little bit. I’m happy I won’t have to face this problem again for a while. Interviewing candidates in person comes with it’s own set of challenges. If you can’t find someone locally that fits the role, you have to fly people in. Hopefully my next company is like Nvidia and has cash to burn on plane tickets!

The other thought I had was that the candidate was still terrible. Even with the AI. Or maybe because of it? I’m not sure. But he was not good. The Cluely subreddit r/cluely is filled with questions from users who clearly have no idea what is going on. This subredditor posted this minutes after my call with the cheater ended (in case you are wondering if it’s the same person, I doubt it. We use zoom, not teams):

I was caught using cluely in interview, not sure I've used browser teams call but the interviewee caught me using cluely

Grammar? Out the window. Are they asking for help? Not sure. Are they complaining about the software or just confessing their sins? Your guess is as good as mine.

In my head, there are two outcomes with my cheater.

1. He reforms from his cheating ways.

2. He figures out how to turn on the basic settings that actually hides the dang app. Or at least keeps it on another screen.

While I would have rejected him even if I hadn’t seen him blatantly using AI to cheat, I’m disturbed by the fact that I likely would not have realized he was cheating had he used two of his brain cells to configure the cheating software properly. It’s completely altered my worldview. Not to be dramatic, but seriously, what is real nowadays? I suspect that the value of in person conferences, in person interviews, and in person interactions will skyrocket in value. The flood of AI slop drowning our inboxes, our social media feeds, and now our interviews threatens to destroy our trust in the digital world. Private community chatrooms and servers are seeing a huge resurgence that is reminiscent of the early days of the internet, except now they are powered by advanced software applications like Discord and Slack.

Bottom line: The unethical use of AI should never be tolerated. Misrepresenting the work of either other humans or AI as your own is unacceptable. Always be transparent in your actions and authentic in your personality. When hiring candidates, look for those that can both use AI and understand the code that is generated.

… ooph.

After playing around with it since its release last year, how is Amazon AI shopping performing generally?

The responses? OK, but not great.

User experience? Pretty clunky.

The responses are pretty mid and don’t infer and adapt to my background very well. The overall functionality is not intuitive. Amazon, being one of the largest companies on the planet, probably has eighteen layers of safeguards to avoid anything remotely “risky” or too creative slipping through. The result is a strictly vanilla experience with a janky UI.

To be fair, nobody has really nailed intuitive AI interfaces yet. Trillions in market cap, but still struggling with an AI shopping experience. With their resources and talent, I’m sure it will steadily improve. Still disappointing, but also, unsurprising coming from a very large player in a new and fast moving space.

This leaves a massive opportunity to get AI shopping right. Your response, along with many many hundreds of others, might be, “I’ll just build a new general AI shopping experience to compete with… Perplexity… ChatGPT… Amazon… Google… and many many others.”

If you solve this at the general shopping level, your only moat is sprinting faster than the competition (see my earlier thoughts on the Gingerbread Man moat). Velocity would be your primary competitive advantage.

What’s a smarter alternative?

Build a purposeful and trustworthy AI shopping experience with specialized integrations. Don’t just slap a “Shopping” tab onto something like Perplexity or Google have. Instead, create a dedicated and purposeful AI shopping experience with specialized data sources and genuinely collaborate with the user.

For example:

Of the many possible data sources, you could connect your Oura ring, Garmin, and dieting app to get tailored recommendations. For example,

• What supplements should a triathlete consider that I’m not already taking?

• What diet adjustments could optimize my performance?

In this example, you can create the bespoke AI experience for physically active users.

Is this possible? Yes, I do it right now. Current platforms can do this in a clunky, DIY way. I personally hack it together, but I’m weird and comfortable jumping through hoops based on my knowledge of AI. For the non-weirdos out there? They’d kill for a polished intuitive product. Honestly, I’d love that too.

Now imagine that implementation across every shopping niche. Can a one-size fits all platform dominate all that specialized data and nuanced experience and win the day? Maybe. I wouldn’t count it out, but I’d put the odds below 50% for a few reasons:

• Some of the “pretty large” companies maintain agility, but the mega corps are inevitably slooooow

• Specialized integrations create too large of a surface area for one company to dominate effectively

• Design opportunities for niches are too numerous for single companies to dominate

This isn’t the same story as Amazon dominating the original web shopping experience. Companies will be going to where the users are instead of driving the user to their website. Accordingly, smaller players will eat away at the winners and a winner-take-all market is unlikely.

As the saying goes, predictions are hard, especially about the future. However, I expect a few mega AI shopping platforms with a swarm of smaller specialized AI shopping experiences better customized and responsive to your unique tastes and journey.

One of the most straightforward and common initial approaches is to pursue a segment where others aren’t playing. Many start this way with a “new idea” and “new approach.” This is wonderful right up until the moment you succeed and others, whether startups or existing players, decide that they want a piece of that.

• Pebble launched a successful and viral smartwatch and app. This was great right up until every major technology hardware company decided to build their own and Pebble went bankrupt.

“Nobody else is doing this yet” is not a moat. So, what can be a moat? Until recently, I’ve liked simplifying it down into three D’s: Distribution, Design, and Data. Before we go over why AI upended our apple cart of competitive moats, let’s walk through what these moats traditionally looked like.

Distribution consists of two abilities: a) acquire customers, b) and retain them.

• Uber was legendary at simultaneously building “two-sided” markets of drivers and users in new cities. A virtuous cycle of network effects sustained that advantage, but city entrance skills were key.

• Atlassian (Jira, Confluence, Trello, etc.) has minimal network effects across companies, but it reached a $20 billion market cap essentially without a sales team. Instead they pursued a word-of-mouth, marketing, and minimal friction sign up.

Data is one of the most commonly recognized moats. In some categories, it’s difficult to acquire customers without data, and it’s difficult to acquire data without customers. The data could come directly from your users, or they could come from a parent company or other relationship. The key to a data moat is that whoever possesses the most data will have the best product.

• Bloomberg has gained a reputation as the “comprehensive repository of accurate, real-time data.” That’s a long hill to climb to provide parity with Bloomberg, and your best hope to compete is to carve small niches.

• Because Google captures how customers respond to search results, they can serve the most relevant search results. The design underlying technology is important, but Google’s granularity for human behavior on search engines was unparalleled for decades.

Design moats create a customer and product experience that creates brand loyalty and equity.

• Headspace is a mindfulness and meditation app that lives on a principle of Design.

  • A user’s 10‑minute session does not improve because millions of others subscribe.

  • The catalog is proprietary, but guided‑audio IP is easy for Calm, Insight Timer, or YouTubers to replicate

  • Headspace’s moat relies on the instantly recognizable design language and trusted tone

• Squarespace’s library of high‑gloss, editorial‑grade templates sets the aesthetic bar for DIY sites.

  • One store or portfolio does not accrue extra value from another customer’s site, so little network effect exists.

  • Competing builders (Wix, Webflow) can match features, and page‑analytics.

  • What keeps many creatives paying is the perception that Squarespace = tasteful design out‑of‑the‑box.

AI is coming for your moats.

Data moat erosion – Because AI has picked up information across the entire internet (stored in its “latent space”), you often only need to tap into that data by providing customized user information and scenarios into the AI. Once you do that, you can generate high quality analyses, classifications, and decisions in return.

New product surface area – Google previously knew “everything” about you, yet in just a few months of ChatGPT use, users willingly divulged far more detailed and valuable personal information.

Tech Speed Unlock – Both incumbents and startups can develop technology faster than ever with AI. Incumbents can rapidly build and deploy AI features to existing customers. Startups can quickly launch competitive AI features within just a couple of months.

These examples are only the beginning. The landscape has shifted.

Only the Gingerbread Man Moat Remains

What is the last moat standing? Velocity. Go faster than the competition. Many used to poke fun of “velocity” as a competitive edge. Erik Torenberg from a16z reminded me of the strategy Ben Thompson once dubbed as “The Gingerbread Man” moat.

Cursor is considered one of the most successful companies of the AI era due to the speed of its high-quality implementation. The design is sharp, the data is helpful, and the distribution strategy is solid. However, if Cursor paused development for six months, it probably wouldn’t survive.

Currently, it feels like velocity is the only moat that matters in AI. If you pause for six months, your company is effectively dead. That wouldn't have spelled doom for any of today's trillion-dollar market-cap companies. Speed has always been important, but Amazon, Facebook, and others likely would have endured a six-month pause. Once your company gains visibility, expect a pack of rabid wolves at your back for years. It’s exciting for bystanders… and exhilarating and terrifying for participants.

How extreme is the velocity shift? GitHub Copilot just flipped to open source. They open-sourced their product because they were falling behind Cursor. If GitHub alone can't outperform Cursor, perhaps the open-source community can at least put them on equal footing.

It seems like a crazy move, but GitHub can afford it because Copilot is a critical moat supporting GitHub’s core product. If GitHub Copilot were a standalone product, they would be fighting for their lives.

“Work smart not hard” only works against dumb people, and AI unleashed a lot of smart in the world. The competition is working both very smart and very hard.

Velocity is the moat that matters. AI’s pack of rabid wolves is nipping at our heels. Life would probably be a lot chiller if I ran an HVAC business right now.

You can feel the echoes from earlier tech eras facing similar skepticism. Personal computers, internet, smartphones, and cloud all had their skeptics, but the pace plodded and the ferocity seemed a bit more tame than today. Perhaps this higher intensity is simply a combination higher velocity of progress coupled with social media contagion.

(To be fair, I was a major skeptic of the iPad. To this day I don’t understand the idea of “a laptop, but worse” unless you’re looking for a portable television or a mediocre gaming device.)

In the mid-2010s, an acquaintance at a neighborhood barbecue asked me about my work. In simple terms, I mentioned building a machine learning recommender system. He literally LOL’d. Absolutely incredulous that anyone would seriously invest in something called "machine learning" that would recommend products or TV shows you might like. It’s all another tech hype cycle to hoodwink the investors and management.

That reaction stuck with me. “Highly confident and dismissive on a subject he knows nothing about.” Where are my similar blind spots? Where are yours?

Today, millions of people hold similar attitudes toward AI, claiming that it fundamentally "doesn't do anything" in their field. You can argue that AI is currently overhyped (fair). You can insist that it's not quite ready for certain complex or nuanced applications (very fair). You might even voice genuine concerns about potential risks or ethical issues (fair-ish depending). Those are valid conversations.

But to categorically claim AI doesn't work is to ignore reality. Millions of individuals and thousands of organizations are actively leveraging AI today, creating tangible value across companies and industries. They’re busy grinding out incremental improvements in healthcare, finance, logistics, retail, and manufacturing.

This stubborn skepticism feels like déjà vu to earlier times. I remember the late '90s when plenty of smart, capable individuals insisted the internet was nothing more than a passing fancy, a solution looking for a problem. But this current dismissiveness feels even more out of step. It’s less “late '90s skepticism,” and more “2006 stubbornly insisting that the internet would never significantly impact retail, even as traditional stores visibly struggle.”

At this stage, confidently proclaiming that AI "doesn't work in my field" is bragging about one's ignorance. It's publicly diagnosing oneself with a severe skill issue.

Perhaps it's time to level up.

Subscribe now

Lately I’ve been enjoying more episodes from the a16z podcast, the official podcast from the venture capital firm Andreessen Horowitz. In a recent episode they reviewed the oft-debated question, “What is an AI Agent?”

Part of me agrees with one of the participants, Matt Bornstein, pointing out that straining over the definition of a made-up concept is a bit of a waste of time. He said, “I think we’ve all just been nerd-sniped. I just think it’s, like, humanities people love classifying and they draw kind of like fine distinctions between different types of things, entities, whatever.”

However, in our day-to-day work we settled on a clear, functional definition that has been helpful for coordinating work among teams. Instead of a buzzword to toss around, “agentic” and “agent” have practical definitions that help us align planning and actions across teams.

We settled on this definition only once we settled on an overarching framework of the underlying properties for what is and is not an agent.

A Definition and a Framework

For clarity, we break it down into a simple 2x2 matrix made up of two parts, orchestration and action.

Orchestration – Does the AI orchestrate the solution?

Action – Can AI be used to take an action?

That’s it. Apple’s Siri was an AI Router, using AI (primarily through NLP) to decide what to do. An AI workflow would be an LLM evaluating a refund request and making a recommendation based on company policy. An Agent does both the orchestration and can choose to take AI actions.

Below is a more detailed explanation, and we’ll provide more in-depth examples in follow-up posts (e.g., how “agentic” is a system?). But for today, we’ve found this a clean place to start from and enhance communication across the team.

Skip to Next Part Unless You Want the Gory Details

Briefly walking through each quadrant:

1. AI Orchestration + AI Actions = Agents

Full AI agents. An AI directs the processes and tools, actively determining how tasks are executed to achieve its goals. The AI agent decides which AI workflows and tools, along with their corresponding actions, to trigger.

Agents offer the greatest flexibility and adaptability. However, complexity, uncertainty, and risk also peak here.

2. AI Orchestration + Non-AI Actions = AI Routers

These systems orchestrate decisions, but rely on non-AI actions or tools. These tools are often predefined system processes. Virtual assistants such as Siri or Alexa typify this quadrant. They decide how to respond to user queries, yet the actual actions, such as playing a song, setting an alarm, or sending a message, remain relatively straightforward and predetermined.

AI Routers simplify execution. They retain adaptive benefits of AI orchestration, but with lower complexity and fewer uncertain outcomes due to the simpler actions executed.

3. Non-AI Orchestration + AI Actions = AI Workflows

This quadrant represents the traditional AI workflows where non-AI orchestration triggers and manages predefined workflows, and these workflows perform AI-driven actions.

AI Workflows allow your teams to harness powerful AI capabilities while controlling orchestration through predictable, rules-based methods. This reduces complexity and simplifies management, at the cost of adaptability.

4. Non-AI Orchestration + Non-AI Actions = Traditional Software Engineering (SWE)

Here we find traditional, rules-based software systems with no significant AI integration. An example would be an automated email-reporting system triggered each morning by a scheduled job. Both orchestration (cron jobs, scheduled triggers) and actions (sending a formatted report) follow predefined rules without AI-driven decisions or adaptive capabilities.

Traditional SWE offers maximum predictability, reliability, and simplicity, but lacks adaptability, innovation, and the powerful analytical leverage provided by AI.

No quadrant is inherently superior. The optimal choice depends on the objectives, the complexity of tasks, and the level of adaptability required.

Contrasting Definitions

How does our definition match up with many definitions offered by organizations, companies, academics, and your neighbor’s dog? Of all of the definitions we’ve seen bandied about, Anthropic’s definition aligns closest to what we feel like is an effective practical definition:

This definition captures a concrete distinction. Workflows follow deterministic paths, which are a series of steps executed consistently each time. In contrast, AI agents select and orchestrate among available workflows and tools, continuously evaluating options and outcomes until they achieve their goal, determine it's not worthwhile to proceed, or are deactivated. However, we diverge from Anthropic's framing in one key aspect: AI orchestration does not necessarily require an LLM. Instead, orchestration might be executed through alternative forms of AI, potentially ones better optimized for rapid feedback loops and decision making.

In the end, don’t get too hung up on definitions. These are evolving terms. However, we’ve settled on a practical definition that lines up relatively closely with one of the major companies and enhances communication among team members. Among the technical members, this works. If you’re looking for how to craft marketing hype for AI agents… that’s for a different post.

You can catch a16z’s podcast episode on “What is an AI Agent?” here if you’re interested: https://a16z.com/podcast/what-is-an-ai-agent/

(I highly recommend the podcast, especially if you’re the type who knows which episodes early on you should skip.)