A/B testing is an experiment where you experiment with two options, and then move forward with whichever one performed better.
If we continually A/B test food, we eventually end with chicken nuggets.
If we continually A/B test our movies, we eventually end up with p0rngraphy.
If we continually A/B test our music, we eventually end up with a four-second TikTok sound and a dance challenge.
But if we continually A/B test our sense of humor, we eventually end up with a p0rnographic chicken nugget, desperate to go viral, dancing in a TikTok challenge.
A/B testing typically evaluates small and short-term changes. In scenarios where you can steadily improve in one direction with no bumps or disjointed gaps along the way, this works splendidly.
Unfortunately, your life doesn’t work this way. Because your life doesn’t work this way, your choices shouldn’t operate this way. Your AI shouldn’t operate this way either.
Machine learning is not like Donkey Kong where the highest score wins. It’s more like golf, but the ideal score is a zero. However, it’s very easy to get stuck in a “local minimum”.
My beautiful drawing above illustrates that each little stick guy is trying to win with the lowest score possible. It achieves this by walking downhill wherever it finds itself. That’s great if you happen to start in the blue zone. You’ll walk down and achieve the lowest score.
If you’re in the red zone though? You’re toast. No matter how hard you try to walk further downhill you can’t win.
For example, you can interview for a better job in your city. You might find something better than what you have. Or… you can move cities, or states, or countries. A drastic change will land you somewhere else entirely on the curve. These larger changes are not easy to A/B test, making them risky, or at the very least they feel risky.
A/B testing is probing the ground to see which way is downhill. That’s great, but it can only get you so far. If you don’t start in the ideal place though, you’ll need to pair it with a different strategy.
Escaping Your Local Minimum, and Helping Your AI Escape as Well
In machine learning, we fight against local minimum with two standard playbooks:
Explore wildly different approaches simultaneously
Continue experimenting, but “take larger steps” that are more drastic
This is great advice for machine learning, but time still moves forward linearly and you can’t clone yourself in simulations (yet), so these approaches can only help so much.
For AI, and I believe this is true for ourselves as well, the key is the combination of knowledge and taste. Phenomenal AI today is influenced by data (knowledge and experience), but directed by taste.
Experts and creators act as curators of the AI system’s “taste” and “feel.” AI system designers shape experience through response quality, interaction patterns, and the intangible sense of fit as humans adapt to it over the long term. This curatorial vision requires both technical fluency and user empathy.
Essentially, expert / creator “vibe checks” capture qualities of tone, fit, and taste that benchmarks alone cannot measure. Taste is an underappreciated talent in the world of subjective AI experiences. Those with vision and taste can create unseen productivity enhancements or beautiful artistic experiences that users didn’t even know they wanted.
The nature of AI is that it magnifies the decisions of the people who wield it. Without taste and expertise you end up with AI slop. With them, the people + AI combination is unstoppable.
I believe this same principle holds true for our lives as well. Have you ever seen someone stuck in their life? Have you ever felt stuck? Taste and expertise pull us out to try something new. Because we’re restricted by our own experiences, the easiest way to drop ourselves into a new zone is either the bold agency to try something new (most common in the young), or tapping into others with taste and expertise to provide insights to push us away from A/B testing and make the jump.