The Wild West: AI Conferences in 2025

Lessons Learned While Traveling

“This reminds me of blockchain four years ago” was what one participant remarked to me as we chatted after my presentation at the AI Agents conference in New York City. “What do you mean?” I asked him. He responded, “In the early days of blockchain, it was the wild west. We had people from every walk of life attending conferences. Today, it’s just a bunch of finance guys in suits.”

Every walk of life is a good description for what I saw in my visit to New York for this conference. You had hungry startups, proudly sharing the cutting edge discoveries and innovations they had built in the hopes of finding more users. The VC sharks were circling the waters, cornering you the moment you looked available and trying to extract as much information from you as possible about you and your company to see if you fit their profile. You had representatives from the big dogs: OpenAI, Anthropic, and Cohere, each trying to position themselves as the one true path to success with AI and agents. And you had a few brave souls such as myself from regular companies, with common reactions of shock when people realized I was neither a VC nor a startup. Alas, I simply represent a B2B company that has a shockingly high amount of annual revenue yet is practically unheard of outside of Utah.

Some things felt normal at this conference: the endless decks, the mingling around food, and cringy jokes from the hosts. Other things felt less normal. Someone kept bumping the projector screen button, causing it to constantly go up and down to the embarrassment of the hosts, not to mention various other technical difficulties with lights turning on and off, struggles to get slides to display, and more. Overall though, I walked away feeling that the conference experience had been positive and worth my time.

There are several themes I heard repeated throughout this ‘AI Agent’ conference. The most common themes included:

• Explaining the difference between an AI agent (which loops over tools) and an AI workflow (which is a one time call/response)

  • I was disappointed not to hear anyone try to bridge the difference between these two. I felt like Harrison Chase from LangChain has a great way of uniting these two ideas under the term ‘agentic systems’, but people seemed incredibly proud just to point out that there are, in fact, two ways of using LLMs. To me, this seemed obvious, but for others was perhaps a revelation.

  • One thing I was happy to hear was that people recognized you can mix and match agents and workflows. Agents can call workflows, and workflows can call agents.

• Pointing out the weaknesses of LLMs and then proceeding to explain how their platform fixed it

  • Generally there are four ways people ‘fixed’ LLMs:

  • 1. Improve the underlying model with finetuning, LORAs, or other methods

  • 2. Improved data and context

  • 3. Evals

  • 4. Frameworks that abstract how you interact with LLMs

• The commerce section showed an intense interest from the audience on the topic of GEO (Generative Engine Optimization) and how consumers are finding brands and products through ChatGPT

  • A Shopify exec in attendance who is responsible for many of their AI initiatives stated that the OpenAI + Shopify integration was rolled out ‘very fast’ and that they are now going to be playing catch up with advising brands on how to they can improve their ability to appear in ChatGPT results

  • The startup Profound is one of several that are leading the way in this new field, with some fascinating insights. Did you know ⅓ of all LLM citations are from comparative data? Think blog posts comparing the top 5 smartphones.

• Predicting the future

  • Lots of people had graphs showing where things are going. Most people are convinced that agents will be everywhere by next year. But once you got off stage and talked with people, whispered conversations discussed the reliability issues of agents, how none of us actually had any in production, and doubt as to whether the challenges of agents could ever truly be overcome

• AX (Agent Experience), AI-Native, and AI-First

  • A lot of buzzwords that boil down to the same thing: Are you making agents your TOP priority?

  • My favorite comparison was a comparison to Blockbuster calling themselves a ‘digital company’ by adding email and a website, compared to Netflix who went literally all in on digital

  • Likewise, companies that are truly ‘AI First’ need to make that sort of transformation

  • Unfortunately, this can have some public backlash if not done right (see Duolingo). Maybe don’t go bragging about it on social media

  • Maybe a better way of phrasing it would be “We are putting humans first by making sure they are 10x more effective via AI”. Doesn’t roll off the tongue though…

• Evals, evals, evals

  • Lots of emphasis on the importance of evals

  • Everybody liked to tote that evaluation was what set you apart. Unfortunately, evals are a difficult thing to get going, and I think many people exaggerated how much they are using evals

The bottom line:

I won’t bore you with repeating everything I heard, but suffice it to say that there is still much in flux in the world of AI. Time will tell which of these ideas and companies will stand the test of time. The best thing each of us can do is continue to be curious and constantly learning. As pieces of the puzzle solidify, the industry will coalesce around best practices and leaders, and new layers of complexity and opportunity will appear.