We recognized AI’s potential to drive both headway and headaches for our customers. However, we believed the opportunities outweighed the risks.

The launch of ChatGPT in 2022 was a watershed moment in Product Development. Large Language Models (LLMs), like ChatGPT, are radically changing the way humans interact with technology and creating new opportunities to solve problems and create value. With the rapid rise of AI, separating the buzz from the benefits isn’t always easy.

The potential applications of AI are vast. The opportunity to utilize AI is exciting but can be paralyzing for product teams unsure of how to take the first step. As the Chief Product Officer at Quantum Workplace, my team and I recognized AI’s potential to drive both headway and headaches for our customers. However, we believed the opportunities outweighed the risks.

In 2023, we launched six AI-powered features and have a dozen more AI-powered workflows currently in research and development. In this GrowthBit, I’ll share what we’ve learned about how to thoughtfully approach AI integration while staying true to the value proposition that attracted your customers in the first place.

At its core, AI can help empower your end users to do their best work. So start there.

Let your customers be your guide

I believe, for most industries, the best time to test AI within your products is now. Not because it’s trendy or newsworthy, but because the current potential to mitigate pain points and generate transformative value for your customers is too great to ignore.

At its core, AI can help empower your end users to do their best work. So start there. Think about your domain: what are the longstanding problems that your end users care about that your product has not been able to solve yet? What’s standing between them and making progress in their work? They’ve enlisted your product to help them do a job. Reframe AI as a solution set that can help you solve their problems.

Quantum Workplace Example:

Our platform helps HR managers monitor employee engagement in several ways, including surveys that solicit open-ended feedback. These surveys are an incredibly rich source, but it can take hours, days or even weeks for our customers to sift through this unstructured input, analyze it and weave it into a meaningful narrative.

This is the kind of complex, linguistic task that generative AI can do in minutes, so we focused our initial efforts on leveraging AI to alleviate the problem. We called this new feature a “Summary Assistant” and so far, AI is not only saving our users’ time; it’s yielding higher quality insights than a human can even when they take the time to try. And our users are blown away.

Ultimately, your product roadmap should be rooted in the biggest challenges affecting your users and the constituents they serve. Let their challenges guide your team’s exploration of AI’s potential within your products. Consider your buyer’s problems and think, “Of all the opportunities we could be chasing, which ones are going to provide the most leverage for us, and can AI help us with those problems?”

Begin by thinking holistically about your customers, market, and competitors to assess your long-term AI goals.

 

Start small, think big

The challenge with AI is that it can be applied in countless ways, across a wide array of applications, and require different levels of effort. Like everything, you gotta start somewhere. My advice is to begin by thinking holistically about your customers, market, and competitors to assess your long-term AI goals.

To clarify your goals, ask yourself: “If we were rebuilding our product today, would we build it the same and enhance it with AI, or would we build an entirely new tool with AI at its heart?” The answer should reflect your buyers’ needs, their adoption of AI, and the expected impact of AI on your market and competitors over time. Depending on your answer to that question, you can begin to define where and how AI might fit into your product strategy.

In the short term, I challenge you to start with a small, low-risk opportunity. One of the best things about AI is that it doesn’t need to be productized immediately. You can start experimenting with prototypes to determine whether AI will be effective at solving a problem.

Quantum Workplace Example:

With the “Summary Assistant” example above, we used two modern Product experimentation techniques to validate the idea. We first used a “fake door test” that hinted at the promise of this feature inside our product. Users that clicked through this “fake door” could opt-in to become Alpha testers of this exciting new feature. We also built a small proof-of-concept that allowed us to employ a “Wizard of Oz” test. Our Beta users could “use” this new feature, with the caveat they had to wait 24 hours for the Summary.

What felt real (albeit slow) to the user was actually our team – behind the metaphorical green curtain – using our Proof of Concept to create and manually insert comment summaries into the tool. This testing allowed us to prove the value of AI-generated comment summaries with real-world users long before we ever built a production-grade solution. You can likely even skip the proof-of-concept and just jump over to ChatGPT, Claude, or Anthropic to experiment but we knew employee comments could be sensitive information and the privacy provided by our Proof of Concept would be important, even to these early users.

If you’re not thinking about solving problems with AI, I’m willing to bet your competitors (or future competitors) are.

Be bold, but conscientious

Even if you don’t see AI as core to your product strategy or see demand from your users, the risk of ignoring its potential entirely could be significant. If you look at all your potential new opportunities and find that none are just now possible through AI, I encourage you to go back to the drawing board and think more divergently about the universe of opportunities. If you’re not thinking about solving problems with AI, I’m willing to bet your competitors (or future competitors) are.

Start poking around the edges of your product to pressure-test the assumption that AI has nothing to offer. Now is the time to take calculated risks and look for those low-stakes opportunities that AI can make possible in your domain.

Be bold, but temper that boldness with a clear internal framework for exploring, evaluating and integrating AI into your products. When Quantum started integrating AI into our platform, we addressed the market skepticism head-on by developing and publishing the principles that guide our AI R&D. Our “4 P’s” lay out the principles that guide our experimentation and innovation with respect to people, privacy, problems, and priorities.

Have a process in place that tests AI-enhanced products to ensure your product is getting better, not worse, with time.

Continuous testing and quality control

AI is evolving at a pace unmatched by most technologies. The models that inform the output are designed to improve over time. And the models themselves are changing every few months. This changes the way these models respond to the same set of prompts. We refer to this tendency to change as AI drift, and it’s like a ticking time bomb for product performance if you’re not watching.

Consistently tracking the effectiveness of AI in your products is necessary to ensure optimal product performance and alignment. Implementing regular assessments and refining your AI models based on user feedback and product output will help maintain standards and adapt to any shifts in the AI landscape. Stay proactive in monitoring and adjusting to keep your competitive edge.

Quantum Workplace Example:

At Quantum, we developed our own models that help to test the stability, context, relevance and overall quality of the outputs to monitor AI drift and its impact on our AI-driven workflows.

ChatGPT has already undergone a few generations of the underlying model since we released our first AI-powered feature. During that time, we saw AI-generated outputs change for our end users. To ensure product quality, we put considerable work and innovation into retraining the algorithm and recapturing the brevity that our users find useful. Essentially, we needed to double up on innovation, innovating once in developing AI-powered products, then again in developing dynamic processes for monitoring and quality control.

Combating AI drift can be difficult, but it’s worth it to have a process in place that tests AI-enhanced products to ensure your product is getting better, not worse, with time.

Here’s the bottom line.

AI has dominated news headlines, but the hype surrounding it is no reason to hesitate. Your product – and your customer base – can almost certainly benefit from it. In this ever-evolving landscape, product teams should approach AI opportunities with a thoughtful and proactive mindset. Start small, but always keep the bigger picture in focus, allowing customer needs and market dynamics to guide your AI initiatives.

Reframe this new generation of AI as a whole new set of tools that can help you solve longstanding problems for your customers. Let their challenges guide your exploration to thoughtful AI integration that drives real value.