Addressing 3 Key Challenges When Integrating AI & Traditional Products

Businesses of all sizes - from startups to large enterprises - are rapidly embracing AI to develop new applications or add AI-powered features to existing products - many of which are API-dependent. This post will explore three common key challenges emerging with managing AI products and practical strategies to overcome them.

1. Murky Visibility into AI/API Performance:

Testing API performance is particularly challenging with AI-centric APIs for several reasons:

• AI’s inherent non-deterministic nature leads to inconsistent response patterns even when using the same inputs.
• The complexity of the algorithms fueling LLM responses differs entirely from a traditional API that executes fixed logic on a fixed dataset.
• AI models learn & adapt over time, leading to additional inconsistencies in results & performance.

Robust analytics about latency, error rates, and usage patterns are crucial for optimizing performance as AI models evolve and products mature.

Because of these factors, real-time observability of production environment performance is more critical when AI is part of your product stack versus traditional APIs. Therefore, robust analytics about latency, error rates, and usage patterns are crucial for optimizing performance as AI models and products mature.

Engineering teams and development managers should start evaluating and integrating next-gen observability solutions into their product stack to address the needs of traditional APIs and AI data models while minimizing technical overhead. Several alternative approaches exist:

• The most common low-code observability solutions use SDKs to integrate and optimize an existing tech stack.
• Even more helpful is real-time observability, which, with just a few lines of code, introduces cloud-based “container-native” observability that automatically monitors API traffic in any Kubernetes pods & docker container. (Want to know more? This recent post details how Revenium leverages eBPF to enable container-native monitoring of API traffic without requiring an API gateway, sidecar, or service mesh.)

2. Addressing Complexity & Scalability

As AI applications advance product complexity, managing scalability is increasingly challenging. Therefore, product managers and dev teams need robust solutions that provide more insight into rapid fluctuations in API usage, ensure consistent performance, and support dynamic user needs. Lacking this expanded performance visibility is equivalent to driving a speeding car on a foggy road.

However, with real-time user experience and app performance data, product managers and infrastructure engineers can proactively monitor and address performance issues. While traditional API and app monitoring tools monitoring can help to some degree, legacy approaches need to catch up in many areas. The most common gaps not addressed by legacy monitoring solutions include:

• Without a way to monitor and measure application performance down to individual customers, an incomplete view of the customer experience results.
• Traditional API level monitoring isn’t robust enough to provide insight into complex product ecosystems, including AI elements and apps.
• Traditional tools cannot measure and demonstrate service level adherence related to complex application performance requirements.
• Traditional tools do not enable cross-platform visibility for complex applications spanning multiple environments.

If you’d like to dig in further on the items above, we recently explored each of these gaps in a separate blog post.

3. Monetizing Dynamic Applications

Anyone subscriber to premium services from OpenAI knows that leveraging third-party AI models or developing & operating your own is costly! Although we’re in a bull rush of investment into new AI applications, any investment into AI-dependent features or products must eventually generate a return.

The product & engineering managers we’ve spoken to have described significant challenges just developing and releasing sound AI applications amidst massive competition. Monetization to recapture their investment is often an afterthought. Unfortunately, just like observability, monetizing AI applications is more complex than traditional applications and, therefore, best addressed in parallel with product development. Several unique characteristics drive AI monetization complexity:

• Highly variable costs - AI models can be resource-intensive, especially those involved in complex tasks such as natural language processing or image recognition. Monetizing AI APIs requires factoring in the highly variable costs of expensive computing resources. Determining a pricing model that accurately reflects these variable costs while remaining competitive and attractive to users is a delicate balancing act. Further, once an optimal pricing model is determined, developing flexible and scalable metering to monetize a complex product is a significant development effort. Our opinion (obviously!) is that relying on metering and monetization functionality engineered to be flexible and “future-proof” is superior to DIY alternatives.
• Experimentation and model training costs - AI models require continuous experimentation, training, and refinement to remain competitive and effective. Monetizing AI APIs involves covering the costs of serving customer API requests and factoring in the ongoing expenses related to model experimentation, fine-tuning, and adapting to changing data patterns. Accurately metering and tracking model development and refinement costs make it easier to price AI products.
• Scaling costs & performance guarantees - As mentioned above, competition in the AI space is exceptionally fierce, and winning lucrative customers increasingly requires financially backed performance guarantees. In addition to managing performance and reporting SLA failures, monetized APIs must consider the costs of associated performance guarantees in their pricing models.
• Rapidly changing market demands - There is no more rapidly changing market t than AI. As customer demands and supplier pricing models change, products require sophisticated agility and adaptable pricing features. To this end, Revenium has seen a recent uptick in the number of companies who want to explore our solution as they move from home-grown metering and usage-based billing solutions that were ‘good enough’ for many years but now are severely limiting their ability to compete.

Putting it all together

Are you curious how Revenium can improve observability and help monetize your AI products? Revenium co-founders Jason Cumberland and John D'Emic presented From Code to Cash: Using a Revenue Mesh to Monetize AI & ML APIs during Gravitee Edge 2023.

The video provides an overview of how Revenium can enable organizations to ease the integration of AI into traditional API products and shows a hands-on demo of the Monetization of an API-first proprietary LLM.

Btw, If you want to jump right to the product demo - start watching at 14:14

Summary

Enterprises and startups that embrace AI applications must address new development, product observability, and monetization challenges. Fortunately, solutions such as Revenium are ideal to help fill the gaps in current offerings without adding significant technical overhead.

If you’d like to try Revenium, create a free account (see the form below - no credit card required). Within minutes of connecting your APIs using our low-code agent, you’ll be on the way to simplifying your metering & monetization and exposing new insights about AI products & API performance.