π§ͺ AI Gateway labs
What's new β¨
β AI Gateway workshop provides a comprehensive learning experience using the Azure Portal
β Refactor most of the labs to use the new LLM built-in logging that supports streaming completions.
β Realtime API (Audio and Text) with Azure OpenAI π₯ experiments with the AOAI Realtime
β Realtime API (Audio and Text) with Azure OpenAI + MCP tools π₯ experiments with the AOAI Realtime + MCP
β Model Context Protocol (MCP) βοΈ experiments with the client authorization flow
β the FinOps Framework lab to manage AI budgets effectively π°
β Agentic β¨ experiments with Model Context Protocol (MCP).
β Agentic β¨ experiments with OpenAI Agents SDK.
β Agentic β¨ experiments with AI Agent Service from Azure AI Foundry.
Contents
- π§ AI Gateway
- π§ͺ Labs with AI Agents
- π§ͺ Labs with the Inference API
- π§ͺ Labs based on Azure OpenAI
- π Getting started
- π¨ Supporting tools
- ποΈ Well-Architected Framework
- π₯ Other Resources
The rapid pace of AI advances demands experimentation-driven approaches for organizations to remain at the forefront of the industry. With AI steadily becoming a game-changer for an array of sectors, maintaining a fast-paced innovation trajectory is crucial for businesses aiming to leverage its full potential.
AI services are predominantly accessed via APIs, underscoring the essential need for a robust and efficient API management strategy. This strategy is instrumental for maintaining control and governance over the consumption of AI models, data and tools.
With the expanding horizons of AI services and their seamless integration with APIs, there is a considerable demand for a comprehensive AI Gateway pattern, which broadens the core principles of API management. Aiming to accelerate the experimentation of advanced use cases and pave the road for further innovation in this rapidly evolving field. The well-architected principles of the AI Gateway provides a framework for the confident deployment of Intelligent Apps into production.
π§ AI Gateway
This repo explores the AI Gateway pattern through a series of experimental labs. The AI Gateway capabilities of Azure API Management plays a crucial role within these labs, handling AI services APIs, with security, reliability, performance, overall operational efficiency and cost controls. The primary focus is on Azure AI Foundry models, which sets the standard reference for Large Language Models (LLM). However, the same principles and design patterns could potentially be applied to any third party model.
Acknowledging the rising dominance of Python, particularly in the realm of AI, along with the powerful experimental capabilities of Jupyter notebooks, the following labs are structured around Jupyter notebooks, with step-by-step instructions with Python scripts, Bicep files and Azure API Management policies:
π§ͺ Labs with AI Agents
π§ͺ MCP Client Authorization
Playground to experiment the Model Context Protocol with the client authorization flow. In this flow, Azure API Management act both as an OAuth client connecting to the Microsoft Entra ID authorization server and as an OAuth authorization server for the MCP client (MCP inspector in this lab).
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Model Context Protocol (MCP)
Playground to experiment the Model Context Protocol with Azure API Management to enable plug & play of tools to LLMs. Leverages the credential manager for managing OAuth 2.0 tokens to backend tools and client token validation to ensure end-to-end authentication and authorization.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ OpenAI Agents
Playground to try the OpenAI Agents with Azure OpenAI models and API based tools controlled by Azure API Management.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ AI Agent Service
Use this playground to explore the Azure AI Agent Service, leveraging Azure API Management to control multiple services, including Azure OpenAI models, Logic Apps Workflows, and OpenAPI-based APIs.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Function calling
Playground to try the OpenAI function calling feature with an Azure Functions API that is also managed by Azure API Management.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Labs with the Inference API
π§ͺ AI Foundry Deepseek
Playground to try the Deepseek R1 model via the AI Model Inference from Azure AI Foundry. This lab uses the Azure AI Model Inference API and two APIM LLM policies: llm-token-limit and llm-emit-token-metric.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ SLM self-hosting (Phi-3)
Playground to try the self-hosted Phi-3 Small Language Model (SLM) through the Azure API Management self-hosted gateway with OpenAI API compatibility.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Labs based on Azure OpenAI
π§ͺ FinOps Framework
This playground leverages the FinOps Framework and Azure API Management to control AI costs. It uses the token limit policy for each product and integrates Azure Monitor alerts with Logic Apps to automatically disable APIM subscriptions that exceed cost quotas.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Backend pool load balancing - Available with Bicep and Terraform
Playground to try the built-in load balancing backend pool functionality of Azure API Management to either a list of Azure OpenAI endpoints or mock servers.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Token rate limiting
Playground to try the token rate limiting policy to one or more Azure OpenAI endpoints. When the token usage is exceeded, the caller receives a 429.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Token metrics emitting
Playground to try the emit token metric policy. The policy sends metrics to Application Insights about consumption of large language model tokens through Azure OpenAI Service APIs.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Semantic caching
Playground to try the semantic caching policy. Uses vector proximity of the prompt to previous requests and a specified similarity score threshold.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Access controlling
Playground to try the OAuth 2.0 authorization feature using identity provider to enable more fine-grained access to OpenAPI APIs by particular users or client.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Zero-to-Production
Playground to create a combination of several policies in an iterative approach. We start with load balancing, then progressively add token emitting, rate limiting, and, eventually, semantic caching. Each of these sets of policies is derived from other labs in this repo.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Model Routing
Playground to try routing to a backend based on Azure OpenAI model and version.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Vector searching
Playground to try the Retrieval Augmented Generation (RAG) pattern with Azure AI Search, Azure OpenAI embeddings and Azure OpenAI completions.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Built-in logging
Playground to try the buil-in LLM logging capabilities of Azure API Management. Logs requests into Azure Monitor to track details and token usage.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Message storing
Playground to test storing message details into Cosmos DB through the LLM Logging to event hub.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
π§ͺ Content Safety
Playground to try the content safety policy. The policy enforces content safety checks on any LLM prompts by transmitting them to the Azure AI Content Safety service before sending to the backend LLM API.
π¦Ύ Bicep β βοΈ Policy β π§Ύ Notebook
Backlog of Labs
This is a list of potential future labs to be developed.
- Logic Apps RAG
- PII handling
π Getting Started
Prerequisites
- Python 3.12 or later version installed
- VS Code installed with the Jupyter notebook extension enabled
- Python environment with the requirements.txt or run
pip install -r requirements.txtin your terminal - An Azure Subscription with Contributor + RBAC Administrator or Owner roles
- Azure CLI installed and Signed into your Azure subscription
Quickstart
- Clone this repo and configure your local machine with the prerequisites. Or just create a GitHub Codespace and run it on the browser or in VS Code.
- Navigate through the available labs and select one that best suits your needs. For starters we recommend the token rate limiting.
- Open the notebook and run the provided steps.
- Tailor the experiment according to your requirements. If you wish to contribute to our collective work, we would appreciate your submission of a pull request.
[!NOTE] πͺ² Please feel free to open a new issue if you find something that should be fixed or enhanced.
π¨ Supporting Tools
- Tracing - Invoke OpenAI API with trace enabled and returns the tracing information.
- Streaming - Invoke OpenAI API with stream enabled and returns response in chunks.
- AI-Gateway Mock server is designed to mimic the behavior and responses of the OpenAI API, thereby creating an efficient simulation environment suitable for testing and development purposes on the integration with Azure API Management and other use cases. The app.py can be customized to tailor the Mock server to specific use cases.
ποΈ Well-Architected Framework
The Azure Well-Architected Framework is a design framework that can improve the quality of a workload. The following table maps labs with the Well-Architected Framework pillars to set you up for success through architectural experimentation.
| Lab | Security | Reliability | Performance | Operations | Costs |
|---|---|---|---|---|---|
| Access controlling | β | ||||
| Backend pool load balancing | β | β | β | ||
| Semantic caching | β | β | |||
| Token rate limiting | β | β | |||
| Built-in LLM logging | β | ||||
| FinOps framework | β | β |
[!TIP] Check the Azure Well-Architected Framework perspective on Azure OpenAI Service for aditional guidance.
π₯ Other resources
For much more API Management content, please also check out APIM Love.
We believe that there may be valuable content that we are currently unaware of. We would greatly appreciate any suggestions or recommendations to enhance this list.
Disclaimer
[!IMPORTANT] This software is provided for demonstration purposes only. It is not intended to be relied upon for any purpose. The creators of this software make no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the software or the information, products, services, or related graphics contained in the software for any purpose. Any reliance you place on such information is therefore strictly at your own risk.