5 min read kubernetes

Observe and Secure AI Traffic: Enter Agentgateway + Kgateway

Observe and Secure AI Traffic: Enter Agentgateway + Kgateway

You can write the cleanest code in the world with incredibly separated components while implementing scalability for proper performance optimization, but none of it matters if the packets flowing throughout the environment aren't correct or secure. It's the make-or-break in who can access an environment and what/who can interact with the application stack(s) running.

In this blog post, you'll learn about one key way to implement governance, security, network observability, and proper connectivity for LLMs and MCP with Agentgateway and Kgateway.

Prerequisites

To follow along with this blog post in a hands-on fashion, you will need:

  1. A Kubernetes cluster. If you don't have a config for one, you can use this AWS EKS HCL here: https://github.com/AdminTurnedDevOps/Kubernetes-Quickstart-Environments/tree/main/aws/eks
  2. Helm

If you don't want to follow along from a hands-on perspective, that's totally fine! There's a lot of theory and pragmatic thinking within this blog post to assist in your journey.

What Are LLMs

Before diving too deep, let’s take a quick section to talk about LLMs as a whole.

Prior to LLMs, you had standard AI and ML practices. You could take a collection of data sets, which could be anything from data on a DB to data in an Excel spreadsheet, feed them to your ML workflow, and the machine learning process would train a Model (I’ve oversimplified this, but this section is supposed to be a recap of sorts).

Taking it a step further, you could use transformer models with proper embeddings and feed something like a RAG to the transformer model so it has access to specific data for your application (and RAGs are still very much used in LLMs).

Now, we have Large Language Models, which are exactly what was described in the ML/training process described in the first paragraph of this section. The biggest difference is it’s A LOT of data (hence the name). It costs millions of dollars to train these models.

TLDR; when you’re interacting with an LLM, it’s a massive collection of data that you’re making API calls to.

So, what’s the problem?

The Biggest Concern For Organizations

When it comes to anything AI-related, whether you’re calling out to an LLM, using an agent tool like Claude Code or GitHub Copilot (think client/server architecture), or interacting with MCP servers, there’s no way “out of the box” to fully understand what’s happening with those packets that are flowing.

For example, let’s say you’re interacting with an MCP Server. How are you observing network traffic, considering the MCP Server is communicating with an LLM? How are you securing that traffic? Taking it a step further, what authentication and authorization policies are in place when interacting with the MCP Server? There’s no Observability or governance in any capacity. It’s very much a free-for-all all.

One of the major problems with AI and LLMs as a whole is that standard network administration and engineering have been completely overlooked, which is causing significant security concerns and reliability issues within organizations.

Enter agentgateway and kgateway.

Kgateway

Every application needs exposure, whether it's north/south traffic or east/west traffic. With north/south traffic, you need users to be able to route to an application properly. Instead of managing a bunch of cloud load balancers and Kubernetes Services, you can implement a Gateway and proper routing protocols (via the HTTPRoute object) to do everything from route traffic to specific applications to enable multi-cluster routing to canary deployments. With east/west traffic, these are pieces of the application stack that need to communicate with other parts of the application. It could even be clusters that need to communicate with each other.

kgateway is an open-source API gateway that's a Kubernetes-native ingress controller that's built on top of Kubernetes Gateway API.

A proper API gateway is important in a world where traffic, packets, and data are being sent to LLMs and MCP Servers without actually knowing what the data is or how secure it is (spoiler alert: by default, it's not).

Agentgateway

With the help of kgateway doing all of the API routing on the backend, kgateway helps you manage routes, policies, and listeners (inbound connectivity) when interacting with LLMs and MCP servers.

The goal of agentgateway is to have the ability to connect agents (a2a), LLMs, and MCP Servers in a secure and reliable fashion. With how popular LLMs and AI Agents have gotten, there's a ton of data going back and forth along with a lot of network packets flowing publicly. With the hype and the excitement, people forgot that at the end of the day, much like any other environment or application stack, underneath the hood LLMs and MCP Servers are 90% data-driven applications and networking from standard ingress/egress to API traffic routing.

Source: https://agentgateway.dev/docs/about/architecture/

agentgateway is an open-source tool and was recently donated to the Linux foundation

Configuring Kgateway + Agentgateway

Now that you know the "why" behind kgateway, agentgateway, and the need for secure and observable network connectivity in the world of LLMs and MCP, let's dive into the hands-on piece of setting up Kgateway and Agentgateway.

💡
You can run agentgateway by itself without kgateway, but if you do that, it'll run as a process locally based on a config that you feed the agentgateway comamnd vs being able to be accessed programmatically from Kubernetes.
  1. Deploy the Kubernetes Gateway API CRDs.
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml
💡
If you're not familiar with CRDs, this is the way to extend the Kubernetes API. All Kubernetes is underneath the hood is a set of APIs, and luckily, there's a built-in feature to extend them to interact with other APIs.
  1. Deploy the kgateway CRDs
helm upgrade -i --create-namespace --namespace kgateway-system --version v2.1.0-main \
kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
--set controller.image.pullPolicy=Always
  1. Install kgateway with Helm and use the true boolean to enable agentgateway.
helm upgrade -i --namespace kgateway-system --version v2.1.0-main kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
  --set agentGateway.enabled=true  \
  --set controller.image.pullPolicy=Always
  1. Ensure that everything is running as expected.
kubectl get all -n kgateway-system
NAME                            READY   STATUS    RESTARTS   AGE
pod/kgateway-644c8df94c-rkqc7   1/1     Running   0          37s

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kgateway   ClusterIP   10.100.251.75   <none>        9977/TCP   37s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kgateway   1/1     1            1           37s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/kgateway-644c8df94c   1         1         1       37s

Now that kgateway is installed with agentgateway enabled, we need to create an Ingress route to access agentgateway. To do that, we'll use the Gateway object along with the gateway class that was created during the kgateway installation process.

  1. Create the new Gateway for agentgateway.
kubectl apply -f- <<EOF
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: agentgateway
  namespace: kgateway-system
  labels:
    app: agentgateway
spec:
  gatewayClassName: agentgateway
  listeners:
  - protocol: HTTP
    port: 8080
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF
  1. Retrieve the gateway configuration, which will contain the public load balancer that you can use to reach agentgateway.
kubectl get gateway -n kgateway-system
NAME           CLASS          ADDRESS                                                                   PROGRAMMED   AGE
agentgateway   agentgateway   acab218459c824ff1b10b4ab1c974144-1482392063.us-east-1.elb.amazonaws.com   True         23s

You can now begin to create routes and use agentgateway programmatically for securely accessing LLMs and MCP servers!

Conclusion

Whether you're a DevOps Engineer, Platform Engineer, Software Engineer, or whatever other title exists, you need to think about network observability, routing, and security. It's just part of the job now, and for good reason. The name of the game is securing connectivity in all aspects of cloud and on-prem environments.

In an upcoming blog post, I will dive into the innards of agentgateway in terms of security, authentication, a2a, and ensuring proper observability and network connectivity between agents and MCP servers.

Michael Levan

Michael Levan