Autoscaling Using KEDA Based On Kafka Lag

In this blog, we will discuss about scaling kubernetes deployments based on Apache Kafka topic lag with KEDA.

Kafka Lag Based Autoscaling with Keda

What is Apache Kafka ?

Kafka is a distributed message streaming platform that uses a publish and subscribe mechanism to stream the records or messages.

What is Keda ?

KEDA is a Kubernetes-based Event Driven Autoscaler. KEDA can be installed into any Kubernetes cluster and can work alongside standard Kubernetes components like the Horizontal Pod Autoscaler (HPA). When we install KEDA in any kubernetes cluster, two containers run. First one is keda-operator and second one is keda-operator-metrics-apiserver. Here, Role of keda-operator is to scale Deployments, Statefulsets or Rollouts from zero or to zero on no events. KEDA acts as a Kubernetes metrics server that exposes rich event data like queue length or stream lag to the HPA to drive scale out.The metric serving is the primary role of the second container i.e. keda-operator-metrics-apiserver.

Installing Keda (Using Helm)

1. Add helm repo & update it

helm repo add kedacore https://kedacore.github.io/charts
helm repo update

2. Install keda Helm chart

helm install keda kedacore/keda --version 1.4.2 --create-namespace --namespace keda

Now as have installed keda, let’s take an example and see how we can trigger autoscaling.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaledobject
  namespace: demo3
spec:
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    name: keda-test-demo3
    kind: Rollout
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-demo3.demo3:9092
        topic: test
        consumerGroup: console-consumer-76109
        lagThreshold: "300"
        offsetResetPolicy: latest
  maxReplicaCount: 10
  minReplicaCount: 1

In .spec.triggers section, we provide the informations that KEDA uses to trigger the autoscaling. Here are some of the parameters which can be used for triggering autoscaling.

keda autoscaling parameters & descriptions