Kubernetes Storage Performance Testing Guide: Comparing AKS and Azure Local Storage

This guide explains how to perform storage performance testing in Kubernetes environments using FIO (Flexible I/O Tester). We’ll compare storage performance between Azure Kubernetes Service (AKS) and Azure Local (Previously known as Azure Stack HCI)

Prerequisites

• Access to both AKS and Azure Local clusters

• kubectl configured to access both clusters

• Storage classes available in both environments

Step 1: Understanding Storage Classes

Azure Local

AKS

Step 2: Create the PersistentVolumeClaim

Create a file named pvc.yaml:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: fio-pvc
spec:
  storageClassName: managed-premium  # Use 'default' for Azure Local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 250Gi

Apply the PVC:

kubectl apply -f pvc.yaml

Step 3: Create the Benchmark Job

Create a file named fio-job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: fio-benchmark
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: fio-benchmark
          image: ubuntu:22.04
          command:
            - bash
            - -c
            - |
              apt update
              apt install -y fio
              echo "=== Storage Performance Test ==="
              date
              fio --name=benchmark_test \
                  --filename=/mnt/data/testfile \
                  --rw=write \
                  --bs=1M \
                  --ioengine=libaio \
                  --iodepth=64 \
                  --direct=1 \
                  --size=200G \
                  --numjobs=8 \
                  --runtime=300 \
                  --time_based \
                  --group_reporting
          volumeMounts:
            - mountPath: /mnt/data
              name: fio-mount
      volumes:
        - name: fio-mount
          persistentVolumeClaim:
            claimName: fio-pvc

Step 4: Run the Benchmark

1. Apply the job:

kubectl apply -f fio-job.yaml

2. Monitor the job:

kubectl get jobs fio-benchmark

3. View the results:

kubectl logs job/fio-benchmark

Understanding the FIO Parameters

• --rw=write: Test write performance

• --bs=1M: Use 1MB block size for large sequential writes

• --ioengine=libaio: Use Linux async I/O for better performance

• --iodepth=64: Queue depth for async I/O

• --direct=1: Use direct I/O, bypassing the OS cache

• --size=200G: Total size of the test file

• --numjobs=8: Number of parallel processes

• --runtime=300: Run test for 5 minutes

• --time_based: Run for the specified time rather than until size is reached

• --group_reporting: Aggregate results from all jobs

Interpreting Results

The output will include:

1. Write bandwidth (MB/s)

2. IOPS (Input/Output Operations Per Second)

3. Latency statistics

4. CPU utilization

5. I/O depth distribution

Key metrics to compare:

• Higher bandwidth indicates better throughput

• Lower latency suggests better response times

• Higher IOPS shows better small I/O performance

Cleanup

Remove the job and PVC after testing:

kubectl delete job fio-benchmark
kubectl delete pvc fio-pvc

Notes

• For AKS, consider using premium storage classes for better performance

• Azure Local uses local storage, which typically provides lower latency

• Results may vary based on node size, storage class, and cluster configuration