Kubernetes Storage Performance Testing Guide: Comparing AKS and Azure Local Storage
This guide explains how to perform storage performance testing in Kubernetes environments using FIO (Flexible I/O Tester). We’ll compare storage performance between Azure Kubernetes Service (AKS) and Azure Local (Previously known as Azure Stack HCI)
Prerequisites
Access to both AKS and Azure Local clusters
kubectl
configured to access both clustersStorage classes available in both environments
Step 1: Understanding Storage Classes
Azure Local
AKS
Step 2: Create the PersistentVolumeClaim
Create a file named pvc.yaml
:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: fio-pvc
spec:
storageClassName: managed-premium # Use 'default' for Azure Local
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Gi
Apply the PVC:
kubectl apply -f pvc.yaml
Step 3: Create the Benchmark Job
Create a file named fio-job.yaml
:
apiVersion: batch/v1
kind: Job
metadata:
name: fio-benchmark
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: fio-benchmark
image: ubuntu:22.04
command:
- bash
- -c
- |
apt update
apt install -y fio
echo "=== Storage Performance Test ==="
date
fio --name=benchmark_test \
--filename=/mnt/data/testfile \
--rw=write \
--bs=1M \
--ioengine=libaio \
--iodepth=64 \
--direct=1 \
--size=200G \
--numjobs=8 \
--runtime=300 \
--time_based \
--group_reporting
volumeMounts:
- mountPath: /mnt/data
name: fio-mount
volumes:
- name: fio-mount
persistentVolumeClaim:
claimName: fio-pvc
Step 4: Run the Benchmark
Apply the job:
kubectl apply -f fio-job.yaml
Monitor the job:
kubectl get jobs fio-benchmark
View the results:
kubectl logs job/fio-benchmark
Understanding the FIO Parameters
--rw=write
: Test write performance--bs=1M
: Use 1MB block size for large sequential writes--ioengine=libaio
: Use Linux async I/O for better performance--iodepth=64
: Queue depth for async I/O--direct=1
: Use direct I/O, bypassing the OS cache--size=200G
: Total size of the test file--numjobs=8
: Number of parallel processes--runtime=300
: Run test for 5 minutes--time_based
: Run for the specified time rather than until size is reached--group_reporting
: Aggregate results from all jobs
Interpreting Results
The output will include:
Write bandwidth (MB/s)
IOPS (Input/Output Operations Per Second)
Latency statistics
CPU utilization
I/O depth distribution
Key metrics to compare:
Higher bandwidth indicates better throughput
Lower latency suggests better response times
Higher IOPS shows better small I/O performance
Cleanup
Remove the job and PVC after testing:
kubectl delete job fio-benchmark
kubectl delete pvc fio-pvc
Notes
For AKS, consider using premium storage classes for better performance
Azure Local uses local storage, which typically provides lower latency
Results may vary based on node size, storage class, and cluster configuration