In the rapidly evolving world of cloud-native infrastructure, the choice of a Container Network Interface (CNI) is the difference between a high-performance cluster and a latent one. During the AIORI-2 Hackathon, team L8NCY from Christ University, Bengaluru, took on the challenge of benchmarking Kubernetes networking using the rigorous standards of the IETF.
By implementing the methodology from draft-ietf-bmwg-containerized-infra, we conducted a head-to-head comparison of three industry leaders: Flannel, Calico, and Cilium.
1. The Benchmarking Framework
Our study wasn’t just a simple test; it was a validation of the Benchmarking Methodology Working Group (BMWG) standards. We measured four critical pillars of network health:
- Throughput: The maximum data transfer rate.
- Latency: The Round-Trip Time (RTT) for packet delivery.
- Jitter: The variation in packet delay.
- Scalability: Performance stability as pod density increases from 20 to 150+ pods.
2. Implementation: The eBPF vs. VXLAN Battle
We deployed identical clusters using Kind and Minikube on a standardized Windows 11/Docker Desktop environment. Each CNI brought a different architectural approach to the table:
- Flannel (VXLAN): Simple “plug-and-play” networking using traditional encapsulation.
- Calico (BGP/IPIP): A robust routing-based approach known for its scalability and policy engine.
- Cilium (eBPF): The cutting edge of networking, using eBPF to process packets directly in the Linux kernel, bypassing much of the traditional networking stack.
3. Key Results & Findings
Our data confirmed that architecture dictates performance. Cilium emerged as the clear winner for high-performance needs, while Calico proved to be the most stable “all-rounder.”
| Metric | Flannel (v0.25) | Calico (v3.28) | Cilium (v1.16) |
|---|---|---|---|
| Avg Latency | 2.8 ms | 1.6 ms | 1.1 ms |
| Throughput | 8.4 Gbps | 9.2 Gbps | 9.5 Gbps |
| Jitter | 0.5 ms | 0.3 ms | 0.2 ms |
| CPU Usage | 22% | 25% | 31% |
| Best Use Case | Small-scale/Dev | Production/Scalable | Latency-sensitive |
4. Technical Challenges & Solutions
- The eBPF Hurdle: Enabling Cilium on Docker Desktop required specific kernel BPF features. This increased CPU overhead but delivered the promised latency gains.
- MTU Mismatches: We discovered that inconsistent Maximum Transmission Unit (MTU) settings between clusters caused significant performance drops. Synchronizing these settings was key to a fair test.
- Scaling Caps: We observed that regardless of the CNI, network performance began to degrade significantly after passing the 150-pod threshold on our testbed, identifying a hardware-resource bottleneck.
5. Open Source & Standards Impact
This project isn’t just a report; it’s a contribution to the global networking community. We have open-sourced our L8NCY Automation Framework, a PowerShell-based tool that automates cluster creation, CNI deployment, and iperf3 testing.
“Implementing BMWG benchmarking in Kubernetes showed how CNI configurations directly influence network performance. It’s the difference between an application that feels snappy and one that feels sluggish.” — Pamuru Ritesh Reddy, Team Lead
Read the full report
