Database Performance on Kubernetes: Tradeoffs and Lessons Learned

Many infrastructure teams default to “put everything in containers.” Kubernetes makes it simple to deploy and manage services at scale, so databases often get pulled along for the ride. However, databases aren’t just another microservice. They rely on predictable performance from CPU, memory, storage, and networking – and containers can get in the way of that.

So although containerizing a database can make operations easier, it can also come with real performance costs. The question is how much, and what you can do about it. That’s what this article explores.

The Cost of Containers and Kubernetes

Containers and Kubernetes are now ubiquitous, even for stateful systems like databases. Should you use them? Probably – unless you have a good reason not to. 

But be aware that there is a performance penalty for the operational convenience of using containers. This is to be expected because of the extra layer of abstraction (the container itself), relaxation of resource isolation, and increased context switches. The good news is that it can certainly be overcome. In our testing using ScyllaDB, we found it is possible to take what was originally a 69% reduction in peak throughput down to a 3% performance penalty.  

Here’s the TL;DR on that specific experiment:

Containerizing applications is not free. In particular, processes comprising the containers have to be run in Linux cgroups and the container receives a virtualized view of the network. Still, the biggest cost of running a close-to-hardware, thread-per-core application like ScyllaDB inside a container comes from the opportunity cost of having to disable most of the performance optimizations that the database employs in VM and bare-metal environments to enable it to run in potentially shared and overcommitted platforms.

The best results with Kubernetes pods are obtained when resources are statically partitioned and we can bring back bare-metal optimizations like CPU pinning and interrupt isolation. There is only a 10% performance penalty in this case as compared to the underlying platform – a penalty that is mostly attributed to the network virtualization. Kubernetes allows users to expose the host network directly for specialized deployments. In cases in which this is possible, we saw that the performance difference compared to the underlying platform falls down to 3%.

How to Mitigate the Potential Performance Penalty

Of course, the potential penalty and mitigation strategies will vary from database to database. But the key takeaway is that there is likely a significant performance penalty – so be sure to hunt it down and research how to mitigate it. Some common mitigation strategies include:

• Ensure that your pods have direct access to the database’s underlying storage
• Expose the host OS network to the pod in order to avoid the performance penalty due to the Kubernetes network virtualization layer
• Allocate enough resources to the pod in question, and ensure these are not overcommitted with other pods or processes running within the same underlying host OS

Kubernetes adds yet another virtualization layer – and thus opens the door to yet another layer of performance issues, as well as different strategies for mitigating them. First off, if you have the choice of multiple options for deploying and managing database clusters on Kubernetes, test them out with an eye on performance. Once you settle on the best fit for your needs, dive into the configuration options that could impact performance. Here are some performance tips that cross databases:

• Consider dedicating specific and independent Kubernetes nodes for your database workloads and make use of affinities in order to configure their placement. 
• Enable hostNetworking and be sure to set up the required kernel parameters as recommended by your vendor (for example, fs.aio-max-nr for increasing the number of events available for asynchronous IO processing in the Linux kernel).
• Ensure that your database pods have a Guaranteed QoS class to avoid other pods from potentially hurting your main workload
• Be sure to make use of an Operator in order to orchestrate and control the lifecycle of your existing Kubernetes cluster. For example, ScyllaDB has its ScyllaDB Operator project.

Running Databases on Kubernetes with Spot Instances

Resource volatility is another consideration when running databases on Kubernetes, especially if you’re relying on spot instances that can be reclaimed at any time. One ScyllaDB user (Visually.io) demonstrated that with strategic use of the ScyllaDB Operator and Kubernetes scheduling, even these unpredictable environments can reliably support demanding database workloads.

In their deployment, they moved a multi-terabyte ScyllaDB cluster onto Kubernetes, backed largely by AWS spot instances. Spot pricing cut their EC2 costs by more than 60%, but the risk of instance terminations meant the cluster had to withstand frequent disruptions. Over the course of testing, they saw hundreds of node interruptions per month. Even under that level of churn, the database kept serving requests with single-digit millisecond latencies and without data loss.

Here are a few of the key practices that enabled this scenario:

• Blending spot and on-demand nodes: On-demand nodes acted as anchors while spot nodes provided elasticity and cost savings.
• Using anti-affinity and zone-aware placement: Replicas were spread across zones so losing a batch of nodes in one zone wouldn’t compromise consistency.
• Graceful handling of terminations: The ScyllaDB Operator reacted to AWS termination notices, draining data before the node vanished and bringing new capacity online quickly.
• Right-sizing pod resources: Database pods were given Guaranteed QoS with dedicated CPU and memory reservations, avoiding noisy-neighbor issues from colocated workloads.

Final Thoughts

Containerization and Kubernetes do add overhead. Our tests showed how tuning can shrink a 69% performance hit down to just 3%. And Visually.io’s experience with spot instances shows that the same abstractions which create penalties also support usage scenarios that would be quite difficult on bare metal.

Bare metal will always win on raw performance, but Kubernetes brings flexibility you can’t get otherwise. With tuning, the performance gap can shrink to a few percentage points. And with the right orchestration, even volatile infrastructure like spot instances becomes viable. Is Kubernetes worth it? It depends. Every deployment option comes with tradeoffs. Before deciding, carefully measure the performance impact in your specific environment, see how much you can mitigate it, and consider how you can benefit from the flexibility that Kubernetes offers. 

KubeCon + CloudNativeCon North America 2025 is taking place in Atlanta, Georgia, from November 10 to 13. Register now.