In Abstract Technology GmbH we recently deployed Open edX on Kubernetes (often shortened to K8s) using NixOS to set up a K3s cluster on Hetzner Cloud VMs. 

But before we dive into K8s with NixOS, let's have a quick introduction to Kubernetes.

Why Run Open edX on Kubernetes?

​

As online learning platforms like the Open edX platform grow in popularity, many organizations are seeking robust, scalable infrastructure solutions to support an increasing volume of users and complex workflows. Kubernetes is a popular choice in managing containerized applications, making it a compelling option for deploying an Open edX platform. This setup offers several key advantages in areas like scalability, automation, and resource optimization, but it also introduces unique considerations that may not suit every organization. Deploying Kubernetes has been, for many years, the best option for our company to offer scalable solutions to our clients.

Pros of Running Open edX on Kubernetes

• Scalability: Kubernetes makes it easy to manage and scale services up or down based on demand. With the Open edX LMS, where traffic may spike around course launches or assessments, Kubernetes ensures that you can respond to these fluctuations smoothly.
• Resilience and High Availability: Kubernetes is designed for high availability, automatically managing resources to ensure services remain up, even when individual nodes or containers fail. This resilience is critical for an LMS platform that requires minimal downtime for users.
• Flexibility Across Platforms: Kubernetes provides flexibility in where and how the Open edX platform is hosted, from on-premises hardware to cloud environments like Hetzner, AWS, or GCP, enabling cost-effective hosting decisions.

Potential Challenges

• Complexity of Setup: Kubernetes is known for its steep learning curve, particularly for teams not yet familiar with container orchestration. Setting up and managing Kubernetes clusters adds operational complexity that requires time, expertise, and proper tooling.
• Infrastructure Overhead: Running Open edX on Kubernetes may introduce additional infrastructure layers, which may be unnecessary for smaller deployments or organizations without high traffic demands.
• Cost of Cluster Maintenance: Although Kubernetes can optimize resources, there are costs associated with managing and maintaining the clusters, particularly for smaller teams that may lack dedicated DevOps resources.

Why Hetzner and K3s?

Hetzner Cloud is affordable and reliable, perfect for this kind of setup. We went with Hetzner's CX11, CX21 and CX31 VMs—small but solid for the price.

To manage everything, We used K3s—a lightweight version of Kubernetes that’s simpler and less resource-hungry. NixOS, which shines for its declarative setup, paired really well with K3s, making our life easier.

Deploying K3s with NixOS

We picked NixOS for its immutability and easy configuration management. Colmena helped us manage all our instances remotely. First, we converted the Hetzner VMs to NixOS using nixos-infect, giving us a consistent base for deployment.

The cluster had three nodes—one server and two agents. We also ran an Nginx server on a separate VM to handle traffic, with a taint to keep it from hosting general pods.

Open edX Challenges

Open edX is pretty heavy, so tuning was key:

• Persistent Storage: Instead of hosting MySQL, MongoDB, Redis, and OpenSearch inside Kubernetes, we used dedicated VMs. This simplified maintenance and boosted stability.
• Networking: All VMs were on a private network, making inter-service communication secure and easy.

Lessons Learned

• NixOS + Kubernetes: Powerful but challenging. Debugging required patience and lots of logs.
• Hetzner Cloud: Budget-friendly with good performance. Network management took some extra work, but it was worth it.
• K3s: Lightweight and efficient, though a few differences from standard Kubernetes needed adjustments.
• Dedicated Databases: Keeping MySQL, MongoDB, etc., on their own VMs made a huge difference in stability and performance.
• Nginx for Ingress: Running Nginx on its own VM for ingress worked great, simplifying traffic handling and SSL termination.

Final Thoughts

Testing and deploying the Open edX platform on Hetzner with Kubernetes and NixOS was definitely tricky, but combining on Hetzner Cloud worked out well.

The setup was reliable, scalable, and cheaper than other options. If you're comfortable with NixOS and want a Kubernetes challenge, it's definitely worth trying.

Price Comparison

The following figures present a comparative pricing analysis of AWS, Azure, and Hetzner. Please note that the VM resources are not directly comparable, but the comparison is as close as possible:

Image Source: Abstract Technology GmbH. All rights reserved. AWS (1 year reserved) K8S infrastructure Price