Blog/A Kubernetes Sunday Afternoon

So at my previous job i used to managed a number of Kubernetes cluster, and I got familiar with that technology. I wouldn't define my competence deep or thorough, but nonetheless I occasionally did something "advanced" and (as usual) this happened the most in a troubleshooting context.

I have a love/hate relationship with Kubernetes, because when it works it's an incredible piece of technology but when it breaks it's incredibly annoying to fix, and the whole ecosystem around it has some quirks that people don't seem to realize.

So far I haven't been working with Kubernetes for a bit more than a year, and in the heat of starting a new job and getting up to speed (oh boy that was intense) I haven't touched Kubernetes pretty much at all, and I've scaled down many of my experimenting at home.

The most important stuff I run at home can and will happily run off a single computer and don't change often enough to require a container orchestrating platform behind, so that's essentially what i do (although i have some safeguards and procedures to protect against disasters).

I'm on vacation now, and i thought it was a nice time to play a bit with kubernetes and see how hard would it be to spin off a simple cluster today, and if that's something i could run at home.

The core idea

I occasionally write some code that would like to deploy in a simple matter ("here's a docker image, please run it for me and don't bother me further") so I decided to spend some time with k3s.

My desiderata were:

• really minimal footprint (i don't plan on running beefy machines)
• simple setup and management
  • i really don't want to spend time reading the documentation for the various options of the kubelet
• highly available
  • I want to be able to drain/stop/restart/reboot/reinstall a node without having services down
• can run off three nodes
  • bonus point: it can run off two nodes
• it must handle storage as well
  • not all things are stateless
  • i don't want to run additional machines to shared storage

So today I spent this lazy afternoon setting up a kubernetes cluster :)

My choices

So far my choices have been:

• k3s as a kubernetes distribution
• OpenEBS or Longhorn for storage
• tested everything in the AWS cloud, using c6g (graviton) instances
  • i didn't bring (all of) my computers with me... i'm on vacation :)
  • this was also to make sure everything could run on arm-based systems (will raspberry pi 4s ever be in stock again?)

I first tried OpenEBS and was largely disappointed. The documentation is severely lacking (the "quick start" guide will leave you in a state where you can just provision local storage, not much useful. The other providers needed additional setup work that, quite frankly, were not worth it (in my case).

Longhorn was a pleasant experience and worked almost immediately. I had some problems because OpenEBS had left some webhooks behind (even though I uninstalled everything via helm, which was supposed not to leave stuff behind...).

Further ideas

• maybe try and do some benchmarks
• deploy some sample stuff
• develop some basic automation
  • do I really need it?
• If i get that running at home, i might look into keepalived to have single ips pointing at the master nodes etc

Notable mentions

• k0s is also very nice, I tried that in the past