Tech/Kubernetes and the TrueNAS CSI

From ~esantoro
Revision as of 20:59, 19 May 2026 by Esantoro (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

So I've been playing a bit with my Lenovo ThinkStation C30.

Despite being a fairly old machine it offers me plenty of resources and with the addition of a couple of 8TB spinning-rust disks, it's starting to get almost-seriously usable for almost-serious things. One such almost-serious things is running one or more kubernetes virtual machines.

As I don't really want to manage storage in a "kubernetic" way, I wanted to have kubernetes offload storage to some third party. In my case such thirt-party is the good old TrueNAS Community Edition.

In short, this page describes setting up the truenas-csi container storage interface so that kubernetes can automagically creates volumes and bind them. You know, as if you were running in a proper cloud environment ;)

AI disclaimer: part of this investigation was done with AI (chatgpt/codex, specifically).



Prerequisites

For starters: each node in the cluster has to have tooling to handle NFS and/or iSCSI shares.

For example (in my case it's a dumb Debian system):

# Debian/Ubuntu
## iSCSI packages
sudo apt-get install -y open-iscsi
sudo systemctl enable iscsid
sudo systemctl start iscsid
## NFS packages
sudo apt-get install -y nfs-common


# RHEL/CentOS
## iSCSI packages
sudo yum install -y iscsi-initiator-utils
sudo systemctl enable iscsid
sudo systemctl start iscsid
## NFS packages
sudo yum install -y nfs-utils

You will also need a TrueNAS api key. You can generate one from profile → API Keys. I used my admin user for the api key, as it most likely needs permission to handle a bit of everything.

Setting an api key from an admin user is most likely not the best security-posture-wise but then again it's a truenas virtual machine serving a kubernetes virtual machine, both connected to the same virtual network bridge. Both machines are hardly exposed to anything so its "safe enough". If i was to do the same with physical machine in a work environment i'd do that differently, of course.

Configuring and deploying the CSI

Configuring

The tuenas-csi does not offer a proper helm chart but rather it recommends installing it from the yaml file in the git repository.

wget https://raw.githubusercontent.com/truenas/truenas-csi/refs/heads/master/deploy/truenas-csi-driver.yaml

## # emacs truenas-csi-driver.yaml # and configure stuff...

## alternatively...
sed -i s/YOUR-TRUENAS-IP/your-truenas-ip/g truenas-csi-driver.yaml
sed -i s/your-api-key/your_api_key/g truenas-csi-driver.yaml
sed -i s/defaultPool: "tank"/defaultPool: "your_pool_name"/g truenas-csi-driver.yaml

Disabling topology features

The controller has topology features. I don't need those, so the solution is to edit the truenas-csi-driver.yaml and disable the feature gate.

Look for the container named csi-provisioner in the deployment named truenas-csi-controller and add the following extra command line arg: --feature-gates=Topology=false"

Otherwise you might have to add fancy (fancy as in "non-standard) labels to all your node according to the topology (if you're wondering, the labels are topology.truenas.io/node and topology.truenas.io/pool)

Actually deploying the CSI

Finally:

kubectl apply -f truenas-csi-driver.yaml

Creating StorageClasses

So you can now primarly export storage via two methods: NFS and iSCSI.

NFS means you're essentially exporting folders over the network whereas iSCSI means you're exporting whole block devices (they might be actual block devices, they might be files on disk, they might be something else: it's irrelevant - in the case of TrueNAS they're most likely ZFS ZVOLs).

But then again, another important aspect is retention: what happens to the volume when the pod using it has been deleted? In kubernetes you can configure retention in two ways: Retain and Delete. Retain means the underlying volume will be kept around whereas Delete means the underlying volume will be deleted (duh). Retain is usually safer but it also means that after a while the infrastructure will be littered with random volumes. In a business context with real workloads they can pile up pretty quickly and I've seen it happen (and I've seen the AWS bills for that).

StorageClasses are, in short, the bridge between workloads and your CSI. They're essentially some pre-set values on how to provision volumes. When creating PersistentVolumeClaims, you will be referencing a StorageClass to express the such pre-set values.

So my recommendation, at least for truenas-csi, is to have two storageClasses for each exporting method: one with reclaimPolicy: Retain and one with reclaimPolicy: Delete. And then pick whatever you need.

For example, this is my nfs storageClass with Delete as reclaimPolicy:

# Basic NFS StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: truenas-nfs-noretain
provisioner: csi.truenas.io
parameters:
  # Storage protocol
  protocol: "nfs"
  # ZFS pool used for provisioning
  pool: "storage"
  # Dataset path within the pool for Kubernetes volumes
  datasetPath: "z8s"
  # ZFS compression (optional): OFF, LZ4, GZIP, ZSTD, ZLE, LZJB
  compression: "LZ4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

A similar StorageClass with reclaimPolicy: Retain exists:

# Basic NFS StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: truenas-nfs-retain
provisioner: csi.truenas.io
parameters:
  # Storage protocol
  protocol: "nfs"
  # ZFS pool used for provisioning
  pool: "storage"
  # Dataset path within the pool for Kubernetes volumes
  datasetPath: "z8s"
  # ZFS compression (optional): OFF, LZ4, GZIP, ZSTD, ZLE, LZJB
  compression: "LZ4"
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true

I haven't played with iSCSI StorageClasses yet.

Links and references

Off-topic

The kubernetes "cluster" is not really a cluster as it's a single machine, running k0s, for playing reasons (i don't really have reliability requirements).

Funnily enough, i've stumbled upon https://github.com/sergelogvinov/proxmox-cloud-controller-manager as a cloud controller manager that establishes a feedback loop between kubernetes and proxmox (the underlying infrastructure). I might play with it in the future.

Retrieved from "https://santoro.tk/w/index.php?title=Tech/Kubernetes_and_the_TrueNAS_CSI&oldid=314"