View source for Tech/Kubernetes and the TrueNAS CSI

So I've been playing a bit with my [[Tech/The Lenovo ThinkStation C30|Lenovo ThinkStation C30]]. 

Despite being a fairly old machine it offers me plenty of resources and with the addition of a couple of 8TB spinning-rust disks, it's starting to get almost-seriously usable for almost-serious things. One such almost-serious things is running one or more kubernetes virtual machines. 

As I don't really want to manage storage in a "kubernetic" way, I wanted to have kubernetes offload storage to some third party. In my case such thirt-party is the good old [https://www.truenas.com/truenas-community-edition/ TrueNAS Community Edition].

In short, this page describes setting up the [https://github.com/truenas/truenas-csi truenas-csi] container storage interface so that kubernetes can automagically creates volumes and bind them. You know, as if you were running in a proper cloud environment ;)

AI disclaimer: part of this investigation was done with AI (chatgpt/codex, specifically).


__TOC__


== Prerequisites ==
For starters: each node in the cluster has to have tooling to handle NFS and/or iSCSI shares.

For example (in my case it's a dumb Debian system):<syntaxhighlight lang="shell-session">
# Debian/Ubuntu
## iSCSI packages
sudo apt-get install -y open-iscsi
sudo systemctl enable iscsid
sudo systemctl start iscsid
## NFS packages
sudo apt-get install -y nfs-common


# RHEL/CentOS
## iSCSI packages
sudo yum install -y iscsi-initiator-utils
sudo systemctl enable iscsid
sudo systemctl start iscsid
## NFS packages
sudo yum install -y nfs-utils
</syntaxhighlight>You will also need a TrueNAS api key. You can generate one from profile → API Keys. I used my admin user for the api key, as it most likely needs permission to handle a bit of everything.

Setting an api key from an admin user is most likely not the best security-posture-wise but then again it's a truenas virtual machine serving a kubernetes virtual machine, both connected to the same virtual network bridge. Both machines are hardly exposed to anything so its "''safe enough''". If i was to do the same with physical machine in a work environment i'd do that differently, of course.

== Configuring and deploying the CSI ==

=== Configuring ===
The [https://github.com/truenas/truenas-csi tuenas-csi] does not offer a proper helm chart but rather it recommends installing it from the yaml file in the git repository.<syntaxhighlight lang="shell-session">
wget https://raw.githubusercontent.com/truenas/truenas-csi/refs/heads/master/deploy/truenas-csi-driver.yaml

## # emacs truenas-csi-driver.yaml # and configure stuff...

## alternatively...
sed -i s/YOUR-TRUENAS-IP/your-truenas-ip/g truenas-csi-driver.yaml
sed -i s/your-api-key/your_api_key/g truenas-csi-driver.yaml
sed -i s/defaultPool: "tank"/defaultPool: "your_pool_name"/g truenas-csi-driver.yaml


</syntaxhighlight>

=== Disabling topology features ===
The controller has topology features. I don't need those, so the solution is to edit the <code>truenas-csi-driver.yaml</code> and disable the feature gate.

Look for the container named <code>csi-provisioner</code> in the deployment named <code>truenas-csi-controller</code> and add the following extra command line arg: <code>--feature-gates=Topology=false"</code>

Otherwise you might have to add fancy (fancy as in "non-standard) labels to all your node according to the topology (if you're wondering, the labels are <code>topology.truenas.io/node</code> and <code>topology.truenas.io/pool</code>)

=== Actually deploying the CSI ===
Finally:<syntaxhighlight lang="shell-session">
kubectl apply -f truenas-csi-driver.yaml
</syntaxhighlight>

== Creating StorageClasses ==
So you can now primarly export storage via two methods: NFS and iSCSI.

NFS means you're essentially exporting folders over the network whereas iSCSI means you're exporting whole block devices (they might be actual block devices, they might be files on disk, they might be something else: it's irrelevant - in the case of TrueNAS they're most likely ZFS ZVOLs).

But then again, another important aspect is retention: what happens to the volume when the pod using it has been deleted? In kubernetes you can configure retention in two ways: <code>Retain</code> and <code>Delete</code>. <code>Retain</code> means the underlying volume will be kept around whereas <code>Delete</code> means the underlying volume will be deleted (duh). Retain is usually safer but it also means that after a while the infrastructure will be littered with random volumes. In a business context with real workloads they can pile up pretty quickly and I've seen it happen (and I've seen the AWS bills for that).

StorageClasses are, in short, the bridge between workloads and your CSI. They're essentially some pre-set values on how to provision volumes. When creating PersistentVolumeClaims, you will be referencing a StorageClass to express the such pre-set values.

So my recommendation, at least for truenas-csi, is to have two storageClasses for each exporting method: one with <code>reclaimPolicy: Retain</code> and one with <code>reclaimPolicy: Delete</code>. And then pick whatever you need.

For example, this is my nfs storageClass with Delete as reclaimPolicy:<syntaxhighlight lang="yaml">
# Basic NFS StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: truenas-nfs-noretain
provisioner: csi.truenas.io
parameters:
  # Storage protocol
  protocol: "nfs"
  # ZFS pool used for provisioning
  pool: "storage"
  # Dataset path within the pool for Kubernetes volumes
  datasetPath: "z8s"
  # ZFS compression (optional): OFF, LZ4, GZIP, ZSTD, ZLE, LZJB
  compression: "LZ4"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
</syntaxhighlight>A similar StorageClass with <code>reclaimPolicy: Retain</code> exists:<syntaxhighlight lang="yaml">
# Basic NFS StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: truenas-nfs-retain
provisioner: csi.truenas.io
parameters:
  # Storage protocol
  protocol: "nfs"
  # ZFS pool used for provisioning
  pool: "storage"
  # Dataset path within the pool for Kubernetes volumes
  datasetPath: "z8s"
  # ZFS compression (optional): OFF, LZ4, GZIP, ZSTD, ZLE, LZJB
  compression: "LZ4"
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
</syntaxhighlight>I haven't played with iSCSI StorageClasses yet.

== Links and references ==

* https://github.com/truenas/truenas-csi
* https://www.truenas.com/docs/solutions/integrations/csidriver/csidriver/
* https://kubernetes.io/docs/concepts/storage/storage-classes/

== Off-topic ==
The kubernetes "cluster" is not really a cluster as it's a single machine, running k0s, for playing reasons (i don't really have reliability requirements).

Funnily enough, i've stumbled upon https://github.com/sergelogvinov/proxmox-cloud-controller-manager as a cloud controller manager that establishes a feedback loop between kubernetes and proxmox (the underlying infrastructure). I might play with it in the future.
[[Category:Tech]]