If you have ever created a bare metal Kubernetes cluster you probably came to the point where you had to think about persistent volumes. One of the nice features of Kubernetes is to have dynamic volume provisioning. The provisioner list is pretty big containing names like AWS, GCE, Cinder and Ceph. But in a small cluster you might not have any of these. I will describe how I deployed GlusterFS in my Kubernetes cluster with computers that only have one disk.

Prerequisites

  • A running Kubernetes cluster with at least three nodes (this tutorial is aimed at bare metal clusters).
  • Some understanding of PersistentVolumes, including claims and storage classes (see the Kubernetes documentation).
  • Empty disk space on each node, ~25GB are enough.

At the end you will be able to create PersistentVolumeClaims that are automatically provisioned. This is awesome for learning Kubernetes in your small toy cluster.

Disclaimer: I only used the described setup for a toy bare metal cluster. I am pretty sure that this is not a good solution for production use. Ideally your storage is not on the same machines as your cluster.

Creating Fake Block Devices

The first hurdle to overcome is that GlusterFS only works on a block devices. In my cluster I only have one disk in each node so I had to find a way to simulate a block device. Linux has the concept of loop devices. By following this blog post I was able to achieve what I wanted.

First on each node I created a file that will be used as the disk.

dd if=/dev/zero of=/home/core/glusterimage bs=1M count=25600

This will create a 25GB file. For the start I would recommend using a smaller file. The creation can take some time and if something goes wrong it can very well be you have to recreate it.

After the file as been created I added it as a loop device with losetup.

sudo losetup /dev/loop0 /home/core/glusterimage

If your system already has other loop devices you might need to choose a different number than 0.

GlusterFS will use thin provisioned logical volumes later. I am using CoreOS which does not load the kernel driver by default for this, so I also had to run:

sudo modprobe dm_thin_pool

And that’s it! Each node should now have a file /home/core/glusterimage and a /dev/loop0 device. We don’t need to mount the devices yet.

If anything goes wrong and you have to start over you can detach the loop devices with the command:

sudo losetup -d /dev/loop0

Creating the Loop Devices at Boot Time

You must make sure that losetup and modprobe are run on each boot of the machine. This depends on your Linux distribution. A common way to do this is to add the commands to /etc/rc.local. With systemd I added the file /etc/systemd/system/loopback_gluster.service to all nodes:

[Unit]
Description=Create the loopback device for GlusterFS
DefaultDependencies=false
Before=local-fs.target
After=systemd-udev-settle.service
Requires=systemd-udev-settle.service

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c "modprobe dm_thin_pool && [ -b /dev/loop0 ] || losetup /dev/loop0 /home/core/glusterimage"

[Install]
WantedBy=local-fs.target

And enabled it with systemctl enable /etc/systemd/system/loopback_gluster.service.

Deploying GlusterFS to Kubernetes

There is already a tool by GlusterFS to do exactly what we want. I had some bad experiences with the shell script, so I followed the heketi documentation in the end. It is basically everything gk-deploy.sh is doing but manually.

GlusterFS will consist of the Gluster application on each node and heketi, the tool to create volumes via a REST interface.

The needed Kubernetes deployment descriptors can be found here. I will refer to them in the following commands. All commands are run on your machine that needs to have access to the Kubernetes cluster via kubectl.

Gluster DaemonSet Creation

pwd # "/home/moritz/workspace/gluster-kubernetes", the git repository https://github.com/gluster/gluster-kubernetes/
kubectl apply -f deploy/kube-templates/glusterfs-daemonset.yaml

Now label all the nodes that should run GlusterFS. This must be at least three nodes and they all need the file and loop device you created before.

kubectl label node node0.techdev.berlin storagenode=glusterfs
kubectl label node node1.techdev.berlin storagenode=glusterfs
kubectl label node node2.techdev.berlin storagenode=glusterfs

On the labeled nodes a Gluster pod should now start. Wait until all of them are ready by checking with kubectl get pods.

Heketi Deployment

The next step is to bootstrap a heketi deployment. I edited the file deploy-heketi-template.yaml so that the Kubernetes Service is of type NodePort. We will need to access heketi from our computer later!

kind: Service
apiVersion: v1
metadata:
  name: heketi
  labels:
    glusterfs: heketi-service
  annotations:
    description: Exposes Heketi Service
spec:
  selector:
    glusterfs: heketi-pod
  type: NodePort
  ports:
  - name: heketi
    port: 8080
    targetPort: 8080
    nodePort: 30625

Then deploy it and the SerivceAccount.

kubectl create -f deploy/kube-templates/heketi-service-account.yaml
kubectl create -f deploy/kube-templates/deploy-heketi-template.yaml

Wait for the heketi pod to be ready! When it is started you should be able to interact with it from your PC, e.g. using curl node0.techdev.berlin:30625/hello.

Setting up the Cluster

Now that heketi is started we can create the Gluster topology. The gluster-kubernetes repository contains a file named topology.json.sample which we will edit to represent our cluster.

{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "node0.techdev.berlin"
              ],
              "storage": [
                "192.168.100.3"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/loop0"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node1.techdev.berlin"
              ],
              "storage": [
                "192.168.100.4"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/loop0"
          ]
        }
    ]
}

The important lines here (for the first node) are 9, 12 and 18. The manage property must contain the Kubernetes node name, the same one you used to label the nodes before. The property storage contains the IP address of the node and finally devices contains our created loop device.

When the file is ready we can tell heketi to create the topology. We need to use the heketi CLI for this. Download it and make sure it is on your path.

heketi-cli -s http://node0.techdev.berlin:30625 topology load --json=topology.json

After this command has executed successfully we use the CLI to create the final needed Kubernetes artifacts.

heketi-cli -s http://node0.techdev.berlin:30625 setup-openshift-heketi-storage

This will create a file named heketi-storage.json in your current directory. It contains some Endpoints, a Job and more for Kubernetes.

kubectl create -f heketi-storage.json

Replacing the Heketi Bootstrap Deployment

Now we are ready to replace the heketi deployment with a „real“ one. First we can remove the old.

kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"

Next I also edited heketi-deployment.yaml to have the same NodePort as deploy-heketi-template.yaml. Then create it with:

kubectl create -f deploy/kube-templates/heketi-deployment.yaml

To verify that it is working you can use the heketi CLI: heketi-cli -s http://node0.techdev.berlin:30625 volume list should now list one volume!

Creating a Storage Class

Now we can create a Kubernetes StorageClass that points to our GlusterFS cluster!

Create the following file gluster_storage_class.yaml.

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster-heketi
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://node0.techdev.berlin:30625"
  # Or, as an alternative use the Service IP of heketi. Use kubectl get svc to find it.
  # resturl: "http://10.3.0.xyz:8080"

Create the storage class in Kubernetes.

kubectl create -f deploy/storage_class.yaml

Done!

Creating Your First PVC

It is now time to finally test if the dynamic provisioning works. For this we will create a PersistentVolumeClaim. The storage class should automatically create a PersistentVolume which then also should show up in heketi.

# test_pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gluster1
  annotations:
    volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
       storage: 1Gi

Create it with kubectl create -f test_pvc.yaml.

Then check if it is getting bound with kubectl get pvc. If the status is „Pending“ for a long time check the PVC with kubectl describe pvc gluster1 for possible errors. If the volume gets bound you also will be able to see it with heketi: heketi -s http://node0.techdev.berlin:30625 volume list.

Conclusion

Using GlusterFS in a local, bare metal setup of Kubernetes is not trivial but possible. It is a great way to use more advanced features of Kubernetes and learn about them before deploying „the real thing“. Especially for constructs like StatefulSets it is very good to have dynamic provisioning available.

I hope you enjoyed reading and got GlusterFS working in your cluster. If you have any questions feel free to contact us.

Stay in the Loop

If you would like to receive an email every now and then with new articles, just sign up below. We will never spam you!