My Journey with SPDK

16.04.2024 — spdk, ubuntu, kubernetes, k3s, docker — 6 min read

Before we jump into SPDK (Storage Performance Development Kit), let's talk about my background. I'm a software engineer, and a big part of my job has been building cloud systems using Kubernetes. Lately, I've been focusing on setting up ML (Machine Learning) infrastructure. I've managed to handle more than 25 ML Kubernetes clusters all by myself, which was pretty challenging. But in all that hustle, I never got around to exploring some important Kubernetes stuff like CSI (Container Storage Interface) and CNI (Container Network Interface). Instead, I mainly relied on AWS specific CNI and CSI because that's what most aws users do. So, most of my skills are centered around managing Kubernetes controllers.

Recently, I got to chat with some engineers from Berlin. They're all about making storage cheaper and faster. Talking to them introduced me to terms like SPDK and DPDK, which I didn't know about before. It got me really interested, so now I'm diving into SPDK to see what it's all about and first day it turn out a horrer

via GIPHY

Understanding SPDK

So, what exactly is SPDK? SPDK, an acronym for Storage Performance Development Kit, is an open-source project spearheaded by Intel. It aims to revolutionize storage development by providing powerful tools and frameworks.

In this blog post, we will embark on a journey to demystify SPDK and explore its applications, particularly focusing on SPDK CSI (Container Storage Interface).

Getting Started with SPDK

Joining the Community

Before delving into the technical intricacies of SPDK, it's beneficial to join the vibrant SPDK community on Slack. These individuals are invaluable resources, offering insights and assistance to newcomers. Their collective knowledge can expedite the setup process and help overcome common hurdles associated with open-source projects.

via GIPHY

Building SPDK Application

To kickstart our journey with SPDK, let's begin by building a simple SPDK application. We'll leverage Docker to encapsulate our development environment, ensuring consistency and portability across different systems.

Below is a Dockerfile tailored for SPDK development:

# Dockerfile for SPDK development environment

FROM ubuntu:20.04
MAINTAINER Yuvraj Yadav <hi@evalsocket.dev>

# Set environment variables
ENV TZ=Asia/Kolkata \
    DEBIAN_FRONTEND=noninteractive \
    LANG=en_US.utf8

# Install dependencies
RUN apt-get update && apt-get install -y \
    locales git curl sudo tzdata pkg-config linux-modules-extra-$(uname -r) && \
    rm -rf /var/lib/apt/lists/* && \
    localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 && \
    echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

# Clone SPDK repository
WORKDIR /src
RUN git clone https://github.com/spdk/spdk --recursive

# Install SPDK dependencies and build
WORKDIR /src/spdk
RUN apt-get update && \
    scripts/pkgdep.sh && \
    ./configure && \
    make

# Copy entrypoint script
COPY run.sh .

# Set entrypoint
ENTRYPOINT ["/src/spdk/run.sh"]

The Dockerfile automates the setup process by installing necessary dependencies and fetching the SPDK repository. It then compiles SPDK using the provided script run.sh.

Setting Up Environment

Edit /etc/default/grub and add the following lines:

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=3 hugepagesz=2M hugepages=1024"

Update grub sudo update-grub
echo 3 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
Mount hugetlbfs, Prepare hugetlbfs individually for the host and each container. In order to mount hugetlbfs, edit /etc/fstab like below.

none /dev/hugepages hugetlbfs pagesize=1G,size=4G 0 0
# none /dev/hugepages2M hugetlbfs pagesize=2M,size=<SIZE> 0 0

Reboot
Check Hugepages

grep Huge /proc/meminfo

Dev Tools:

sudo apt-get install hugeadm

Troubleshoot:

Check Huge Pages

➜  ~ hugeadm --pool-list
➜  ~ hugeadm --pool-list
      Size  Minimum  Current  Maximum  Default
   2097152     1024     1024     1024        *
1073741824        3        3        3

➜  ~ hugeadm    --list-all-mounts

Mount Point          Options
/dev/hugepages       rw,relatime,pagesize=1024M,size=4294967296

Ref:

Testing SPDK Docker Image

Once the environment is set up, we can test our SPDK Docker image as follows:

docker build -t evalsocket/spdk:latest .
docker run -d --name spdk --privileged -v /dev:/dev evalsocket/spdk:latest
# Initiate an NVMf (NVMe over Fabrics) transport
$ docker exec -it spdk scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -m 8 -c 8192

If everything works as expected, we can proceed to integrate SPDK with Kubernetes.

Integrating SPDK with Kubernetes

#!/bin/bash

# Set ulimit
ulimit -l

# Disable firewall
ufw disable

# Allow necessary ports
ufw allow 6443/tcp #apiserver
ufw allow from 10.42.0.0/16 to any #pods
ufw allow from 10.43.0.0/16 to any #services

# Reload firewall rules
sudo ufw reload

# Install k3s
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s -

# Set KUBECONFIG
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

# Restart k3s
sudo systemctl restart k3s.service

# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod a+x ./kubectl
mv ./kubectl /usr/local/bin/kubectl

After setting up the k3s cluster, we can verify its status and node resources using kubectl commands.

$ kubectl get pods -A 
$ kubectl get nodes -oyaml | less 
    allocatable:
      cpu: "4"
      ephemeral-storage: "39358207765"
      hugepages-1Gi: 3Gi
      hugepages-2Mi: 2Gi
      memory: 11126996Ki
      pods: "110"
    capacity:
      cpu: "4"
      ephemeral-storage: 40458684Ki
      hugepages-1Gi: 3Gi
      hugepages-2Mi: 2Gi
      memory: 16369876Ki
      pods: "110"

Deploying SPDK Application on Kubernetes

Once you have verified the correct configuration of huge pages resources, you are ready to deploy the SPDK application. Without the correct huge pages resource allocation, the Kubernetes scheduler may fail to schedule the pod. If everything is set up correctly, proceed with creating the following pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: spdk
spec:
  containers:
  - name: spdk-container
    image: evalsocket/spdk:v5
    volumeMounts:
    - name: dev-hostpath
      mountPath: /host/dev
      readOnly: true
    ports:
    - containerPort: 9009
    - containerPort: 4444
    - containerPort: 5555
    securityContext:
      privileged: true
    resources:
      limits:
        hugepages-1Gi: 1Gi
      requests:
        memory: 100Mi
  volumes:
  - name: dev-hostpath
    hostPath:
      path: /dev

Use the following commands to create the pod and view its logs:

# Create Pod
$ kubectl create -f app.yaml 

# View Pod Logs
$ kubectl logs spdk

Once the pod is successfully deployed and running, you can initiate an NVMf (NVMe over Fabrics) transport using the following command:

$ kubectl exec spdk -- scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -m 8 -c 8192

Exploring SPDK CSI (Container Storage Interface)

Now, let's delve into SPDK CSI (Container Storage Interface). You might be wondering about the difference between the previous SPDK application and SPDK CSI. Well, SPDK CSI serves as the interface for Kubernetes to interact with an SPDK server. While our previous focus was on deploying a standalone SPDK application, SPDK CSI offers a way to seamlessly integrate our software with Kubernetes, thereby enabling the provision of storage as a service.

By leveraging SPDK CSI, you can effectively expose SPDK functionality within Kubernetes, allowing for dynamic provisioning, snapshotting, and resizing of storage volumes. This integration empowers you to harness the performance benefits of SPDK while leveraging the orchestration capabilities of Kubernetes.

In the subsequent sections, we'll explore the setup and usage of SPDK CSI, unlocking new possibilities for storage management within Kubernetes environments.