Testing of CSI drivers

Wednesday, January 08, 2020

Testing of CSI drivers

Author: Patrick Ohly (Intel)

When developing a Container Storage Interface (CSI) driver, it is useful to leverage as much prior work as possible. This includes source code (like the sample CSI hostpath driver) but also existing tests. Besides saving time, using tests written by someone else has the advantage that it can point out aspects of the specification that might have been overlooked otherwise.

An earlier blog post about end-to-end testing already showed how to use the Kubernetes storage tests for testing of a third-party CSI driver. That approach makes sense when the goal to also add custom E2E tests, but depends on quite a bit of effort for setting up and maintaining a test suite.

When the goal is to merely run the existing tests, then there are simpler approaches. This blog post introduces those.

Sanity testing

csi-test sanity ensures that a CSI driver conforms to the CSI specification by calling the gRPC methods in various ways and checking that the outcome is as required. Despite its current hosting under the Kubernetes-CSI organization, it is completely independent of Kubernetes. Tests connect to a running CSI driver through its Unix domain socket, so although the tests are written in Go, the driver itself can be implemented in any language.

The main README explains how to include those tests into an existing Go test suite. The simpler alternative is to just invoke the csi-sanity command.

Installation

Starting with csi-test v3.0.0, you can build the csi-sanity command with go get github.com/kubernetes-csi/csi-test/cmd/csi-sanity and you’ll find the compiled binary in $GOPATH/bin/csi-sanity.

go get always builds the latest revision from the master branch. To build a certain release, get the source code and run make -C cmd/csi-sanity. This produces cmd/csi-sanity/csi-sanity.

Usage

The csi-sanity binary is a full Ginkgo test suite and thus has the usual -gingko command line flags. In particular, -ginkgo.focus and -ginkgo.skip can be used to select which tests are run resp. not run.

During a test run, csi-sanity simulates the behavior of a container orchestrator (CO) by creating staging and target directories as required by the CSI spec and calling a CSI driver via gRPC. The driver must be started before invoking csi-sanity. Although the tests currently only check the gRPC return codes, that might change and so the driver really should make the changes requested by a call, like mounting a filesystem. That may mean that it has to run as root.

At least one gRPC endpoint must be specified via the -csi.endpoint parameter when invoking csi-sanity, either as absolute path (unix:/tmp/csi.sock) for a Unix domain socket or as host name plus port (dns:///my-machine:9000) for TCP. csi-sanity then uses that endpoint for both node and controller operations. A separate endpoint for controller operations can be specified with -csi.controllerendpoint. Directories are created in /tmp by default. This can be changed via -csi.mountdir and -csi.stagingdir.

Some drivers cannot be deployed such that everything is guaranteed to run on the same host. In such a case, custom scripts have to be used to handle directories: they log into the host where the CSI node controller runs and create or remove the directories there.

For example, during CI testing the CSI hostpath example driver gets deployed on a real Kubernetes cluster before invoking csi-sanity and then csi-sanity connects to it through port forwarding provided by socat. Scripts are used to create and remove the directories.

Here’s how one can replicate that, using the v1.2.0 release of the CSI hostpath driver:

$ cd csi-driver-host-path
$ git describe --tags HEAD
v1.2.0
$ kubectl get nodes
NAME        STATUS   ROLES    AGE   VERSION
127.0.0.1   Ready    <none>   42m   v1.16.0

$ deploy/kubernetes-1.16/deploy-hostpath.sh 
applying RBAC rules
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.4.0/deploy/kubernetes/rbac.yaml
...
deploying hostpath components
   deploy/kubernetes-1.16/hostpath/csi-hostpath-attacher.yaml
        using           image: quay.io/k8scsi/csi-attacher:v2.0.0
service/csi-hostpath-attacher created
statefulset.apps/csi-hostpath-attacher created
   deploy/kubernetes-1.16/hostpath/csi-hostpath-driverinfo.yaml
csidriver.storage.k8s.io/hostpath.csi.k8s.io created
   deploy/kubernetes-1.16/hostpath/csi-hostpath-plugin.yaml
        using           image: quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
        using           image: quay.io/k8scsi/hostpathplugin:v1.2.0
        using           image: quay.io/k8scsi/livenessprobe:v1.1.0
...
service/hostpath-service created
statefulset.apps/csi-hostpath-socat created
07:38:46 waiting for hostpath deployment to complete, attempt #0
deploying snapshotclass
volumesnapshotclass.snapshot.storage.k8s.io/csi-hostpath-snapclass created

$ cat >mkdir_in_pod.sh <<EOF
#!/bin/sh
kubectl exec csi-hostpathplugin-0 -c hostpath -- mktemp -d /tmp/csi-sanity.XXXXXX
EOF

$ cat >rmdir_in_pod.sh <<EOF
#!/bin/sh
kubectl exec csi-hostpathplugin-0 -c hostpath -- rmdir "\$@"
EOF

$ chmod u+x *_in_pod.sh
$ csi-sanity -ginkgo.v \
             -csi.endpoint dns:///127.0.0.1:$(kubectl get "services/hostpath-service" -o "jsonpath={..nodePort}") \
             -csi.createstagingpathcmd ./mkdir_in_pod.sh \
             -csi.createmountpathcmd ./mkdir_in_pod.sh \
             -csi.removestagingpathcmd ./rmdir_in_pod.sh \
             -csi.removemountpathcmd ./rmdir_in_pod.sh

Running Suite: CSI Driver Test Suite
====================================
Random Seed: 1570540138
Will run 72 of 72 specs
...
Controller Service [Controller Server] ControllerGetCapabilities 
  should return appropriate capabilities
  /nvme/gopath/src/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:111
STEP: connecting to CSI driver
STEP: creating mount and staging directories
STEP: checking successful response
•
------------------------------
Controller Service [Controller Server] GetCapacity 
  should return capacity (no optional values added)
  /nvme/gopath/src/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:149
STEP: reusing connection to CSI driver at dns:///127.0.0.1:30056
STEP: creating mount and staging directories
...
Ran 53 of 72 Specs in 148.206 seconds
SUCCESS! -- 53 Passed | 0 Failed | 0 Pending | 19 Skipped
PASS

Some comments:

• The source code of these tests is in the pkg/sanity package.
• How to determine the external IP address of the node depends on the cluster. In this example, the cluster was brought up with hack/local-up-cluster.sh and thus runs on the local host (127.0.0.1). It uses a port allocated by Kubernetes, obtained above with kubectl get "services/hostpath-service". The Kubernetes-CSI CI uses kind and there a Docker command can be used.
• The create script must print the final directory. Using a unique directory for each test case has the advantage that if something goes wrong in one test case, others still start with a clean slate.
• The “staging directory”, aka NodePublishVolumeRequest.target_path in the CSI spec, must be created and deleted by the CSI driver while the CO is responsible for the parent directory. csi-sanity handles that by creating a directory and then giving the CSI driver that directory path with /target appended at the end. Kubernetes got this wrong and creates the actual target_path directory, so CSI drivers which want to work with Kubernetes currently have to be lenient and must not fail when that directory already exists.
• The “mount directory” corresponds to NodeStageVolumeRequest.staging_target_path and really gets created by the CO, i.e. csi-sanity.

End-to-end testing

In contrast to csi-sanity, end-to-end testing interacts with the CSI driver through the Kubernetes API, i.e. it simulates operations from a normal user, like creating a PersistentVolumeClaim. Support for testing external CSI drivers was added in Kubernetes 1.14.0.

Installation

For each Kubernetes release, a test tar archive is published. It’s not listed in the release notes (for example, the ones for 1.16), so one has to know that the full URL is https://dl.k8s.io/<version>/kubernetes-test-linux-amd64.tar.gz (like for v1.16.0).

These include a e2e.test binary for Linux on x86-64. Archives for other platforms are also available, see this KEP. The e2e.test binary is completely self-contained, so one can “install” it and the ginkgo test runner with:

curl --location https://dl.k8s.io/v1.16.0/kubernetes-test-linux-amd64.tar.gz | \
  tar --strip-components=3 -zxf - kubernetes/test/bin/e2e.test kubernetes/test/bin/ginkgo

Each e2e.test binary contains tests that match the features available in the corresponding release. In particular, the [Feature: xyz] tags change between releases: they separate tests of alpha features from tests of non-alpha features. Also, the tests from an older release might rely on APIs that were removed in more recent Kubernetes releases. To avoid problems, it’s best to simply use the e2e.test binary that matches the Kubernetes release that is used for testing.

Usage

Not all features of a CSI driver can be discovered through the Kubernetes API. Therefore a configuration file in YAML or JSON format is needed which describes the driver that is to be tested. That file is used to populate the driverDefinition struct and the DriverInfo struct that is embedded inside it. For detailed usage instructions of individual fields refer to these structs.

A word of warning: tests are often only run when setting some fields and the file parser does not warn about unknown fields, so always check that the file really matches those structs.

Here is an example that tests the csi-driver-host-path:

$ cat >test-driver.yaml <<EOF
StorageClass:
  FromName: true
SnapshotClass:
  FromName: true
DriverInfo:
  Name: hostpath.csi.k8s.io
  Capabilities:
    block: true
    controllerExpansion: true
    exec: true
    multipods: true
    persistence: true
    pvcDataSource: true
    snapshotDataSource: true
InlineVolumes:
- Attributes: {}
EOF

At a minimum, you need to define the storage class you want to use in the test, the name of your driver, and what capabilities you want to test. As with csi-sanity, the driver has to be running in the cluster before testing it. The actual e2e.test invocation then enables tests for this driver with -storage.testdriver and selects the storage tests for it with -ginkgo.focus:

$ ./e2e.test -ginkgo.v \
             -ginkgo.focus='External.Storage' \
             -storage.testdriver=test-driver.yaml
Oct  8 17:17:42.230: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used.
I1008 17:17:42.230210  648569 e2e.go:92] Starting e2e run "90b9adb0-a3a2-435f-80e0-640742d56104" on Ginkgo node 1
Running Suite: Kubernetes e2e suite
===================================
Random Seed: 1570547861 - Will randomize all specs
Will run 163 of 5060 specs

Oct  8 17:17:42.237: INFO: >>> kubeConfig: /var/run/kubernetes/admin.kubeconfig
Oct  8 17:17:42.241: INFO: Waiting up to 30m0s for all (but 0) nodes to be schedulable
...
------------------------------
SSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: hostpath.csi.k8s.io] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] 
  should access to two volumes with different volume mode and retain data across pod recreation on the same node
  /workspace/anago-v1.16.0-rc.2.1+2bd9643cee5b3b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:191
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
...

You can use ginkgo to run some kinds of test in parallel. Alpha feature tests or those that by design have to be run sequentially then need to be run separately:

$ ./ginkgo -p -v \
         -focus='External.Storage' \
         -skip='\[Feature:|\[Disruptive\]|\[Serial\]' \
         ./e2e.test \
         -- \
         -storage.testdriver=test-driver.yaml
$ ./ginkgo -v \
         -focus='External.Storage.*(\[Feature:|\[Disruptive\]|\[Serial\])' \
         ./e2e.test \
         -- \
         -storage.testdriver=test-driver.yaml

Getting involved

Both the Kubernetes storage tests and the sanity tests are meant to be applicable to arbitrary CSI drivers. But perhaps tests are based on additional assumptions and your driver does not pass the testing although it complies with the CSI specification. If that happens then please file issues (links below).

These are open source projects which depend on the help of those using them, so once a problem has been acknowledged, a pull request addressing it will be highly welcome.

The same applies to writing new tests. The following searches in the issue trackers select issues that have been marked specifically as something that needs someone’s help: - csi-test - Kubernetes

Happy testing! May the issues it finds be few and easy to fix.