Docker: Customizing/Developing Ceph on a Cluster

Written by Ivo Jimenez
This blog was adapted from one of our other blogs [link]

We want to layer the binaries for our Ceph projects (e.g., customized Ceph, Mantle, and ZLog) on top of the official Ceph daemon image. The idea is that each node in a cluster will pull the same Docker daemon image and start up as a MON, MDS, or OSD. This orchestration would be done with ceph-ansible. For a more detailed description of how the Docker daemon image should be used, see the official Docker daemon image repository, ceph/docker-ceph.

Developing Ceph with Docker Images

The idea for doing fast docker-based Ceph development is illustrated in the following diagram:

In short, there is a builder image that has a very specific dev env, takes a folder that is bind-mounted from the host that contains Ceph, compiles Ceph and maybe other type of artifacts that are needed for the customized version (like plugins). Then, there is a helper script that builds an ansible-deployable image by adding a new layer to the official ceph-daemon image (which is what upstream uses to deploy ceph in containers).

The key here is to structure this process in such a way that only the actual changes (the delta) and not all the install/ subtree get appended as the last layer, since we only want to change what was rebuilt in the dev iteration (I’m using rsync for this). So this is efficient and doesn’t involve packaging. Of course, the first time we append to the official ceph-daemon image, it will result in a big delta (around 1GB for all of ceph’s binaries), but after that only the changed binares get overwritten.

This approach allows us to provide any dependency of our customized version of Ceph by placing in the install/ folder. For example, a developer for project requiring libraries for, say, blkin would define a builder image that takes care of building libblkin.so and putting these in install/lib, as well as compiling the version of Ceph that works with blkin.

Lastly, strictly speaking, we don’t need a builder image in order to generate the custom ceph-daemon image. The dependencies for generating the install/ folder could be on a person’s local machine. BUT, and this is important in the context of Popper, it is preferable to define builder images since that makes the end-to-end workflow entirely reproducible since we know exactly what we used to generate all the binaries.

Jekyll theme inspired by researcher