We want to layer the binaries for our Ceph projects (e.g., customized Ceph, Mantle, and ZLog) on top of the official Ceph daemon image. The idea is that each node in a cluster will pull the same Docker daemon image and start up as a MON, MDS, or OSD. This orchestration would be done with ceph-ansible. For a more detailed description of how the Docker daemon image should be used, see the official Docker daemon image repository, ceph/docker-ceph.
The idea for doing fast docker-based Ceph development is illustrated in the following diagram:
In short, there is a builder image that has a very specific dev env, takes a folder that is bind-mounted from the host that contains Ceph, compiles Ceph and maybe other type of artifacts that are needed for the customized version (like plugins). Then, there is a helper script that builds an ansible-deployable image by adding a new layer to the official ceph-daemon image (which is what upstream uses to deploy ceph in containers).
The key here is to structure this process in such a way that only the actual
changes (the delta) and not all the
install/ subtree get appended as the last
layer, since we only want to change what was rebuilt in the dev iteration (I’m
using rsync for this). So this is efficient and doesn’t involve packaging. Of
course, the first time we append to the official
ceph-daemon image, it will
result in a big delta (around 1GB for all of ceph’s binaries), but after that
only the changed binares get overwritten.
This approach allows us to provide any dependency of our customized version of
Ceph by placing in the
install/ folder. For example, a developer for project
requiring libraries for, say,
blkin would define a builder image that takes
care of building
libblkin.so and putting these in
install/lib, as well as
compiling the version of Ceph that works with
Lastly, strictly speaking, we don’t need a builder image in order to generate
ceph-daemon image. The dependencies for generating the
folder could be on a person’s local machine. BUT, and this is important in the
context of Popper, it is preferable to define builder images since that makes
the end-to-end workflow entirely reproducible since we know exactly what we used
to generate all the binaries.
Jekyll theme inspired by researcher