Please note that this post is a linear and unedited brain dump of what I did. Many things might have changed meanwhile, and I may have learned how to do things better. This is an experiment in progress.
|This is a continuation of [[lass week’s post||/posts/2016-03-28-cooperating-with-travisci.mdwn]], refreshing my exiscan module using retrospec-puppet. In the last installment I managed to get the full-system tests running in docker. Yay! It turns out that the SUT from within the docker “container” changes my sound settings and causes weird interactions with my desktop environment. Boo! Isolating myself from the contents of the container will be the goal for today. Since it’s the 15th anniversary of finding my significant other, I’ll spend most of the day at the Festival of Colours.|
Isolating the “container”
Of course, I am aware that I’m using docker not in the way it was designed for: I’m booting a “complete” system(d), instead of just “the one process” of the application. On the other hand, the security story of “do not run things that will do unspeakable things to your kernel”, is not very convincing.
Beaker as hypervisor manager doesn’t allow passing arbitrary options to docker, so any of the possibilities of enabling seccomp profiles or disabling capabilities are impossible to use without patching beaker. Additionally it would mean to handcraft a policy that disallows futzing with my desktop (plausible for the audio, a unknown for the colord issue, and who knows what else?). Alternatively, I could just run docker within a VM (yay vagrant) and get proper containment of the system running in docker.
Here’s the relevant vagrant provision command to setup a
puppetlabs/debian-8.2-64-nocm box to be able to run beaker in docker on that VM. This has also the advantage, that anyone following along at home gets a pristine, knwon-good environment to run all of this.
config.vm.provision "shell", inline: <<-SHELL # do not use dash's built-in echo, which doesn't understand -e /bin/echo -e "en_US.UTF-8 UTF-8\nde_AT.UTF-8 UTF-8" > /etc/locale.gen locale-gen apt-get update apt-get install -y apt-transport-https ca-certificates echo deb https://apt.dockerproject.org/repo debian-jessie main > /etc/apt/sources.list.d/docker.list apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D apt-get update apt-get install -y docker-engine ruby-dev ruby bundler build-essential libxml2-dev zlib1g-dev systemctl enable docker systemctl start docker adduser vagrant docker su - vagrant -c 'echo "cd /vagrant" >> /home/vagrant/.bashrc; echo "exec sudo -i" >> /home/vagrant/.bash_history; echo "BEAKER_provision=no BEAKER_destroy=no PUPPET_INSTALL_TYPE=agent BEAKER_set=docker/debian-8 bundle exec rake beaker" >> /home/vagrant/.bash_history; cd /vagrant; bundle install --path=/tmp/bundle; PUPPET_INSTALL_TYPE=agent BEAKER_set=docker/debian-8 bundle exec rake beaker' SHELL
vagrant init puppetlabs/debian-8.2-64-nocm in a temporary directory and added the above commands to the generated
Vagrantfile to get a box that had all the necessary software installed. In the end I added a few commands to the bash history and installed the required gems, so I have them ready when the machine is finalized.
Waiting for gems to compile, I figured adding a little bit more oomph to the box would help too. Researching the exact incantation I also found the linked_clone option, which should help when recycling the box.
config.vm.provider "virtualbox" do |vb| vb.cpus = 4 vb.memory = "2048" vb.linked_clone = true end
While the gems were compiling (the next time after adding more -dev packages) I considered storing the compiled gems in a persistent folder outside of the box. Turns out this is as easy as:
config.vm.synced_folder "/tmp/vbox_gems", "/tmp/bundle"
Depending on your system setup,
/tmp might be a ram disk. In that case you might want to use a different place.
All of this together means that I can run the beaker tests with
vagrant ssh, enter, up, enter. Building the initial docker file also takes a bit of time, so I add a initial beaker run to the vagrant provision step. (See above, there are limits to my no-editing/braindump rule)
Preparing the docker image also takes its time. I guess I’ll mount
/var/lib/docker for persisting the docker images.
The first full run failed on
==> default: Error: Evaluation Error: Error while evaluating a Resource Statement, Could not find declared class exiscan at /tmp/apply_manifest.pp.ndZ2Lh:1:7 on node debian-8-x64
after 20 minutes.
To debug this, look at
docker ps output and connect to that port with ssh. The default password setup by beaker is
vagrant@localhost:/vagrant$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 069b9b964ede 3e3ff809749a "/sbin/init" 2 minutes ago Up 2 minutes 0.0.0.0:32769->22/tcp nauseous_hopper vagrant@localhost:/vagrant$ ssh root@localhost -p 32769 The authenticity of host '[localhost]:32769 ([::1]:32769)' can't be established. ECDSA key fingerprint is 81:ff:c9:01:3f:4d:7b:93:a7:a1:e2:af:3c:4f:ef:a9. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '[localhost]:32769' (ECDSA) to the list of known hosts. root@localhost's password: The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. root@debian-8-x64:~#
One thing that is always annoying with the stripped down systems, like docker images, is missing locales. I thought I had already fixed that by installing the
locales package and genning the data for the en_US.UTF-8 locale, but there is something more going on, keeping the locale data missing, even after reinstalling the locales package. A divert perhaps? Nope:
root@debian-8-x64:~# dpkg-divert --list local diversion of /sbin/initctl to /sbin/initctl.distrib diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash diversion of /bin/sh to /bin/sh.distrib by dash root@debian-8-x64:~#
It turns out that I was completely confused about the actual syntax of the
locale.gen (already fixed above). Leave me alone in my shame!
Back to the original problem of the exiscan class not being found. This does not look healthy.
root@debian-8-x64:/etc/puppetlabs/code/modules# ls exiscan vagrantfilesdefaultconf.dretry vagrantfilesgreylistexim4conf.dmain vagrantfilesspamassassinpostgres tp vagrantfilesdefaultconf.drewrite vagrantfilesgreylistpostgres vagrantjunit vagrantfiles vagrantfilesdefaultconf.drouter vagrantfilesscanner vagrantjunitdebian-8 vagrantfilesdefault vagrantfilesdefaultconf.dtransport vagrantfilesscanneracls vagrantjunitdefault vagrantfilesdefaultconf.d vagrantfilesgreylist vagrantfilesscannerconf.d vagrantmanifests vagrantfilesdefaultconf.dacl vagrantfilesgreylistexim4 vagrantfilesscannerconf.dacl vagranttemplates vagrantfilesdefaultconf.dauth vagrantfilesgreylistexim4conf.d vagrantfilesscannerconf.dmain vagrantfilesdefaultconf.dmain vagrantfilesgreylistexim4conf.dacl vagrantfilesspamassassin root@debian-8-x64:/etc/puppetlabs/code/modules#
It does appear to be a weird issue with copying the module into the docker container. The weird thing is how apparently all the slashes seem to be missing from the filenames, e.g.
vagrantfilesgreylistexim4conf.dmain should be
/vagrant/files/greylist/exim4/conf.d/main. Looking at that, even more suspicious is that these are only the directories, and none of the contained files. Maybe beaker or scp gets confused by copying from /vagrant? I’ll run an experiment from
On the performance side, the next thing I’d need to look into is having a local mirrors of debian, puppetlabs, and rubygems. But that may be a problem for another weekend, meanwhile I’ll bask in the speed of my home cable downlink. Thinking more about this, replacing the puppetlabs’ repo will be a pain, since the list file is installed by beaker internals. ngngngng.
Meanwhile the testrun where I copied the module to
/tmp/foo/bar finished successfully. Let’s see if there is anything “obvious” in beaker’s source. The
scp_to function looks straight forward enough.
@ssh is a
Net::SSH object, but the documentation doesn’t seem to contain any references to scp. Looking into the gemspec I realize that there is a separate net-scp gem. (“this project is in maintenance mode”) Helpfully, there is a progress block attached to beaker’s call of the scp. Its output is stored in the usual beaker result object, so let’s check that output in our setup code.
result = copy_module_to(host, :source => proj_root, :module_name => 'exiscan') puts result.stdout
Again I run afoul of my unfounded optimism. Looking at the
copy_module_to source, the scp_to result is not passed up the call chain. In these cases I usually start modifying the installed gem source that is used by bundler. Having a separate install dir for the vbox makes that not as terrible as it sounds at first. Having that directory persistent across boxes is both a boon and an annoyance: I don’t lose my local changes, but when I forget to clean up, I’ll confuse the heck out of future-me. Except that the retrospec templates install beaker from git, which puts it somewhere else. I remove the git references, since the recent releases are just fine anyways. Trying again.
The debug output is not very helpful:
copying /vagrant/Gemfile.lock: 0/8562 copying /vagrant/Gemfile.lock: 8562/8562 SCP'ed file /vagrant/Gemfile.lock to debian-8-x64:/etc/puppetlabs/code/modules/vagrant
Yes. that’s all. Maybe the chunk size of the ssh connection is interfering? I spy a rsync transfer protocol implementation in the same method, so I try that out before continuing down the scp rabbit hole.
copy_module_to(host, source: proj_root, module_name: 'exiscan', protocol: 'rsync')
411 kB/s is not enough download speed for the debian packages. :-(
rsync requires manual password entry, and then fails, because the target container doesn’t have rsync installed.
Another dead end. Instead of further fighting these demons, I simply switch out the mount on /vagrant to /tmp/exiscan. I bet you it’ll turn out that some weirdness of the vboxsf, and not the path, was the issue when copying. -.-
I’ve also cleaned out the bundler cache to revert my changes to beaker. I’ll not wait for another 20 minutes until this box has built. There is literally not enough time in this day left: it’s after 23:00.
Rebuilding the box suddenly failed with docker complaining about:
Apr 03 15:18:56 localhost docker: time="2016-04-03T15:18:56.491483351-07:00" level=fatal msg="Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get bridge network configurations from store: error while populating kmap: invalid argument"
Why do I even bother??