My development log. Read at your own risk! Please send comments to me by mail or @dev_el_ops.

Profiles Code
xing github
linked.in
twitter
stackoverflow

Latest Blog Posts

Please note that this post is a linear and unedited brain dump of what I did. Many things might have changed meanwhile, and I may have learned how to do things better. This is an experiment in progress.

This is a continuation of lass week's post, refreshing my exiscan module using retrospec-puppet, and a few other nifty technologies. In the last installment I contained docker in a virtualbox to avoid magic action at a distance on my desktop (dialogs popping up, audio being muted).

Picking up the Pieces

Last week, I gave up on this, after provisioning a fresh box with a vboxsf for /var/lib/docker, which confused docker to the point of refusing to work at all. Looking at it with fresh eyes, I have to admit, that it was a very optimistic approach. Rebuilding the box without this mount succeeded, and toock little more than ten minutes.

While waiting on the box to build, I created a unprivileged user to try out whether that fixed my issues, but of course, these being kernel things, and docker running as root, this did change a thing. Finally I could try to hack the boot sequence to avoid fondling the kernel in unspeakable ways, but the box is already running the tests, and I couldn't be bothered. Rome wasn't built in a day either.

Running the Tests

Now that the tests are running without annoyance, I can continue working on the actual code.

Accelerating docker

Jinxed it. For some reason installing packages in docker in the box causes 100% iowait and approx 150kB/s write I/O in the box, while the host is idling. Failing to spot any obvious causes, I suspect the linked_clone option, and disable that one. Additionally I add strace, and lsof to the biult-in tools for the docker image, so next time around I can have a more in-depth look. Writing this, I also need to check next time whether the same I/O "performance" is seen outside the docker container in the box. Maybe docker's underlying FS layering is botching up?

Meanwhile my handiest bash function gets another go:

function unfuck() { vagrant destroy -f "$@" && vagrant up "$@" ; }

Installing packages in the vbox is fast and has a peak I/O bandwidth of 32MB/s. Reading up on docker storage drivers highlights how the default aufs driver apparantly is not suited for "high write activity", suggesting the LVM/devicemapper driver instead.

Adding a second disk

It's really quite easy, and only took me an hour or so to figure out.

config.vm.provider "virtualbox" do |vb|
  disk_image = "/tmp/docker-lvm.vdi"

  vb.customize ['createhd', '--filename', disk_image, '--size', 500 * 1024] unless File.exists?(disk_image)
  vb.customize ['storagectl', :id, '--add', 'sata', '--name', 'SATA', '--hostiocache', 'off' ]
  vb.customize ['storageattach', :id, '--storagectl', 'SATA', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', disk_image]
end

Of course, the disk_image is not bound to the life cycle of the vagrant box, and vagrant doesn't support any extended operations, other than calling out to the underlying hypervisor.

Configuring docker

Docker's devicemapper documentation is pretty comprehensive.

Beware the traps:

# /etc/defaults/docker
# THIS FILE DOES NOT APPLY TO SYSTEMD

It also points to this gem of documentation: "In this example, we’ll assume that your docker.service file looks something like: [...]". I installed the thing from your packages. You should perfectly well know how your config looks like!

The new provision section for docker on DM looks now like this:

DISK=$(/bin/echo -e 'sda\nsdb' | grep -v $(ls /dev/disk/by-id/ata-VBOX_HARDDISK_VB????????-????????-part1 -la | cut -d/ -f 7 | cut -b 1-3))

pvcreate /dev/$DISK
vgcreate docker /dev/$DISK
lvcreate -L 90G -n data docker
lvcreate -L 4G -n metadata docker
mkdir -p /etc/systemd/system/docker.service.d

cat > /etc/systemd/system/docker.service.d/storage-driver.conf <<EOF
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --storage-driver=devicemapper --storage-opt dm.datadev=/dev/docker/data --storage-opt dm.metadatadev=/dev/docker/metadata
EOF

systemctl daemon-reload
systemctl enable docker
systemctl restart docker

It turns out that the linux kernel will hold true to its claim that block device initialization order is random. The next try failed with Device /dev/sda not found (or ignored by filtering). because sdb is now the second disk. This causes the dance with inspecting the /dev/disk/by-id symlinks.

After jumping through enough hoops, dpkg package installation is still slow af. "Good" that I'm "learning" on my "personal" time "here".

Back to the Desktop

Since docker on Debian 8 in a virtualbox doesn't want to cooperate at acceptable levels, I have to accept the fiddling with my devices and resume running docker directly on my main kernel. 2 minutes 22 seconds total runtime for a beaker run from scratch, and 1:55 when the caches are hot and the base beaker image is already built. I could probably shave off another 30 seconds by caching Debian and Puppet locally, or moving some of the setup code into the docker build command in the nodeset.

Fixing the Test

Irrespective of the virt tech used to run the tests, they always failed the idempotency test:

Failures:

  1) exiscan is idempotent
     Failure/Error: expect(result.exit_code).to eq 0

       expected: 0
            got: 2

       (compared using ==)

     # ./spec/acceptance/exiscan_spec.rb:23:in `block (2 levels) in <top (required)>'

because of

Notice: /Stage[main]/Exiscan::Spamassassin/Service[clamav-daemon]/ensure: ensure changed 'stopped' to 'running'

poking into the docker container,

root@debian-8-x64:~# systemctl status clamav-daemon
● clamav-daemon.service - Clam AntiVirus userspace daemon
   Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled)
   Active: inactive (dead)
           start condition failed at Sam 2016-04-09 17:35:12 UTC; 1min 35s ago
           ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc} was not met

which can easily be fixed by waiting for freshclam to finish downloading the virus pattern definitions:

# freshclam needs time to download the patterns
# Instead of failing clamav-daemon, we wait for this to finish
exec { 'wait-for-freshclam':
  command => "/bin/bash -c 'while [ ! -d /var/lib/clamav/daily.cvd ]; do sleep 1; done'",
  creates => '/var/lib/clamav/daily.cvd',
  require => Service['clamav-freshclam'],
  before  => Service['clamav-daemon'],
}

That takes ... a while:

Error: /Stage[main]/Exiscan::Spamassassin/Exec[wait-for-freshclam]/returns: change from notrun to 0 failed: Command exceeded timeout

It would have helped to check for existence (-e) instead of directory (-d).

But at least, now it works!

Finished in 2 minutes 45.4 seconds (files took 27.62 seconds to load)
2 examples, 0 failures

real    3m14.596s

Conclusions

  • I should rename this blog to "Do not ask how the sausages are made."
  • Nesting virtualisation is bad for performance.

Update

To stave off boredom, I upgraded the vagrant box to Debian "stretch", which is what I'm using on my desktop too. This has the kernel version 4.4.0-1 and suddenly the beaker tests run in 3m27s on hot caches. Not too bad - except for all the hassle.

Posted Sat 09 Apr 2016 09:06:58 PM BST Tags:

Please note that this post is a linear and unedited brain dump of what I did. Many things might have changed meanwhile, and I may have learned how to do things better. This is an experiment in progress.

This is a continuation of lass week's post, refreshing my exiscan module using retrospec-puppet. In the last installment I managed to get the full-system tests running in docker. Yay! It turns out that the SUT from within the docker "container" changes my sound settings and causes weird interactions with my desktop environment. Boo! Isolating myself from the contents of the container will be the goal for today. Since it's the 15th anniversary of finding my significant other, I'll spend most of the day at the Festival of Colours.

Isolating the "container"

Of course, I am aware that I'm using docker not in the way it was designed for: I'm booting a "complete" system(d), instead of just "the one process" of the application. On the other hand, the security story of "do not run things that will do unspeakable things to your kernel", is not very convincing.

Beaker as hypervisor manager doesn't allow passing arbitrary options to docker, so any of the possibilities of enabling seccomp profiles or disabling capabilities are impossible to use without patching beaker. Additionally it would mean to handcraft a policy that disallows futzing with my desktop (plausible for the audio, a unknown for the colord issue, and who knows what else?). Alternatively, I could just run docker within a VM (yay vagrant) and get proper containment of the system running in docker.

Here's the relevant vagrant provision command to setup a puppetlabs/debian-8.2-64-nocm box to be able to run beaker in docker on that VM. This has also the advantage, that anyone following along at home gets a pristine, knwon-good environment to run all of this.

config.vm.provision "shell", inline: <<-SHELL
  # do not use dash's built-in echo, which doesn't understand -e
  /bin/echo -e "en_US.UTF-8 UTF-8\nde_AT.UTF-8 UTF-8" > /etc/locale.gen
  locale-gen

  apt-get update
  apt-get install -y apt-transport-https ca-certificates

  echo deb https://apt.dockerproject.org/repo debian-jessie main > /etc/apt/sources.list.d/docker.list
  apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

  apt-get update
  apt-get install -y docker-engine ruby-dev ruby bundler build-essential libxml2-dev zlib1g-dev

  systemctl enable docker
  systemctl start docker

  adduser vagrant docker

  su - vagrant -c 'echo "cd /vagrant" >> /home/vagrant/.bashrc;
    echo "exec sudo -i" >> /home/vagrant/.bash_history;
    echo "BEAKER_provision=no BEAKER_destroy=no PUPPET_INSTALL_TYPE=agent BEAKER_set=docker/debian-8 bundle exec rake beaker" >> /home/vagrant/.bash_history;
    cd /vagrant;
    bundle install --path=/tmp/bundle;
    PUPPET_INSTALL_TYPE=agent BEAKER_set=docker/debian-8 bundle exec rake beaker'
SHELL

I used vagrant init puppetlabs/debian-8.2-64-nocm in a temporary directory and added the above commands to the generated Vagrantfile to get a box that had all the necessary software installed. In the end I added a few commands to the bash history and installed the required gems, so I have them ready when the machine is finalized.

Waiting for gems to compile, I figured adding a little bit more oomph to the box would help too. Researching the exact incantation I also found the linked_clone option, which should help when recycling the box.

config.vm.provider "virtualbox" do |vb|
  vb.cpus = 4
  vb.memory = "2048"
  vb.linked_clone = true
end

While the gems were compiling (the next time after adding more -dev packages) I considered storing the compiled gems in a persistent folder outside of the box. Turns out this is as easy as:

config.vm.synced_folder "/tmp/vbox_gems", "/tmp/bundle"

Depending on your system setup, /tmp might be a ram disk. In that case you might want to use a different place.

All of this together means that I can run the beaker tests with vagrant ssh, enter, up, enter. Building the initial docker file also takes a bit of time, so I add a initial beaker run to the vagrant provision step. (See above, there are limits to my no-editing/braindump rule)

Preparing the docker image also takes its time. I guess I'll mount /var/lib/docker for persisting the docker images.

The first full run failed on

==> default:   Error: Evaluation Error: Error while evaluating a Resource Statement, Could not find declared class exiscan at /tmp/apply_manifest.pp.ndZ2Lh:1:7 on node debian-8-x64

after 20 minutes.

To debug this, look at docker ps output and connect to that port with ssh. The default password setup by beaker is root.

vagrant@localhost:/vagrant$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                   NAMES
069b9b964ede        3e3ff809749a        "/sbin/init"        2 minutes ago       Up 2 minutes        0.0.0.0:32769->22/tcp   nauseous_hopper
vagrant@localhost:/vagrant$ ssh root@localhost -p 32769
The authenticity of host '[localhost]:32769 ([::1]:32769)' can't be established.
ECDSA key fingerprint is 81:ff:c9:01:3f:4d:7b:93:a7:a1:e2:af:3c:4f:ef:a9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:32769' (ECDSA) to the list of known hosts.
root@localhost's password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@debian-8-x64:~#

One thing that is always annoying with the stripped down systems, like docker images, is missing locales. I thought I had already fixed that by installing the locales package and genning the data for the en_US.UTF-8 locale, but there is something more going on, keeping the locale data missing, even after reinstalling the locales package. A divert perhaps? Nope:

root@debian-8-x64:~# dpkg-divert --list
local diversion of /sbin/initctl to /sbin/initctl.distrib
diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash
diversion of /bin/sh to /bin/sh.distrib by dash
root@debian-8-x64:~#

It turns out that I was completely confused about the actual syntax of the locale.gen (already fixed above). Leave me alone in my shame!

Back to the original problem of the exiscan class not being found. This does not look healthy.

root@debian-8-x64:/etc/puppetlabs/code/modules# ls
exiscan                vagrantfilesdefaultconf.dretry      vagrantfilesgreylistexim4conf.dmain  vagrantfilesspamassassinpostgres
tp                 vagrantfilesdefaultconf.drewrite    vagrantfilesgreylistpostgres     vagrantjunit
vagrantfiles               vagrantfilesdefaultconf.drouter     vagrantfilesscanner          vagrantjunitdebian-8
vagrantfilesdefault        vagrantfilesdefaultconf.dtransport  vagrantfilesscanneracls      vagrantjunitdefault
vagrantfilesdefaultconf.d      vagrantfilesgreylist        vagrantfilesscannerconf.d        vagrantmanifests
vagrantfilesdefaultconf.dacl   vagrantfilesgreylistexim4       vagrantfilesscannerconf.dacl     vagranttemplates
vagrantfilesdefaultconf.dauth  vagrantfilesgreylistexim4conf.d     vagrantfilesscannerconf.dmain
vagrantfilesdefaultconf.dmain  vagrantfilesgreylistexim4conf.dacl  vagrantfilesspamassassin
root@debian-8-x64:/etc/puppetlabs/code/modules#

It does appear to be a weird issue with copying the module into the docker container. The weird thing is how apparently all the slashes seem to be missing from the filenames, e.g. vagrantfilesgreylistexim4conf.dmain should be /vagrant/files/greylist/exim4/conf.d/main. Looking at that, even more suspicious is that these are only the directories, and none of the contained files. Maybe beaker or scp gets confused by copying from /vagrant? I'll run an experiment from /tmp/foo/bar.

On the performance side, the next thing I'd need to look into is having a local mirrors of debian, puppetlabs, and rubygems. But that may be a problem for another weekend, meanwhile I'll bask in the speed of my home cable downlink. Thinking more about this, replacing the puppetlabs' repo will be a pain, since the list file is installed by beaker internals. ngngngng.

Meanwhile the testrun where I copied the module to /tmp/foo/bar finished successfully. Let's see if there is anything "obvious" in beaker's source. The scp_to function looks straight forward enough. @ssh is a Net::SSH object, but the documentation doesn't seem to contain any references to scp. Looking into the gemspec I realize that there is a separate net-scp gem. ("this project is in maintenance mode") Helpfully, there is a progress block attached to beaker's call of the scp. Its output is stored in the usual beaker result object, so let's check that output in our setup code.

result = copy_module_to(host, :source => proj_root, :module_name => 'exiscan')
puts result.stdout

Again I run afoul of my unfounded optimism. Looking at the copy_module_to source, the scp_to result is not passed up the call chain. In these cases I usually start modifying the installed gem source that is used by bundler. Having a separate install dir for the vbox makes that not as terrible as it sounds at first. Having that directory persistent across boxes is both a boon and an annoyance: I don't lose my local changes, but when I forget to clean up, I'll confuse the heck out of future-me. Except that the retrospec templates install beaker from git, which puts it somewhere else. I remove the git references, since the recent releases are just fine anyways. Trying again.

The debug output is not very helpful:

copying /vagrant/Gemfile.lock:          0/8562
copying /vagrant/Gemfile.lock:       8562/8562
SCP'ed file /vagrant/Gemfile.lock to debian-8-x64:/etc/puppetlabs/code/modules/vagrant

Yes. that's all. Maybe the chunk size of the ssh connection is interfering? I spy a rsync transfer protocol implementation in the same method, so I try that out before continuing down the scp rabbit hole.

copy_module_to(host, source: proj_root, module_name: 'exiscan', protocol: 'rsync')

411 kB/s is not enough download speed for the debian packages. :-(

rsync requires manual password entry, and then fails, because the target container doesn't have rsync installed.

Another dead end. Instead of further fighting these demons, I simply switch out the mount on /vagrant to /tmp/exiscan. I bet you it'll turn out that some weirdness of the vboxsf, and not the path, was the issue when copying. -.-

I've also cleaned out the bundler cache to revert my changes to beaker. I'll not wait for another 20 minutes until this box has built. There is literally not enough time in this day left: it's after 23:00.

Good night.

Update

Rebuilding the box suddenly failed with docker complaining about:

Apr 03 15:18:56 localhost docker[29239]: time="2016-04-03T15:18:56.491483351-07:00" level=fatal msg="Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get bridge network configurations from store: error while populating kmap: invalid argument"

Why do I even bother??

Posted Sun 03 Apr 2016 11:07:39 PM BST Tags:

Please note that this post is a linear and unedited brain dump of what I did. Many things might have changed meanwhile, and I may have learned how to do things better. This is an experiment in progress.

This is a continuation of yesterday's post, refreshing my exiscan module using retrospec-puppet. Today I'm aiming for getting the release_checks running on travis. This should ensure that going forward, the module stays neat and clean.

At work we recently released a new version of puppetlabs_spec_helper that has a new rake task that will run all checks that we want to pass before releasing a module.

Braindump

The first check is puppet-lint, but I replaced retrospec's default tests with the ones from Puppet Labs' modulesync_configs:

manifests/greylist_db.pp - WARNING: case statement without a default case on line 11
manifests/greylist_db.pp - WARNING: double quoted string containing no variables on line 19
...
manifests/greylist_db.pp - WARNING: mode should be represented as a 4 digit octal value or symbolic mode on line 33
manifests/init.pp - WARNING: double quoted string containing no variables on line 11
...
manifests/init.pp - WARNING: unquoted resource title on line 36
manifests/spamassassin.pp - WARNING: double quoted string containing no variables on line 10
manifests/spamassassin.pp - WARNING: double quoted string containing no variables on line 93
...
manifests/spamassassin_db.pp - WARNING: case statement without a default case on line 12
manifests/spamassassin_db.pp - WARNING: double quoted string containing no variables on line 20
...
manifests/spamassassin_db.pp - WARNING: mode should be represented as a 4 digit octal value or symbolic mode on line 34

I skipped duplicate errors, in the above listing, to keep the length down. In the module's current state puppet-lint complained about 64 things. Some string fixing to do. I'll be right back. Also, apparently I missed some file modes, but it turned out that lint is only very particular about what is acceptable for mode values.

Fixing the "case statement without a default case" warning on greylist_db reminds me that I desperately need to replace all the validation code with puppet 4 type annotations. Ican't wait to be rid of them!

I also fixed up a few other minor issues with the aesthetics as I worked through the files, like removing unnecessary parenthesis.

Syntax checks and specs worked just fine, thanks to yesterday's labour.

The next thing that fails is

14687107      4 lrwxrwxrwx   1 david    david          56 Mär 26 20:37 ./.git/hooks/pre-commit -> /home/david/.retrospec/repos/puppet-git-hooks/pre-commit
rake aborted!
A symlink exists within this directory

This was fixed by Federico Voges, so I upgrade the puppetlabs_spec_helper reference in the Gemfile to the current master, so I can get rid of the false positive, adjust the .travis.yml file and push everything up to see how it is doing. I adjusted the test matrix to something more sensible, other than the eight cells of outdated versions the default had. Finally I also add the caching instructions and request travis' container infrastructure for less resource usage and quicker answer times.

Of course it failed spectacularly, since I've not yet uploaded my tp fix. Referencing the git branch on github instead, et voila! GREEN!

Validating the configuration

Now that the module is at least syntactically valid and doesn't blow up at the first touch, I'll try to add some acceptance/validation tests using beaker-rspec. These tests provision a complete setup and try to verify the functionality as deeply as possible. For the exiscan module, this means setting up a host and trying to send a valid mail, a spam, and a virus, and then check that they are correctly filtered. Luckily the retrospec templates have setup beaker tests for us, all ready to go.

Only a few adjustments and we're ready to go:

spec/acceptance/nodesets/default.yml needs to be changed to be a debian 8 box:

HOSTS:
  debian-8-x64:
    roles:
      - agent
      - default
    platform: debian-8-amd64
    box: puppetlabs/debian-8.2-64-nocm
    hypervisor: vagrant
CONFIG:
  log_level: debug
  type: git

All other nodesets can be deleted, as I'm only targeting one distro.

Here is the first "successful" beaker run:

david@zion:~/git/davids-exiscan$ PUPPET_INSTALL_TYPE=agent BEAKER_destroy=no bundle exec rake beaker
/usr/bin/ruby2.3 -I/home/david/gems/ruby/2.3.0/gems/rspec-core-3.4.4/lib:/home/david/gems/ruby/2.3.0/gems/rspec-support-3.4.1/lib /home/david/gems/ruby/2.3.0/gems/rspec-core-3.4.4/exe/rspec spec/acceptance --color
/home/david/gems/ruby/2.3.0/bundler/gems/beaker-rspec-a617f7bbc3e6/lib/beaker-rspec/helpers/serverspec.rb:43: warning: already initialized constant Module::VALID_OPTIONS_KEYS
/home/david/gems/ruby/2.3.0/gems/specinfra-2.54.1/lib/specinfra/configuration.rb:4: warning: previous definition of VALID_OPTIONS_KEYS was here
Hypervisor for debian-8-x64 is vagrant
Beaker::Hypervisor, found some vagrant boxes to create
==> debian-8-x64: Forcing shutdown of VM...
==> debian-8-x64: Destroying VM and associated drives...
created Vagrantfile for VagrantHost debian-8-x64
Bringing machine 'debian-8-x64' up with 'virtualbox' provider...
==> debian-8-x64: Importing base box 'puppetlabs/debian-8.2-64-nocm'...
Progress: 10%Progress: 20%Progress: 40%Progress: 60%Progress: 70%Progress: 80%Progress: 90%==> debian-8-x64: Matching MAC address for NAT networking...
==> debian-8-x64: Checking if box 'puppetlabs/debian-8.2-64-nocm' is up to date...
==> debian-8-x64: A newer version of the box 'puppetlabs/debian-8.2-64-nocm' is available! You currently
==> debian-8-x64: have version '1.0.0'. The latest is version '1.0.1'. Run
==> debian-8-x64: `vagrant box update` to update.
==> debian-8-x64: Setting the name of the VM: defaultyml_debian-8-x64_1459190378202_18534
==> debian-8-x64: Clearing any previously set network interfaces...
==> debian-8-x64: Preparing network interfaces based on configuration...
    debian-8-x64: Adapter 1: nat
    debian-8-x64: Adapter 2: hostonly
==> debian-8-x64: Forwarding ports...
    debian-8-x64: 22 (guest) => 2222 (host) (adapter 1)
==> debian-8-x64: Running 'pre-boot' VM customizations...
==> debian-8-x64: Booting VM...
==> debian-8-x64: Waiting for machine to boot. This may take a few minutes...
    debian-8-x64: SSH address: 127.0.0.1:2222
    debian-8-x64: SSH username: vagrant
    debian-8-x64: SSH auth method: private key
==> debian-8-x64: Machine booted and ready!
==> debian-8-x64: Checking for guest additions in VM...
    debian-8-x64: The guest additions on this VM do not match the installed version of
    debian-8-x64: VirtualBox! In most cases this is fine, but in rare cases it can
    debian-8-x64: prevent things such as shared folders from working properly. If you see
    debian-8-x64: shared folder errors, please make sure the guest additions within the
    debian-8-x64: virtual machine match the version of VirtualBox you have installed on
    debian-8-x64: your host and reload your VM.
    debian-8-x64:
    debian-8-x64: Guest Additions Version: 4.3.22
    debian-8-x64: VirtualBox Version: 5.0
==> debian-8-x64: Setting hostname...
==> debian-8-x64: Configuring and enabling network interfaces...
==> debian-8-x64: Mounting shared folders...
    debian-8-x64: /vagrant => /home/david/git/davids-exiscan/.vagrant/beaker_vagrant_files/default.yml
configure vagrant boxes (set ssh-config, switch to root user, hack etc/hosts)
Give root a copy of current user's keys, on debian-8-x64

debian-8-x64 19:40:05$ sudo su -c "cp -r .ssh /root/."
  Attempting ssh connection to 10.255.181.132, user: vagrant, opts: {:config=>"/tmp/debian-8-x6420160328-18447-cwdn8w"}
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:67:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:84:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
  Allocated a PTY on debian-8-x64 for "sudo su -c \"cp -r .ssh /root/.\""

debian-8-x64 executed in 0.14 seconds
Update /etc/ssh/sshd_config to allow root login

debian-8-x64 19:40:05$ sudo su -c "sed -ri 's/^#?PermitRootLogin no|^#?PermitRootLogin yes/PermitRootLogin yes/' /etc/ssh/sshd_config"
  Allocated a PTY on debian-8-x64 for "sudo su -c \"sed -ri 's/^#?PermitRootLogin no|^#?PermitRootLogin yes/PermitRootLogin yes/' /etc/ssh/sshd_config\""

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:05$ sudo su -c "service ssh restart"
  Allocated a PTY on debian-8-x64 for "sudo su -c \"service ssh restart\""

debian-8-x64 executed in 0.04 seconds
Warning: ssh connection to debian-8-x64 has been terminated

debian-8-x64 19:40:07$ cat /etc/resolv.conf
  Attempting ssh connection to 10.255.181.132, user: root, opts: {:config=>"/tmp/debian-8-x6420160328-18447-125v0dx"}
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:67:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:84:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
  nameserver 10.0.2.3

debian-8-x64 executed in 0.14 seconds

debian-8-x64 19:40:07$ echo '127.0.0.1  localhost localhost.localdomain
10.255.181.132  debian-8-x64. debian-8-x64
' >> /etc/hosts

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:07$ dpkg -s curl
  Package: curl
  Status: install ok installed
  Priority: optional
  Section: web
  Installed-Size: 325
  Maintainer: Alessandro Ghedini <ghedo@debian.org>
  Architecture: amd64
  Multi-Arch: foreign
  Version: 7.38.0-4+deb8u2
  Depends: libc6 (>= 2.17), libcurl3 (= 7.38.0-4+deb8u2), zlib1g (>= 1:1.1.4)
  Description: command line tool for transferring data with URL syntax
   curl is a command line tool for transferring data with URL syntax, supporting
   DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3,
   POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP.
   .
   curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form
   based upload, proxies, cookies, user+password authentication (Basic, Digest,
   NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and a
   busload of other useful tricks.
  Homepage: http://curl.haxx.se

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:07$ dpkg -s ntpdate
  dpkg-query: package 'ntpdate' is not installed and no information is available
  Use dpkg --info (= dpkg-deb --info) to examine archive files,
  and dpkg --contents (= dpkg-deb --contents) to list their contents.

debian-8-x64 executed in 0.04 seconds
Exited: 1

debian-8-x64 19:40:07$ apt-get update
  Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
  Ign http://http.us.debian.org jessie InRelease
  Get:2 http://security.debian.org jessie/updates/main Sources [122 kB]
  Get:3 http://http.us.debian.org jessie-updates InRelease [142 kB]
  Get:4 http://security.debian.org jessie/updates/main amd64 Packages [221 kB]
  Get:5 http://security.debian.org jessie/updates/main Translation-en [122 kB]
  Get:6 http://http.us.debian.org jessie Release.gpg [2,373 B]
  Get:7 http://http.us.debian.org jessie-updates/main Sources [4,092 B]
  Get:8 http://http.us.debian.org jessie-updates/main amd64 Packages/DiffIndex [1,504 B]
  Get:9 http://http.us.debian.org jessie-updates/main Translation-en/DiffIndex [736 B]
  Get:10 http://http.us.debian.org jessie Release [148 kB]
  Get:11 http://http.us.debian.org jessie-updates/main amd64 2016-03-04-0853.34.pdiff [3,697 B]
  Get:12 http://http.us.debian.org jessie-updates/main amd64 2016-03-26-2053.48.pdiff [527 B]
  Get:13 http://http.us.debian.org jessie-updates/main amd64 2016-03-26-2053.48.pdiff [527 B]
  Get:14 http://http.us.debian.org jessie-updates/main 2016-03-04-0853.34.pdiff [1,371 B]
  Get:15 http://http.us.debian.org jessie-updates/main 2016-03-04-0853.34.pdiff [1,371 B]
  Get:16 http://http.us.debian.org jessie/main Sources [7,058 kB]
  Get:17 http://http.us.debian.org jessie/main amd64 Packages [6,763 kB]
  Get:18 http://http.us.debian.org jessie/main Translation-en [4,582 kB]
  Fetched 19.2 MB in 11s (1,614 kB/s)
  Reading package lists...

debian-8-x64 executed in 11.29 seconds

debian-8-x64 19:40:18$ apt-get install --force-yes  -y ntpdate
  Reading package lists...
  Building dependency tree...
  Reading state information...
  The following extra packages will be installed:
    lockfile-progs
  The following NEW packages will be installed:
    lockfile-progs ntpdate
  0 upgraded, 2 newly installed, 0 to remove and 88 not upgraded.
  Need to get 85.5 kB of archives.
  After this operation, 306 kB of additional disk space will be used.
  Get:1 http://http.us.debian.org/debian/ jessie/main ntpdate amd64 1:4.2.6.p5+dfsg-7+deb8u1 [74.5 kB]
  Get:2 http://http.us.debian.org/debian/ jessie/main lockfile-progs amd64 0.1.17 [11.0 kB]
  debconf: unable to initialize frontend: Dialog
  debconf: (TERM is not set, so the dialog frontend is not usable.)
  debconf: falling back to frontend: Readline
  debconf: unable to initialize frontend: Readline
  debconf: (This frontend requires a controlling tty.)
  debconf: falling back to frontend: Teletype
  dpkg-preconfigure: unable to re-open stdin:
  Fetched 85.5 kB in 0s (123 kB/s)
  Selecting previously unselected package ntpdate.
  (Reading database ...   (Reading database ... 5%  (Reading database ... 10%  (Reading database ... 15%  (Reading database ... 20%  (Reading database ... 25%  (Reading database ... 30%  (Reading database ... 35%  (Reading database ... 40%  (Reading database ... 45%  (Reading database ... 50%  (Reading database ... 55%  (Reading database ... 60%  (Reading database ... 65%  (Reading database ... 70%  (Reading database ... 75%  (Reading database ... 80%  (Reading database ... 85%  (Reading database ... 90%  (Reading database ... 95%  (Reading database ... 100%  (Reading database ...   46030 files and directories currently installed.)
  Preparing to unpack .../ntpdate_1%3a4.2.6.p5+dfsg-7+deb8u1_amd64.deb ...
  Unpacking ntpdate (1:4.2.6.p5+dfsg-7+deb8u1) ...
  Selecting previously unselected package lockfile-progs.
  Preparing to unpack .../lockfile-progs_0.1.17_amd64.deb ...
  Unpacking lockfile-progs (0.1.17) ...
  Processing triggers for man-db (2.7.0.2-5) ...
  Setting up ntpdate (1:4.2.6.p5+dfsg-7+deb8u1) ...
  Setting up lockfile-progs (0.1.17) ...

debian-8-x64 executed in 1.70 seconds

debian-8-x64 19:40:20$ dpkg -s lsb-release
  Package: lsb-release
  Status: install ok installed
  Priority: optional
  Section: misc
  Installed-Size: 97
  Maintainer: Debian LSB Team <debian-lsb@lists.debian.org>
  Architecture: all
  Multi-Arch: foreign
  Source: lsb
  Version: 4.1+Debian13+nmu1
  Depends: python (>= 2.7), python (<< 2.8)
  Recommends: apt
  Suggests: lsb
  Description: Linux Standard Base version reporting utility
   The Linux Standard Base (http://www.linuxbase.org/) is a standard
   core system that third-party applications written for Linux can
   depend upon.
   .
   The lsb-release command is a simple tool to help identify the Linux
   distribution being used and its compliance with the Linux Standard Base.
   LSB conformance will not be reported unless the required metapackages are
   installed.
   .
   While it is intended for use by LSB packages, this command may also
   be useful for programmatically distinguishing between a pure Debian
   installation and derived distributions.
  Homepage: http://www.linuxfoundation.org/collaborate/workgroups/lsb

debian-8-x64 executed in 0.04 seconds
setting local environment on debian-8-x64

debian-8-x64 19:40:20$ getent passwd root
  root:x:0:0:root:/root:/bin/bash

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:20$ mktemp -dt .XXXXXX
  /tmp/.3yIuRm

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:20$ chown root:root /tmp/.3yIuRm

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:20$ echo 'PermitUserEnvironment yes' | cat - /etc/ssh/sshd_config > /tmp/.3yIuRm/sshd_config.permit

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:20$ mv /tmp/.3yIuRm/sshd_config.permit /etc/ssh/sshd_config

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:20$ service ssh restart

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:20$ mkdir -p ~/.ssh

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:20$ chmod 0600 ~/.ssh

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:20$ touch ~/.ssh/environment

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:20$ grep ^PATH=.*\$PATH ~/.ssh/environment

debian-8-x64 executed in 0.04 seconds
Exited: 1

debian-8-x64 19:40:20$ grep ^PATH ~/.ssh/environment

debian-8-x64 executed in 0.04 seconds
Exited: 1

debian-8-x64 19:40:20$ echo "PATH=$PATH" >> ~/.ssh/environment

debian-8-x64 executed in 0.04 seconds
mirroring environment to /etc/profile.d on sles platform host

debian-8-x64 19:40:21$ cat ~/.ssh/environment
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ cat << EOF > /etc/profile.d/beaker_env.sh
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
EOF

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ chmod +x /etc/profile.d/beaker_env.sh

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ source /etc/profile.d/beaker_env.sh

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ echo "/usr/bin"
  /usr/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ echo "/opt/puppet-git-repos/hiera/bin"
  /opt/puppet-git-repos/hiera/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ grep ^PATH=.*\/usr\/bin:\/opt\/puppet\-git\-repos\/hiera\/bin ~/.ssh/environment

debian-8-x64 executed in 0.04 seconds
Exited: 1

debian-8-x64 19:40:21$ grep ^PATH ~/.ssh/environment
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ sed -i -e "s/^PATH=/PATH=\/usr\/bin:\/opt\/puppet\-git\-repos\/hiera\/bin:/" ~/.ssh/environment

debian-8-x64 executed in 0.04 seconds
mirroring environment to /etc/profile.d on sles platform host

debian-8-x64 19:40:21$ cat ~/.ssh/environment
  PATH=/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ cat << EOF > /etc/profile.d/beaker_env.sh
export PATH=/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
EOF

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ chmod +x /etc/profile.d/beaker_env.sh

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ source /etc/profile.d/beaker_env.sh

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ grep ^PATH=.*PATH ~/.ssh/environment

debian-8-x64 executed in 0.04 seconds
Exited: 1

debian-8-x64 19:40:21$ grep ^PATH ~/.ssh/environment
  PATH=/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ sed -i -e "s/^PATH=/PATH=PATH:/" ~/.ssh/environment

debian-8-x64 executed in 0.04 seconds
mirroring environment to /etc/profile.d on sles platform host

debian-8-x64 19:40:21$ cat ~/.ssh/environment
  PATH=PATH:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ cat << EOF > /etc/profile.d/beaker_env.sh
export PATH=PATH:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
EOF

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ chmod +x /etc/profile.d/beaker_env.sh

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:21$ source /etc/profile.d/beaker_env.sh

debian-8-x64 executed in 0.04 seconds
Warning: ssh connection to debian-8-x64 has been terminated

debian-8-x64 19:40:21$ cat ~/.ssh/environment
  Attempting ssh connection to 10.255.181.132, user: root, opts: {:config=>"/tmp/debian-8-x6420160328-18447-125v0dx", :user=>"root"}
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:67:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:84:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
  PATH=PATH:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.13 seconds
Disabling updates.puppetlabs.com by modifying hosts file to resolve updates to 127.0.0.1 on debian-8-x64

debian-8-x64 19:40:21$ echo '127.0.0.1  updates.puppetlabs.com
' >> /etc/hosts

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:22$ wget -O /tmp/puppet.deb http://apt.puppetlabs.com/puppetlabs-release-jessie.deb
  --2016-03-28 11:40:22--  http://apt.puppetlabs.com/puppetlabs-release-jessie.deb
  Resolving apt.puppetlabs.com (apt.puppetlabs.com)...   192.155.89.90  ,   2600:3c03::f03c:91ff:fedb:6b1d
  Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|192.155.89.90|:80...   connected.
  HTTP request sent, awaiting response...   200 OK
  Length:   7368   (7.2K)   [application/x-debian-package]
  Saving to: ‘/tmp/puppet.deb’

       0K     .  .  .  .  .  .  .                                                                                           100% 7.93M=0.001s

  2016-03-28 11:40:22 (7.93 MB/s) - ‘/tmp/puppet.deb’ saved [7368/7368]

debian-8-x64 executed in 0.41 seconds

debian-8-x64 19:40:22$ dpkg -i --force-all /tmp/puppet.deb
  Selecting previously unselected package puppetlabs-release.
  (Reading database ... 46063 files and directories currently installed.)
  Preparing to unpack /tmp/puppet.deb ...
  Unpacking puppetlabs-release (1.0-11) ...
  Setting up puppetlabs-release (1.0-11) ...

  Configuration file `/etc/apt/trusted.gpg.d/puppetlabs-keyring.gpg', does not exist on system.
  Installing new config file as you requested.

  Configuration file `/etc/apt/trusted.gpg.d/puppetlabs-nightly-keyring.gpg', does not exist on system.
  Installing new config file as you requested.

  Configuration file `/etc/apt/sources.list.d/puppetlabs.list', does not exist on system.
  Installing new config file as you requested.

debian-8-x64 executed in 0.22 seconds

debian-8-x64 19:40:22$ apt-get update
  Hit http://security.debian.org jessie/updates InRelease
  Hit http://security.debian.org jessie/updates/main Sources
  Hit http://security.debian.org jessie/updates/main amd64 Packages
  Ign http://http.us.debian.org jessie InRelease
  Ign http://apt.puppetlabs.com jessie InRelease
  Hit http://security.debian.org jessie/updates/main Translation-en
  Hit http://http.us.debian.org jessie-updates InRelease
  Get:1 http://apt.puppetlabs.com jessie Release.gpg [836 B]
  Hit http://http.us.debian.org jessie Release.gpg
  Get:2 http://apt.puppetlabs.com jessie Release [24.3 kB]
  Hit http://http.us.debian.org jessie-updates/main Sources
  Get:3 http://http.us.debian.org jessie-updates/main amd64 Packages/DiffIndex [1,504 B]
  Get:4 http://apt.puppetlabs.com jessie/main Sources [685 B]
  Get:5 http://http.us.debian.org jessie-updates/main Translation-en/DiffIndex [736 B]
  Get:6 http://apt.puppetlabs.com jessie/dependencies Sources [682 B]
  Hit http://http.us.debian.org jessie Release
  Get:7 http://apt.puppetlabs.com jessie/main amd64 Packages [410 B]
  Hit http://http.us.debian.org jessie/main Sources
  Get:8 http://apt.puppetlabs.com jessie/dependencies amd64 Packages [413 B]
  Hit http://http.us.debian.org jessie/main amd64 Packages
  Hit http://http.us.debian.org jessie/main Translation-en
  Ign http://apt.puppetlabs.com jessie/dependencies Translation-en_US
  Ign http://apt.puppetlabs.com jessie/dependencies Translation-en
  Ign http://apt.puppetlabs.com jessie/main Translation-en_US
  Ign http://apt.puppetlabs.com jessie/main Translation-en
  Fetched 29.6 kB in 4s (5,973 B/s)
  Reading package lists...

debian-8-x64 executed in 5.86 seconds

debian-8-x64 19:40:28$ echo "/usr/bin"
  /usr/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:28$ echo "/opt/puppet-git-repos/hiera/bin"
  /opt/puppet-git-repos/hiera/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:28$ grep ^PATH=.*\/usr\/bin:\/opt\/puppet\-git\-repos\/hiera\/bin ~/.ssh/environment
  PATH=PATH:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:28$ grep ^PATH=.*PATH ~/.ssh/environment
  PATH=PATH:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:28$ apt-get install --force-yes  -y puppet
  Reading package lists...
  Building dependency tree...
  Reading state information...
  The following extra packages will be installed:
    augeas-lenses debconf-utils facter hiera javascript-common libaugeas0
    libjs-jquery libruby2.1 libyaml-0-2 puppet-common ruby ruby-augeas
    ruby-hiera ruby-json ruby-rgen ruby-safe-yaml ruby-selinux ruby-shadow
    ruby2.1 rubygems-integration virt-what
  Suggested packages:
    augeas-doc mcollective-common apache2 lighttpd httpd augeas-tools puppet-el
    vim-puppet etckeeper ruby-rrd librrd-ruby ri ruby-dev bundler
  The following NEW packages will be installed:
    augeas-lenses debconf-utils facter hiera javascript-common libaugeas0
    libjs-jquery libruby2.1 libyaml-0-2 puppet puppet-common ruby ruby-augeas
    ruby-hiera ruby-json ruby-rgen ruby-safe-yaml ruby-selinux ruby-shadow
    ruby2.1 rubygems-integration virt-what
  0 upgraded, 22 newly installed, 0 to remove and 88 not upgraded.
  Need to get 5,758 kB of archives.
  After this operation, 24.2 MB of additional disk space will be used.
  Get:1 http://http.us.debian.org/debian/ jessie/main libyaml-0-2 amd64 0.1.6-3 [50.4 kB]
  Get:2 http://http.us.debian.org/debian/ jessie/main augeas-lenses all 1.2.0-0.2+deb8u1 [335 kB]
  Get:3 http://http.us.debian.org/debian/ jessie/main debconf-utils all 1.5.56 [57.6 kB]
  Get:4 http://http.us.debian.org/debian/ jessie/main libruby2.1 amd64 2.1.5-2+deb8u2 [3,278 kB]
  Get:5 http://http.us.debian.org/debian/ jessie/main rubygems-integration all 1.8 [4,514 B]
  Get:6 http://http.us.debian.org/debian/ jessie/main ruby2.1 amd64 2.1.5-2+deb8u2 [275 kB]
  Get:7 http://http.us.debian.org/debian/ jessie/main ruby all 1:2.1.5+deb8u1 [9,620 B]
  Get:8 http://http.us.debian.org/debian/ jessie/main ruby-json amd64 1.8.1-1+b2 [52.1 kB]
  Get:9 http://http.us.debian.org/debian/ jessie/main facter all 2.2.0-1 [74.5 kB]
  Get:10 http://http.us.debian.org/debian/ jessie/main hiera all 1.3.4-1 [18.0 kB]
  Get:11 http://http.us.debian.org/debian/ jessie/main javascript-common all 11 [6,120 B]
  Get:12 http://http.us.debian.org/debian/ jessie/main libaugeas0 amd64 1.2.0-0.2+deb8u1 [256 kB]
  Get:13 http://http.us.debian.org/debian/ jessie/main libjs-jquery all 1.7.2+dfsg-3.2 [97.5 kB]
  Get:14 http://http.us.debian.org/debian/ jessie/main ruby-augeas amd64 0.5.0-2+b2 [11.2 kB]
  Get:15 http://http.us.debian.org/debian/ jessie/main ruby-hiera all 1.3.4-1 [2,890 B]
  Get:16 http://http.us.debian.org/debian/ jessie/main ruby-safe-yaml all 1.0.3-1 [18.4 kB]
  Get:17 http://http.us.debian.org/debian/ jessie/main ruby-shadow amd64 2.3.4-2 [11.0 kB]
  Get:18 http://http.us.debian.org/debian/ jessie/main puppet-common all 3.7.2-4 [1,010 kB]
  Get:19 http://http.us.debian.org/debian/ jessie/main puppet all 3.7.2-4 [25.7 kB]
  Get:20 http://http.us.debian.org/debian/ jessie/main ruby-rgen all 0.7.0-1 [73.5 kB]
  Get:21 http://http.us.debian.org/debian/ jessie/main ruby-selinux amd64 2.3-2 [77.0 kB]
  Get:22 http://http.us.debian.org/debian/ jessie/main virt-what amd64 1.14-1 [12.9 kB]
  debconf: unable to initialize frontend: Dialog
  debconf: (TERM is not set, so the dialog frontend is not usable.)
  debconf: falling back to frontend: Readline
  debconf: unable to initialize frontend: Readline
  debconf: (This frontend requires a controlling tty.)
  debconf: falling back to frontend: Teletype
  dpkg-preconfigure: unable to re-open stdin:
  Fetched 5,758 kB in 6s (957 kB/s)
  Selecting previously unselected package libyaml-0-2:amd64.
  (Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 46070 files and directories currently installed.)
  Preparing to unpack .../libyaml-0-2_0.1.6-3_amd64.deb ...
  Unpacking libyaml-0-2:amd64 (0.1.6-3) ...
  Selecting previously unselected package augeas-lenses.
  Preparing to unpack .../augeas-lenses_1.2.0-0.2+deb8u1_all.deb ...
  Unpacking augeas-lenses (1.2.0-0.2+deb8u1) ...
  Selecting previously unselected package debconf-utils.
  Preparing to unpack .../debconf-utils_1.5.56_all.deb ...
  Unpacking debconf-utils (1.5.56) ...
  Selecting previously unselected package libruby2.1:amd64.
  Preparing to unpack .../libruby2.1_2.1.5-2+deb8u2_amd64.deb ...
  Unpacking libruby2.1:amd64 (2.1.5-2+deb8u2) ...
  Selecting previously unselected package rubygems-integration.
  Preparing to unpack .../rubygems-integration_1.8_all.deb ...
  Unpacking rubygems-integration (1.8) ...
  Selecting previously unselected package ruby2.1.
  Preparing to unpack .../ruby2.1_2.1.5-2+deb8u2_amd64.deb ...
  Unpacking ruby2.1 (2.1.5-2+deb8u2) ...
  Selecting previously unselected package ruby.
  Preparing to unpack .../ruby_1%3a2.1.5+deb8u1_all.deb ...
  Unpacking ruby (1:2.1.5+deb8u1) ...
  Selecting previously unselected package ruby-json.
  Preparing to unpack .../ruby-json_1.8.1-1+b2_amd64.deb ...
  Unpacking ruby-json (1.8.1-1+b2) ...
  Selecting previously unselected package facter.
  Preparing to unpack .../facter_2.2.0-1_all.deb ...
  Unpacking facter (2.2.0-1) ...
  Selecting previously unselected package hiera.
  Preparing to unpack .../archives/hiera_1.3.4-1_all.deb ...
  Unpacking hiera (1.3.4-1) ...
  Selecting previously unselected package javascript-common.
  Preparing to unpack .../javascript-common_11_all.deb ...
  Unpacking javascript-common (11) ...
  Selecting previously unselected package libaugeas0.
  Preparing to unpack .../libaugeas0_1.2.0-0.2+deb8u1_amd64.deb ...
  Unpacking libaugeas0 (1.2.0-0.2+deb8u1) ...
  Selecting previously unselected package libjs-jquery.
  Preparing to unpack .../libjs-jquery_1.7.2+dfsg-3.2_all.deb ...
  Unpacking libjs-jquery (1.7.2+dfsg-3.2) ...
  Selecting previously unselected package ruby-augeas.
  Preparing to unpack .../ruby-augeas_0.5.0-2+b2_amd64.deb ...
  Unpacking ruby-augeas (0.5.0-2+b2) ...
  Selecting previously unselected package ruby-hiera.
  Preparing to unpack .../ruby-hiera_1.3.4-1_all.deb ...
  Unpacking ruby-hiera (1.3.4-1) ...
  Selecting previously unselected package ruby-safe-yaml.
  Preparing to unpack .../ruby-safe-yaml_1.0.3-1_all.deb ...
  Unpacking ruby-safe-yaml (1.0.3-1) ...
  Selecting previously unselected package ruby-shadow.
  Preparing to unpack .../ruby-shadow_2.3.4-2_amd64.deb ...
  Unpacking ruby-shadow (2.3.4-2) ...
  Selecting previously unselected package puppet-common.
  Preparing to unpack .../puppet-common_3.7.2-4_all.deb ...
  Unpacking puppet-common (3.7.2-4) ...
  Selecting previously unselected package puppet.
  Preparing to unpack .../puppet_3.7.2-4_all.deb ...
  Unpacking puppet (3.7.2-4) ...
  Selecting previously unselected package ruby-rgen.
  Preparing to unpack .../ruby-rgen_0.7.0-1_all.deb ...
  Unpacking ruby-rgen (0.7.0-1) ...
  Selecting previously unselected package ruby-selinux.
  Preparing to unpack .../ruby-selinux_2.3-2_amd64.deb ...
  Unpacking ruby-selinux (2.3-2) ...
  Selecting previously unselected package virt-what.
  Preparing to unpack .../virt-what_1.14-1_amd64.deb ...
  Unpacking virt-what (1.14-1) ...
  Processing triggers for man-db (2.7.0.2-5) ...
  Processing triggers for systemd (215-17+deb8u2) ...
  Setting up libyaml-0-2:amd64 (0.1.6-3) ...
  Setting up augeas-lenses (1.2.0-0.2+deb8u1) ...
  Setting up debconf-utils (1.5.56) ...
  Setting up libruby2.1:amd64 (2.1.5-2+deb8u2) ...
  Setting up rubygems-integration (1.8) ...
  Setting up ruby2.1 (2.1.5-2+deb8u2) ...
  Setting up ruby (1:2.1.5+deb8u1) ...
  Setting up ruby-json (1.8.1-1+b2) ...
  Setting up facter (2.2.0-1) ...
  Setting up hiera (1.3.4-1) ...
  Setting up javascript-common (11) ...
  Setting up libaugeas0 (1.2.0-0.2+deb8u1) ...
  Setting up libjs-jquery (1.7.2+dfsg-3.2) ...
  Setting up ruby-augeas (0.5.0-2+b2) ...
  Setting up ruby-hiera (1.3.4-1) ...
  Setting up ruby-safe-yaml (1.0.3-1) ...
  Setting up ruby-shadow (2.3.4-2) ...
  Setting up puppet-common (3.7.2-4) ...
  Setting up puppet (3.7.2-4) ...
  Setting up ruby-rgen (0.7.0-1) ...
  Setting up ruby-selinux (2.3-2) ...
  Setting up virt-what (1.14-1) ...
  Processing triggers for libc-bin (2.19-18+deb8u1) ...
  Processing triggers for systemd (215-17+deb8u2) ...

debian-8-x64 executed in 9.86 seconds

debian-8-x64 19:40:38$ echo "/usr/bin"
  /usr/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:38$ echo "/opt/puppet-git-repos/hiera/bin"
  /opt/puppet-git-repos/hiera/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:38$ grep ^PATH=.*\/usr\/bin:\/opt\/puppet\-git\-repos\/hiera\/bin ~/.ssh/environment
  PATH=PATH:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:38$ grep ^PATH=.*PATH ~/.ssh/environment
  PATH=PATH:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:38$ mkdir -p /etc/puppet

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:38$ puppet agent --configprint hiera_config
  /etc/puppet/hiera.yaml

debian-8-x64 executed in 0.64 seconds

debian-8-x64 19:40:39$ echo '' >> /etc/puppet/hiera.yaml

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:39$ mkdir -p /etc/puppet/modules

debian-8-x64 executed in 0.04 seconds
No examples found.

debian-8-x64 19:40:39$ echo /etc/puppet/modules
  /etc/puppet/modules

debian-8-x64 executed in 0.04 seconds
Using scp to transfer /home/david/git/davids-exiscan to /etc/puppet/modules/exiscan
localhost $ scp /home/david/git/davids-exiscan debian-8-x64:/etc/puppet/modules {:ignore => [".bundle", ".git", ".idea", ".vagrant", ".vendor", "vendor", "acceptance", "bundle", "spec", "tests", "log", ".", ".."]}
going to ignore (?-mix:((\/|\A)\.bundle(\/|\z))|((\/|\A)\.git(\/|\z))|((\/|\A)\.idea(\/|\z))|((\/|\A)\.vagrant(\/|\z))|((\/|\A)\.vendor(\/|\z))|((\/|\A)vendor(\/|\z))|((\/|\A)acceptance(\/|\z))|((\/|\A)bundle(\/|\z))|((\/|\A)spec(\/|\z))|((\/|\A)tests(\/|\z))|((\/|\A)log(\/|\z))|((\/|\A)\.(\/|\z))|((\/|\A)\.\.(\/|\z)))

debian-8-x64 19:40:49$ rm -rf /etc/puppet/modules/exiscan

debian-8-x64 executed in 0.05 seconds

debian-8-x64 19:40:50$ mv /etc/puppet/modules/davids-exiscan /etc/puppet/modules/exiscan

debian-8-x64 executed in 0.04 seconds

debian-8-x64 19:40:50$ puppet module install puppetlabs-stdlib
  Notice: Preparing to install into /etc/puppet/modules ...
  Notice: Downloading from https://forgeapi.puppetlabs.com ...
  Notice: Installing -- do not interrupt ...
  /etc/puppet/modules
  └── puppetlabs-stdlib (v4.11.0)

debian-8-x64 executed in 5.60 seconds

Finished in 16.2 seconds (files took 1 minute 15.57 seconds to load)
0 examples, 0 failures

david@zion:~/git/davids-exiscan$

Once the vagrant hosts have been configured correctly, I can add BEAKER_provision=no and save quite a bit of time by re-using the existing boxes.

The first example is the trivial "it needs to apply correctly":

require 'spec_helper_acceptance'

describe 'exiscan' do
  before(:all) do
    @manifest = <<-PP
      class {
        'exiscan':
          sa_bayes_sql_dsn      => 'sa_bayes_sql_dsn_value',
          sa_bayes_sql_username => 'sa_bayes_sql_username_value',
          greylist_dsn          => 'greylist_dsn',
          greylist_sql_username => 'greylist_sql_username_value',
      }
    PP
    @result = apply_manifest_on default, @manifest, accepted_exitcodes: [0..255]
  end

  it 'runs with changes' do
    expect(@result.exit_code).to eq 2
  end
end

in spec/acceptance/exiscan_spec.rb. Of course this requires also a proper set of modules installed, since beaker-rspec can't re-use the .fixtures.yml from puppetlabs_spec_helper. Or, puppetlabs_spec_helper doesn't correctly set up fixtures for beaker, or whatever. Again the hacked version of tp rears its ugly head.

# Install this module
copy_module_to(host, :source => proj_root, :module_name => 'exiscan')
# List other dependencies here so they are installed on the host
on host, puppet('module', 'install', 'example42-tinydata')
# on host, puppet('module', 'install', 'example42-tp')
install_dev_puppet_module( :source => './spec/fixtures/modules/tp', :module_name => 'tp' )
on host, puppet('module', 'install', 'puppetlabs-stdlib')
on host, puppet('module', 'install', 'ripienaar-concat')

I hack-upon-hack and re-use the already installed ./spec/fixtures/modules/tp. It might be interesting to ask beaker-rspec to just deploy that directory to the master, but then the main symlink becomes a liability. Ugh.

Meanwhile I realize that the retrospec templates are not yet using beaker-puppet_install_helper, and the installed puppet defaulted to 3.7.something. Ugh.

The helper code looks like this:

require 'beaker/puppet_install_helper'
run_puppet_install_helper

Much better, but I need to re-provision the nodes.

/home/david/git/davids-exiscan/spec/spec_helper_acceptance.rb:2:in `require': cannot load such file -- beaker-puppet_install_helper (LoadError)

I might want to add the helper to the Gemfile, though...

It is alive!

One minor change is needed for the test:

@result = apply_manifest_on default, @manifest, accept_all_exit_codes: true, expect_changes: true

This will enable detailed exit codes and allow the spec to do the testing. This needs to go away anyways, as I won't copy and paste this call all over the tests. For now it is fine, as it is, and runs green.

Gareth's Greatest Trick

Recently Gareth Rushgrove has hacked up a patch to run beaker tests in docker on travisci. This is the last thing I want to try for today.

I add the entry to the travisci matrix, adjust the nodeset file:

HOSTS:
  debian-8-x64:
    roles:
      - agent
      - default
    platform: debian-8-amd64
    image: debian:8
    hypervisor: docker
    docker_preserve_image: true
    docker_cmd: '["/sbin/init"]'
    docker_image_commands:
      - 'apt-get install -y net-tools wget'
CONFIG:
  log_level: debug
  trace_limit: 200

and run

david@zion:~/git/davids-exiscan$ PUPPET_INSTALL_TYPE=agent BEAKER_set="docker/debian-8" bundle exec rake beaker
/usr/bin/ruby2.3 -I/home/david/gems/ruby/2.3.0/gems/rspec-core-3.4.4/lib:/home/david/gems/ruby/2.3.0/gems/rspec-support-3.4.1/lib /home/david/gems/ruby/2.3.0/gems/rspec-core-3.4.4/exe/rspec spec/acceptance --color
/home/david/gems/ruby/2.3.0/bundler/gems/beaker-rspec-a617f7bbc3e6/lib/beaker-rspec/helpers/serverspec.rb:43: warning: already initialized constant Module::VALID_OPTIONS_KEYS
/home/david/gems/ruby/2.3.0/gems/specinfra-2.54.1/lib/specinfra/configuration.rb:4: warning: previous definition of VALID_OPTIONS_KEYS was here
Hypervisor for debian-8-x64 is docker
Beaker::Hypervisor, found some docker boxes to create
get
/v1.16/version
{}

Provisioning docker
provisioning debian-8-x64
Creating image
Dockerfile is           FROM debian:8
            RUN apt-get update
            RUN apt-get install -y openssh-server openssh-client curl ntpdate lsb-release
          RUN mkdir -p /var/run/sshd
          RUN echo root:root | chpasswd
          RUN sed -ri 's/^#?PermitRootLogin .*/PermitRootLogin yes/' /etc/ssh/sshd_config
          RUN sed -ri 's/^#?PasswordAuthentication .*/PasswordAuthentication yes/' /etc/ssh/sshd_config
RUN apt-get install -y net-tools wget
          EXPOSE 22
          CMD ["/sbin/init"]
post
/v1.16/build
{:rm=>true}
Dockerfile0000640000000000000000000000076712676310361013323 0ustar00wheelwheel00000000000000          FROM debian:8
            RUN apt-get update
            RUN apt-get install -y openssh-server openssh-client curl ntpdate lsb-release
          RUN mkdir -p /var/run/sshd
          RUN echo root:root | chpasswd
          RUN sed -ri 's/^#?PermitRootLogin .*/PermitRootLogin yes/' /etc/ssh/sshd_config
          RUN sed -ri 's/^#?PasswordAuthentication .*/PasswordAuthentication yes/' /etc/ssh/sshd_config
RUN apt-get install -y net-tools wget
          EXPOSE 22
          CMD ["/sbin/init"]

Creating container from image 52103cc9f308
post
/v1.16/containers/create
{}
{"Image":"52103cc9f308","Hostname":"debian-8-x64"}
Starting container 817b21ba6a5de43a05e3d285bfddcf1aef9add78e397596767453cdead2d6128
post
/v1.16/containers/817b21ba6a5de43a05e3d285bfddcf1aef9add78e397596767453cdead2d6128/start
{}
{"PublishAllPorts":true,"Privileged":true}
get
/v1.16/containers/817b21ba6a5de43a05e3d285bfddcf1aef9add78e397596767453cdead2d6128/json
{}

Using docker server at 0.0.0.0
get
/v1.16/containers/817b21ba6a5de43a05e3d285bfddcf1aef9add78e397596767453cdead2d6128/json
{}

node available as  ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@0.0.0.0 -p 32770
get
/v1.16/containers/817b21ba6a5de43a05e3d285bfddcf1aef9add78e397596767453cdead2d6128/json
{}


debian-8-x64 21:15:47$ cat /etc/resolv.conf
  Attempting ssh connection to 0.0.0.0, user: root, opts: {:password=>"root", :port=>"32770", :forward_agent=>false}
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:67:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:84:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
  Warning: Try 1 -- Host 0.0.0.0 unreachable: Net::SSH::Disconnect - connection closed by remote host
  Warning: Trying again in 3 seconds
  Attempting ssh connection to 0.0.0.0, user: root, opts: {:password=>"root", :port=>"32770", :forward_agent=>false, :user=>"root"}
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:67:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
/home/david/gems/ruby/2.3.0/gems/net-ssh-2.9.4/lib/net/ssh/transport/session.rb:84:in `initialize': Object#timeout is deprecated, use Timeout.timeout instead.
  # Generated by NetworkManager
  nameserver 8.8.8.8

debian-8-x64 executed in 3.09 seconds
[...]

Side note: every time the docker container boots it opens GNOME's colord dialog. Soooo funny. Not.

And, it works! Incredible. Still needs a bit fine-tuning to re-establich Gareth's locale workaround and to install the fixtures, so my tp-hack works, but that's easy enough. Build #13 was the lucky one.

Posted Mon 28 Mar 2016 09:36:12 PM BST Tags:

Please note that this post is a linear and unedited brain dump of what I did. Many things might have changed meanwhile, and I may have learned how to do things better. This is an experiment in progress.

This is a continuation of yesterday's post, refreshing my exiscan module using retrospec-puppet.

Braindump

To recap from yesterday, I stopped after finally getting all the boilerplate code up and running, so that the new tests told me that the exim class was missing. In the current code this was a local version of example42/exim. Alessandro has deprecated that module and replaced it with his example42/tp thing that is mostly data driven. This is something I wanted to have a look at for the longest time.

First, I started with changing the .fixtures.yml to install the example42/tp module which I wanted to try to for the longest time:

fixtures:
    symlinks:
      exiscan: "#{source_dir}"
    forge_modules:
      stdlib: "puppetlabs/stdlib"
      tp: "example42/tp"

Installing:

david@zion:~/git/davids-exiscan$ bundle exec rake spec_prep
Notice: Preparing to install into /home/david/git/davids-exiscan/spec/fixtures/modules ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/home/david/git/davids-exiscan/spec/fixtures/modules
└── puppetlabs-stdlib (v4.11.0)
Notice: Preparing to install into /home/david/git/davids-exiscan/spec/fixtures/modules ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/home/david/git/davids-exiscan/spec/fixtures/modules
└── example42-tp (v1.0.0)
david@zion:~/git/davids-exiscan$

Looking at the manifest, I will start at one of the leaf classes: exiscan::spamassassin, which configures a spamd to use with exiscan. It is a very simple package/file/service class, and should lend itself well to getting all gears greased up, for when I start working on the more complex parts. There is no predefined data blob for tp so I'll stay with a manual implementation until I have used tp with one of the existing blobs.

Fixing test failures

The tests were created using puppet-restrospec and puppet 3.7. Since puppet4 is my target, I've changed the Gemfile to default to puppet ~> 4.0, which got me 4.4.1. The first easy failure is this:

9) exiscan::spamassassin should contain File[/etc/systemd/system/spamassassin.service] with ensure => "present", group => "root", mode => "0644", notify => "Service[spamassassin]", owner => "root", require => "Package[spamassassin]" and source => "puppet:///modules/exiscan/spamassassin/spamassassin.service"
   Failure/Error:
     is_expected.to contain_file('/etc/systemd/system/spamassassin.service')
       .with(
         'ensure'  => 'present',
         'group'   => 'root',
         'mode'    => '0644',
         'notify'  => 'Service[spamassassin]',
         'owner'   => 'root',
         'require' => 'Package[spamassassin]',
         'source'  => 'puppet:///modules/exiscan/spamassassin/spamassassin.service'
       )

     expected that the catalogue would contain File[/etc/systemd/system/spamassassin.service] with mode set to "0644" but it is set to 420
   # ./spec/classes/spamassassin_spec.rb:118:in `block (2 levels) in <top (required)>'

This is both easily explained and fixed. Puppet 3 interprets everything as a string. Specifically, a file mode like 0644 is passed through to the implementation as the four characters '0', '6', '4', and '4'. Puppet 4 on the other hand, sees the digits and interprets them as a number, and - thanks to the leading zero - reads this as a number in base 8, passing 420 to the file type, which would be utterly confused by this. I quote all modes and fix four out of the nine current failures. I also fix all occurrences in the other manifests, so I don't have to think about them later.

Relationships: It's Complicated

5) exiscan::spamassassin should contain File[/var/spool/exim4/scan] with ensure => "directory", group => "clamav", mode => "2750", notify => "Service[exim4]", owner => "Debian-exim" and require => "[Package[$exim::package], Package[clamav-daemon]]"
   Failure/Error:
     is_expected.to contain_file('/var/spool/exim4/scan')
       .with(
         'ensure'  => 'directory',
         'group'   => 'clamav',
         'mode'    => '2750',
         'notify'  => 'Service[exim4]',
         'owner'   => 'Debian-exim',
         'require' => '[Package[$exim::package], Package[clamav-daemon]]'
       )

     expected that the catalogue would contain File[/var/spool/exim4/scan] with require set to "[Package[$exim::package], Package[clamav-daemon]]" but it is set to [:undef, Package[clamav-daemon]{:name=>"clamav-daemon"}]
     Diff:
     @@ -1,2 +1,4 @@
     -[Package[$exim::package], Package[clamav-daemon]]
     +undef
     +
     +Package[clamav-daemon]

   # ./spec/classes/spamassassin_spec.rb:107:in `block (2 levels) in <top (required)>'

The reference to $exim::package is broken. No surprise there, there is no exim class. The rendered value had more issues. I changed the line in the test to this:

'require' => ['Package[exim4-daemon-heavy]', 'Package[clamav-daemon]']

as this is the expected package. The error looks much friendlier now:

Diff:
@@ -1,4 +1,4 @@
-Package[exim4-daemon-heavy]
+undef

 Package[clamav-daemon]

Reconsidering, I add tp::install{exim:} to the manifest, and require Tp::Install[exim], instead of the package. Since depending on the tp:install is only an implementation detail, the tests now need to use rspec-puppet's relatively new support for transitive dependency checks:

it do
  is_expected.to contain_file('/var/spool/exim4/scan')
    .that_requires(['Package[exim4-daemon-heavy]', 'Package[clamav-daemon]'])
    .with(
      'ensure'  => 'directory',
      'group'   => 'clamav',
      'mode'    => '2750',
      'notify'  => 'Service[exim4]',
      'owner'   => 'Debian-exim',
    )
end

After adding the required example42/tinydata module to the fixtures, the tests worked on first try. Unexpected, but welcome. Two more down, three to go.

Basic Resources

The next two failures are a mixture of the above errors. The generated tests for multiple packages and resources are not quite right. I replace them with these improved versions:

['spamassassin', 'libmail-dkim-perl', 'clamav-daemon', 'libclass-dbi-pg-perl', 'spf-tools-perl'].each do |p|
  it { is_expected.to contain_package(p).with('ensure' => 'installed') }
end
['spamassassin', 'clamav-freshclam', 'clamav-daemon'].each do |s|
  it do
    is_expected.to contain_service(s)
      .that_requires(['spamassassin', 'libmail-dkim-perl', 'clamav-daemon', 'libclass-dbi-pg-perl', 'spf-tools-perl'].collect {|p| "Package[#{p}]" })
      .with(
        'enable'  => 'true',
        'ensure'  => 'running',
    )
  end
end

This creates separate examples for each package and service, which helps debugging, when something goes wrong.

The last failure is again a rendering problem, where the content of a file is supplied by a template, which puppet-retrospec (luckily, who wants oodles of config in the test?) did not expand. I just remove the test for content. Checking the service's configuration is best left to the service itself, which will be covered in a beaker test later anyways.

Finished in 1.67 seconds (files took 0.52661 seconds to load)
16 examples, 0 failures

Wohoo!

Shaping up the Design

Now that the first class is passing tests, I can take a step back to think about the new design. Having exiscan::spamassassin install exim is obviously a no-go, as it should not be its concern. For now, I'll claim that the exiscan::spamassassin is a private implementation detail and the parent class will have to take care to setup its environment properly. I move the tp::install to the main class. This also requires the tests to have that class pre-configured:

let(:pre_condition) do
  <<-PP
    class { 'exiscan':
      sa_bayes_sql_dsn => 'place_value_here',
      sa_bayes_sql_username => 'place_value_here',
      greylist_dsn => 'place_value_here',
      greylist_sql_username => 'place_value_here',
    }
  PP
end

Of course, this now requires that the main class' tests pass. The first error is, again, dependencies into the exim class, which I replace with Tp::Install[exim].

During fixing the tests, I sent a PR upstream to improve readability of default error messages, and a issue when rendering values with escapes. Sometimes I think I should go into QA.


Random tip: How to find the start of your test run?

david@zion:~/git/davids-exiscan$ echo ------------- | figlet; bundle exec rspec -fd -c spec/classes/exiscan_spec.rb ;


 _____ _____ _____ _____ _____ _____ _____ _____ _____ _____ _____ _____ _____
|_____|_____|_____|_____|_____|_____|_____|_____|_____|_____|_____|_____|_____|



exiscan
  should contain Tp::Install[exim] with settings_hash => {"package_name"=>"exim4-daemon-heavy"}
  should contain Class[exiscan::spamassassin] with bayes_sql_dsn => "sa_bayes_sql_dsn_value", bayes_sql_password => "s3cr3t", bayes_sql_username => "sa_bayes_sql_username_value" and trusted_networks => "10.0.0.1"
[...]

Another interesting test was something that required a optional parameter set to a non-default value. The generated test was

it do
  is_expected.to contain_class('exiscan::spamassassin_db')
    .with(
      'db_password' => '',
      'db_username' => ''
    )
end

The improved tests looks like this, and test for both cases of sa_bayes_sql_local:

context 'without a local sa_bayes_sql' do
  # sa_bayes_sql_local = false is default
  it do
    is_expected.not_to contain_class('exiscan::spamassassin_db')
  end
end

context 'with a local sa_bayes_sql' do
  let(:params) do
    super().merge({
      sa_bayes_sql_local: true
    })
  end
  it do
    is_expected.to contain_class('exiscan::spamassassin_db')
      .with(
        'db_password' => '',
        'db_username' => ''
      )
  end
end

More Dependencies

After removing some of the more syntactical issues, it turns out that I've converted many dependency chains from Package[exim] -> something local ~> Service[exim] to loops, as Tp::Install[exim] contains Service[exim]. Even worse, it is not really called exim anymore, because that name is dependent on the underlying OS and when I added in the proper OS facts (used elsewhere) everything blew up even harder.

To get a stable base for such dependencies, I've enabled tp to provide stable names for the main package and service resources.

The main class no tests fine, but it directly includes the exiscan::spamassassin so all the tests for that class fail on duplicate resource for the two class declarations (exiscan's and the test's). To keep the tests, I move them into the main exiscan_spec.rb and adapt the to fit. Amongst other things that meant restoring the Package[exim] dependencies and tests.

I've also added the compile test to all contexts that had non-default params set:

it { is_expected.to compile.with_all_deps }

With "good" results:

1) exiscan with a local sa_bayes db should compile into a catalogue without dependency cycles
   Failure/Error: it { is_expected.to compile.with_all_deps }
     error during compilation: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Invalid resource type concat at /home/david/git/davids-exiscan/spec/fixtures/modules/postgresql/manifests/hbaconcat.pp:7:3 at /home/david/git/davids-exiscan/spec/fixtures/modules/postgresql/manifests/dbcreate.pp:38 on node zion.black.co.at
   # ./spec/classes/exiscan_spec.rb:114:in `block (3 levels) in <top (required)>'

Turned out that my concat fixture was broken. Re-downloading fixed it. Revealing a more pertinent error:

1) exiscan with a local sa_bayes db should compile into a catalogue without dependency cycles
   Failure/Error: it { is_expected.to compile.with_all_deps }
     error during compilation: Parameter mode failed on File[spamassassin_3_2_2_initial.sql]: The file mode specification must be a string, not 'Fixnum' at /home/david/git/davids-exiscan/spec/fixtures/modules/exiscan/manifests/spamassassin_db.pp:23
   # ./spec/classes/exiscan_spec.rb:114:in `block (3 levels) in <top (required)>'

File mode, my old nemesis!

Misc Last Words

Another small improvement to the templates, adding the default compile test.

So finally the spec tests for the main class pass. The *_db_spec were generated empty, so I need to have a look at them too. Puppet-lint is also complaining massively about my last-year style. And finally, all of these efforts are for naught, if the module doesn't actually configure exim properly, which needs to be validated on a running system. Luckily, tomorrow is another day off!

Posted Sun 27 Mar 2016 10:53:41 AM BST Tags:

Please note that this post is a linear and unedited brain dump of what I did. Many things might have changed meanwhile, and I may have learned how to do things better. This is an experiment in progress.

In an effort to refresh my hosting solution, I started looking at refreshing some of the modules I built back then. One thing I need to bring forward, for example is The Olde Exiscan Module. Here's a log of what I did towards refreshing it to current standards, using retrospec-puppet.

Install restrospec-puppet

Since the original module (see commits before today) was still using a Modulefile and had no test infrastructure at all, I started out by adding a minimal Gemfile and install retrospec-puppet:

david@zion:~/git/davids-exiscan$ ls
files  manifests  Modulefile  templates
david@zion:~/git/davids-exiscan$ cat Gemfile
source 'https://rubygems.org'

gem 'puppet-retrospec'
gem 'puppet'
david@zion:~/git/davids-exiscan$ bundle install --path=~/gems
Using awesome_print 1.6.1
Using facets 3.0.0
Using facter 2.4.6
Using json_pure 1.8.3
Using trollop 2.1.2
Using bundler 1.11.2
Using hiera 3.1.1
Using retrospec 0.4.0
Using puppet 4.4.1
Using puppet-retrospec 0.12.1
Bundle complete! 2 Gemfile dependencies, 10 gems now installed.
Bundled gems are installed into /home/david/gems.
david@zion:~/git/davids-exiscan$ bundle exec retrospec puppet --help
Generates puppet rspec test code based on the classes and defines inside the manifests directory.

Subcommands:
new_module
new_fact
new_type
new_provider
new_function
  -t, --template-dir=<s>        Path to templates directory (only for overriding Retrospec templates) (default:
                                /home/david/.retrospec/repos/retrospec-puppet-templates)
  -s, --scm-url=<s>             SCM url for retrospec templates (default: https://github.com/nwops/retrospec-templates)
  -b, --branch=<s>              Branch you want to use for the retrospec template repo (default: master)
  -e, --enable-beaker-tests     Enable the creation of beaker tests
  -n, --enable-future-parser    Enables the future parser only during validation
  -v, --version                 Print version and exit
  -h, --help                    Show this message
david@zion:~/git/davids-exiscan$

new_module kept breaking on the already existing code in the module, so I moved everything out of the way and just took the generated files:

david@zion:~/git/davids-exiscan$ bundle exec retrospec puppet new_module --name=davids-exiscan -a 'David Schmitt <david@black.co.at>'
Successfully ran hook: /home/david/.retrospec/repos/retrospec-puppet-templates/clone-hook

The module located at: /home/david/git/davids-exiscan does not exist, do you wish to create it? (y/n): y
 + /home/david/git/davids-exiscan/manifests/
 + /home/david/git/davids-exiscan/manifests/init.pp
 + /home/david/git/davids-exiscan/metadata.json
david@zion:~/git/davids-exiscan$ mv tmp/* ..

The metadata.json and .gitignore needed a few touches to add URLs and such. Then I could commit those changes to build upon them.

Trying to generate the tests, I ran into https://github.com/nwops/puppet-retrospec/issues/54 and started using the future branch, which already has puppet 4 vendored:

gem 'puppet-retrospec', git: 'https://github.com/nwops/puppet-retrospec.git', ref: 'future'

Almost:

david@zion:~/git/davids-exiscan$ bundle install --path=~/gems
Using awesome_print 1.6.1
Using facets 3.0.0
Using facter 2.4.6
Using json_pure 1.8.3
Using trollop 2.1.2
Using bundler 1.11.2
Using hiera 3.1.1
Using retrospec 0.4.0
Using puppet 4.4.1
Using puppet-retrospec 0.12.0 from https://github.com/nwops/puppet-retrospec.git (at future@a300496)
Bundle complete! 2 Gemfile dependencies, 10 gems now installed.
Bundled gems are installed into /home/david/gems.
david@zion:~/git/davids-exiscan$ bundle exec retrospec puppet
Successfully ran hook: /home/david/.retrospec/repos/retrospec-puppet-templates/clone-hook

Attempt to assign a value to unknown setting :parser
david@zion:~/git/davids-exiscan$

Some quick code removal later this problem is also fixed:

Using puppet-retrospec 0.12.0 from file:///home/david/git/puppet-retrospec (at future@a250574)
Bundle complete! 2 Gemfile dependencies, 10 gems now installed.
Bundled gems are installed into /home/david/gems.
david@zion:~/git/davids-exiscan$ bundle exec retrospec puppet
Successfully ran hook: /home/david/.retrospec/repos/retrospec-puppet-templates/clone-hook

Cloning into '/home/david/.retrospec/repos/puppet-git-hooks'...
remote: Counting objects: 524, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 524 (delta 6), reused 0 (delta 0), pack-reused 506
Receiving objects: 100% (524/524), 115.56 KiB | 0 bytes/s, done.
Resolving deltas: 100% (285/285), done.
Checking connectivity... done.
Successfully ran hook: /home/david/.retrospec/repos/retrospec-puppet-templates/pre-hook

!! /home/david/git/davids-exiscan/.bundle/config already exists
 + /home/david/git/davids-exiscan/.fixtures.yml
 + /home/david/git/davids-exiscan/.git/hooks/pre-commit
!! /home/david/git/davids-exiscan/.gitignore already exists and differs from template
 + /home/david/git/davids-exiscan/.puppet-lint.rc
 + /home/david/git/davids-exiscan/.travis.yml
 + /home/david/git/davids-exiscan/DEVELOPMENT.md
!! /home/david/git/davids-exiscan/Gemfile already exists
 + /home/david/git/davids-exiscan/Rakefile
 + /home/david/git/davids-exiscan/Vagrantfile
 + /home/david/git/davids-exiscan/files/.gitkeep
 + /home/david/git/davids-exiscan/spec/
 + /home/david/git/davids-exiscan/spec/acceptance/
 + /home/david/git/davids-exiscan/spec/shared_contexts.rb
 + /home/david/git/davids-exiscan/spec/spec_helper.rb
 + /home/david/git/davids-exiscan/templates/.gitkeep
 + /home/david/git/davids-exiscan/tests/
 + /home/david/git/davids-exiscan/tests/.gitkeep
 + /home/david/git/davids-exiscan/davids-exiscan_schema.yaml
Successfully ran hook: /home/david/.retrospec/repos/retrospec-puppet-templates/post-hook

david@zion:~/git/davids-exiscan$

Remember to replace the bootstrap Gemfile by retrospec's version. I generated a completely noew module and cribbed it from there.

Also, annoying, but understandable, is the lack of puppet-retrospec itself in the Gemfile. Since I haven't installed it in my system, I just re-added my reference to the local checkout I've been hacking on.

Additionally the template defaults to 3.x puppet, which also needed fixing. And https://github.com/nwops/retrospec-templates/pull/6 .

Generating tests now didn't fail anymore, but also didn't generate any tests. In the good old divide and conquer strategy, I've removed all manifests, which might not parse sanely, and replaced them by a trivial class, to see if that would work.

Having no luck with the future branch, I rolled back to the released puppet-retrospec gem and retried everything on a ruby (2.2) that was able to run puppet 3.7, the vendored version. Thankfully Debian provides ruby2.2 and ruby2.2-dev packages, so that was quite "painless".

Corey also hinted at yet unreleased code that will fix these pains.

Did I say "painless" ? ruby2.2 ALSO cannot run puppet 3.7

I've also tried building older versions of ruby with rbenv, a few weeks ago. Doesn't work, because those depend on SSLv2 functions that were removed upstream.

Instead I ripped out the vendored safe_yaml gem that is causing the ruby2.3 issues and replaced it with the normal safe_yaml 1.0.4. This required some persuasion (of the forced kind) so that puppet would start up, but since YAML is only used on the wire, we're not touching those parts anyways.

side note: installing bundler and safe_yaml with gem2.2 from Debian into the 2.2 ruby install created /usr/local/bin/bundler causing even more "fun" after removing ruby2.2.

After more testing, it turned out that just adding safe_yaml to the Gemfile, or installing Debian's ruby-safe-yaml are enough to keep puppet from loading its broken vendored version.

At last the tests are running and are finally complaining about something that is wrong with the actual module itself: it can't find the exim module. A expected error as it is a dependency I did not migrate into the new metadata.json as it was replaced upstream with a tp plugin.

But that will be a story for another day.

Posted Sat 26 Mar 2016 09:19:45 PM GMT Tags:

I've just created a PR implmenting the scanf function for the classical parser. Watch it or the ticket for the merge. According to Henrik it should be just in time for 3.7.5 release.

Henrik also pointed me at the nice fact that the future parser will auto-coerce (fail-safely) strings to numbers in arithmetic expressions. This means $var + 0 will be either the numerical value of $var or a compile error.

Posted Tue 10 Feb 2015 05:14:10 PM GMT Tags:

At the Puppet Contributor Summit 2015 Gent I rediscovered @garethr's wonderful puppet-module-skeleton. It plugs into the puppet module face and enables you to use puppet module generate myname-modulename and get a working sekelton with all testing goodies enabled.

Sadly there were some bring-up issues, which he now started to adress, adding a travis job to keep it that way. More help is always welcome to improve this resource.

Advanced usage

Gareth's tool is great to create a new module. For working with existing modules, modulesync is the way to go. This tool can be configured with a set of templates and data to keep all the tedious meta-bits in a set of a big number of modules aligned.

To cover all bases, modulesync needed a small modification. Now I'm porting over Gareth's skeleton into a modulesync config, so I can apply this to all my modules.

I'm making good progress and will update you as soon as I've got something proper to show.

Posted Sat 07 Feb 2015 08:16:39 AM GMT Tags:

The future parser in puppet is the compatibility shim to use before moving to Puppet 4.0, whose release is imminent. The future parser in 3.7 allows us to run the stable version with the parser of the new version, making manifests and modules ready for a seamless upgrade. Here I'll describe the steps I needded to make this move.

Goal

To make the transition possible and easy, I've two goals. first, I want to make sure that I change as little as possible. This avoids trying too much in a single pass of the codebase, which helps general stability. Secondly, the code needs to run on both the old and the new parser to avoid having a flag day across all people using the example42 modules.

Preparation

First I've upgraded to the most recent stable puppet version (3.7.4). This ensure that I have a up-to-date iteration of the future parser, and I'm developing against the state of the art.

Then I've created a new puppet.conf to use while testing. This way I can leave the rest of my system running without impact while hacking on the future branch. More sensible people would use vagrant, but being my own biggest customer in this prod environment makes things easier.

--- /etc/puppet/puppet.conf 2015-02-04 12:45:02.000000000 +0100
+++ /etc/puppet/future.conf 2015-02-04 12:45:16.000000000 +0100
@@ -32,21 +32,17 @@
   listen = false
   runinterval = 1800
   localconfig = $vardir/localconfig
-  environment = production
+  environment = future
+  masterport = 8888
+  noop = true

 [master]
   bindaddress = 0.0.0.0
   autosign = false

-  environment = production
-  manifest    = /srv/puppet/configuration/manifests/site.pp
-  modulepath  = /srv/puppet/configuration/modules
-  manifestdir=/srv/puppet/configuration/manifests
+  masterport = 8888
+  pidfile = /var/run/puppet/future.pid
+  parser = future
+  environment = future
+  environmentpath = /srv/puppet/environments

Commands:

puppet master --no-daemonize --verbose --config=/etc/puppet/future.conf
puppet agent --test --config=/etc/puppet/future.conf --noop

I hadn't enabled directory environments and manifest directory, so you see that there too. They are required to get access to a more nuanced deployment workflow, caching and getting rid of "import", which helps the autoloader to actually notice that things have changed.

Puppet changes

After those preparations, puppet now can tell me what I'm doing wrong, let's get to the various things to fix:

  • Error: This 'if' statement is not productive. A non productive construct may only be placed last in a block/sequence
  • Error: Evaluation Error: Use of 'import' has been discontinued in favor of a manifest directory.

  • Error: Evaluation Error: No matching entry for selector parameter with value '6': this one is nasty. Here's the code:

    # Cope with Debian's folies
    $debian_isc_era = $::operatingsystem ? {
      /(?i:Ubuntu)/ => $::lsbmajdistrelease ? {
        8       => '5',
        9       => '5',
        default => '6',
      },
      /(?i:Debian)/ => $::lsbmajdistrelease ? {
        5       => '5',
        default => '6',
      },
      default   => '6',
    }
    
    ### Application related parameters
    
    $package = $::operatingsystem ? {
      /(?i:Debian|Ubuntu|Mint)/ => $debian_isc_era ? {
        5 => 'dhcp3-server',
        6 => 'isc-dhcp-server',   # <<<<<  Error HERE
      },
      /(?i:SLES|OpenSuSE)/      => 'dhcp-server',
      default                   => 'dhcp',
    }
    

    The error is flagged on the marked line with "Error HERE", luckily, because this selector has no default case. What happens, is that facter delivers $::lsbmajdistrelease as a string and the ? { 5 => is not matching "5" as it is a different type. The first selectors all have default statements that fall through to the default label due to the mismatch.

    The recommended solution is to use scanf to type-convert safely.

Intermission

Using scanf brings several problems. First, it is only available in the future parser. Secondly, there is no good way to find all instances where the stricter interpretation will cause mismatches.

To actually prepare for a safe and regression-free migration, I've also upgraded travis files to actually test against the future parser.

Or, at least, tried to. The module I was working on proved to be of the older sort and testing @garethr's puppet-module-skeleton didn't really work out either.

Acknowledgements

This post was written at the Puppet Contributor Summit Gent 2015, graciously sponsored by Puppetlabs.

Posted Fri 06 Feb 2015 08:11:38 PM GMT Tags:

Static Resource Dispatch

Something I've already used to good effect is putting data for create_resources into hiera to configure customer stuff. My prime example is this hosting module. On the host where tihs is running, I'm loading the data from a YAML file and pass it on the the customer define. This allows me to configure a customer's domains, databasesr, P.O. boxes and other stuff from a private yaml like this:

---
customers:
  dasz:
    type: owner
    admin_user: david-dasz
    admin_fullname: "David Schmitt"
    db_password: geheim1
    domains:
      dasz.at:
        serial: 2014021900
        additional_rrs:
          - "office.dasz.at.  A 88.198.141.234"
    users:
      david-dasz:
        comment: David Schmitt
    mysql_databases:
      dasz_wordpress:
        password: geheim1
    pg_databases:
      dasz_owncloud:
        password: geheim1

This way I can keep the private data private while still publishing my modules on github without remorse.

Dynamic Resource Dispatch

One thing that bugged me about the above setup is that the hosting::customer is a bit verbose in checking for hash-ness and passing the right type name to create_resources. To rectify this, I've had this idea of doing a dynamic dispatch resource, that takes the data from yaml and builds resources without having to repeat boilerplate code.

$data = {
  'file' => {
    'args' => {
      '/home/david/tmp' => {
        'ensure' => 'directory'
      },
      '/home/david/tmp2' => {
        'ensure' => 'directory'
      },
    },
  },
  'notify' => {
    'args' => {
      'msg1' => {
        'message' => 'hello'
      },
      'msg2' => {
        'message' => 'world'
      },
    },
  },
}

create_resources('drd', $data)

define drd($args) {
  if (is_hash($args)) {
    create_resources($name, $args)
  }
}

This is just a sketch how this could work. The drd define will transform any data from the hash into proper resources. This currently has a few downsides:

  • the additional $args level required
  • there can only be one Drd[file] in the catalog

On the positive side, you can have a drd for each use-case you're having and add customization like a little transformation to the type name to clean up the input data:

define hosting::drd($args) { create_resources("hosting::${name}", $args) }

Automatic Password Rollover

The final one really demonstrates the power of automation. At a client's site the current enterprise security rat race is ensuring that all app's database passwords are rotated once a year. While most databases in our group are PostgreSQL where the apps are authenticaed on the Unix socket and therefore do not need a password (yay!), PuppetDB is only able to use TCP to connect to its database. Using the cache_data and random_password functions from foreman's puppet module and stdlib's strftime, the following call will create a password that changes once a week:

cache_data(strftime('puppetdb_%Y_%W'), random_password(32))

Using this, the puppetdb database now gets a new password every monday morning and the Information Protection Officer is officially "Happiest Person Of The Day". Problem solved!

Posted Wed 21 Jan 2015 04:47:28 PM GMT Tags:

Tonight I'll be talking at the Vienna System Architects Meetup December Edition about holistic systems testing with Beaker.

Here're the tools and repos I'll be talking about:

Posted Wed 17 Dec 2014 07:28:12 PM GMT Tags:

Notes

Some example42 notes
Posted Tue 16 Dec 2014 07:55:08 PM GMT

This blog is powered by ikiwiki.