17 May 2016

Introduction

When exploring creating Asciidoctor based sites with Awestruct, setting up the Ruby environment was quite a bit of work. Some of the gems that had to be installed were native, and weren’t easy to come by on Windows, let alone compile. Also, installing the required gems wasn’t always that straightforward, depending on the installation history of the host environment. The error messages were not always clear, and quite some searching on the internet was required. For each PC where I wanted to work with Awestruct and Asciidoctor, this work repeated itself, of course with variations.

In addition, as a Windows user, I missed the Linux command line, and a good Vim installation. I managed neither with Babun, cygwin directly, or gvim to get Vim to look and work the way I wanted to. One major issue for me was colouring in the terminal. After a few attempts I gave up, and decided to set up an environment using Vagrant and Puppet. I did explore docker, but again, couldn’t get colouring to work in a reasonable amount of time. Also, this environment should work on both Windows, Linux, and Mac: so Vagrant with Puppet appeared to be a good choice. Using Ansible was not an option, as it does not run on a Windows control machine.

This article is about setting up a Vagrant project, and provisioning the environment with Puppet. The complete code is available on github. This article is not a detailed explanation of all steps, but attempts to describe some of the questions that came up, and how a terminal based development environment could look like.

Make sure that the core.autocrlf for the repository containing the vagrant project is set to input, especially if you also work on this on Windows. The reason is: if you check out the project on Windows with core.autocrlf=true, then provision a Linux machine, files will be copied over with CRLF line-endings, and shell script will stop working. The visible effect is ^M showing up in files and error messages.

Solution Description

The objectives for this development environment were initially to set up a virtual box with git, ruby, nginx, and vim (spf13-vim). Ruby should be set up and configured such that all gems required for an Awestruct/Asciidoctor project can be installed using bundler and the gemfile in the project.

Later, Java, Maven, and JBake were added to the mix, because it was so easy to do. Also, I looked at JBake as an alternative to Awestruct (and Jekyll), and needed a Java/Maven environment for other tasks as well.

The environment should be as simple to use and set up as possible. This resulted in the following additional points:

  • No special users, just the default vagrant user.

  • No special authentication / authorization.

  • No virus protection.

  • Default folder mapping just to the project directory, mapped by /vagrant.

  • Documents to be worked on can be version controlled from within the box if desired.

Setup

  • Install Vagrant.

  • Install Puppet as Windows standalone node. This is needed if you want to use Puppet to install modules to the Vagrant project.

  • Install your favorite version control system client. We’ll use git throughout this tutorial.

  • Open a command line prompt and navigate to a folder of your choice.

  • Create a new subfolder that will contain the Vagrant configuration, and will be the version controlled project. We’ll call it the vagrant folder throughout this tutorial.

Folder structure

The folder structure in the vagrant folder looks like this:

.
├── puppet
│   ├── manifests
│   │   └── vagrant.pp (1)
│   └── modules (2)
│       ├── git
│       ├── java
│       ├── jbake
│       ├── maven
│       ├── nginx
│       ├── shell
│       ├── ssh_keygen
│       ├── stdlib
│       └── vim
├── src (3)
├── tmp (3)
│   └── id_rsa.pub
└── Vagrantfile (4)
1 This file contains the provisioning definition.
2 Modules are used for provisioning.
3 Folders synchronized between the host and the guest system.
4 The file defining the Vagrant configuration.

Vagrant File

The vagrant file, Vagrantfile, is used to define the Vagrant configuration. The current version looks like this:

Vagrant.configure(2) do |config|
  # Base the project on Ubuntu (Trusty):
  config.vm.box = "ubuntu/trusty64"

  config.vm.hostname = "adoc-devenv"

  config.vm.network "private_network", ip: "192.168.33.10"

  # Setting up the private  network doesn't work on windows 10 with Virtualbox 5.0.10, unless you manually
  # check the "VirtualBox NDIS6 Bridged Networking Driver" in the network adapters property page after the adapter has been created.
  # In other words: running vagrant up for the first time will create the adapter, but fail to startup the
  # virtual box. Find the created network adapter in the Windows 10 settings, select its properties, and select (check)
  # "VirtualBox NDIS6 Bridged Networking Driver". Run vagrant up again, this time the box should start normally.
  # As a workaround, a public network can be configured instead, using
  # config.vm.network "public_network"
  # or with a specific IP address:
  # config.vm.network "public_network", ip: "192.168.1.166"

  # Map the local folder containing possible sources to the folder /work/src in the box.
  config.vm.synced_folder "src/", "/work/src"

  # Installation of "VirtualBox Guest Additions" (https://github.com/dotless-de/vagrant-vbguest):
  # Install vbguest plugin with:
  # vagrant plugin install vagrant-vbguest
  # Uncomment the following (change version if necessary)
  # config.vbguest.auto_update = true
  # config.vbguest.iso_path = 'http://download.virtualbox.org/virtualbox/5.0.18/VBoxGuestAdditions_5.0.18.iso'

  # Configure provisioning with Puppet:
  config.vm.provision :puppet do |puppet|
    puppet.manifests_path = "puppet/manifests"
    puppet.module_path = "puppet/modules"
    puppet.manifest_file  = "vagrant.pp"
  end

end

Puppet Modules

Puppet modules implement functionality and/or provide files and directories to provision the target box with.

A module is just a directory in the modules folder. You can create your own module, e.g. to provide shell configuration, the module shell was created. Note its files subfolder, which contains the files that can be copied over by explicit Puppet directives in a manifest.

Existing modules can be used by copying them from the official distribution site to the modules folder, or by installing them using a local (standalone) Puppet installation. For instance, the Java module can be installed by executing, in the puppet folder,

puppet module install puppetlabs-java --modulepath .\modules

Puppet Manifest File

The Puppet manifest file, vagrant.pp, declares the target state of the development environment. We will not discuss it in detail, but mention a few points I used some time to figure out.

Executing Arbitrary Commands

Commands can be executed by using exec. For instance, to update the Ubuntu package manager index files, use

exec { 'apt-update':
  command => '/usr/bin/apt-get update'
}

Installing Software

In general, the package manager of the target box should be used to install software. This makes the puppet manifest more portable. It is accomplished using package, e.g. in order to install unzip, add the following to the vagrant file:

# install unzip
package { 'unzip':
  require => Exec['apt-update'],
  ensure => installed,
}

Files and Directories

A file can be copied from the host to the box using:

# shell configuration
file { '/home/vagrant/.profile':
  owner  => vagrant,
  group  => vagrant,
  mode => 664,
  ensure => 'present',
  source => 'puppet:///modules/shell/.profile',
}

The file .profile has to be stored in folder called files in the shell module folder. Note that the name of the module, shell, is an arbitrary choice of name, but files is not.

A directory can be created using:

file { '/home/vagrant/bin/':
  owner  => vagrant,
  group  => vagrant,
  mode => 755,
  ensure => 'directory',
}

A directory can be copied recursively using:

# install maven
file { '/home/vagrant/bin/maven-3.3.3':
  owner  => vagrant,
  group  => vagrant,
  mode => 664,
  ensure => 'present',
  source => 'puppet:///modules/maven/maven-3.3.3/',
  recurse => true,
  require => File['/home/vagrant/bin/'],
}

Note the recurse ⇒ true property and the requirement for the directory /home/vagrant/bin to be present.

Installing Java

In order to install the Java OpenJDK, install the puppetlabs-java module, and add the following line to vagrant.pp:

# install java
# this installs openjdk-7-jdk on Trusty
include java

Creating ssh-Keys

Install the ssh_keygen module, and add the following line to the vagrant file:

ssh_keygen { 'vagrant': }

This will create a private/public key pair in /home/vagrant/.ssh.

Configuring git

Install the git module, and add the following to the vagrant file (make sure to use your name and email address!):

# install and configure git
include git

git::config { 'user.name':
  value   => 'First Last',
  user    => 'vagrant',
  require => Class['git'],
}

git::config { 'user.email':
  value   => 'first@last.net',
  user    => 'vagrant',
  require => Class['git'],
}

git::config { 'core.autocrlf':
  value   => 'input',
  user    => 'vagrant',
  require => Class['git'],
}

git::config { 'core.editor':
  value   => 'vim',
  user    => 'vagrant',
  require => Class['git'],
}

Shared Folders

Folders can be shared by the host and the box. By default, the vagrant folder on the host is mapped to /vagrant in the box. Two folder were set up for sharing (see Folder Structure): src, which might contain version controlled code you want to work on, and tmp, the content of which should not be version controlled. The src -folder is mapped to /work/src in the box as defined in the vagrant file. tmp is available as /vagrant/tmp in the box.

In the tmp folder, create a .gitignore file with the following content:

*
!.gitignore

Using the Development Environment

  • Open a command line prompt, and navigate to the vagrant folder.

  • Tip: on Windows, use Babun as your shell.

  • Start your environment by typing vagrant up. This might take a while, especially if it is the first time you do this.

  • Connect to your box by executing vagrant ssh. You need to have an ssh client installed in order for this to work.

  • If you want to clone your site’s git repository in your box, you need to copy the public key that has been generated as part of provisioning of your box in the directory /home/vagrant/.ssh to the remote git server. Copy the public key to the /vagrant/tmp mapped folder, and copy it from there, on the host, to e.g. github.

  • Once finished working on your project, run vagrant halt on the host. Do not use vagrant destroy, unless all your data is safely stored away in some other place.

Working With a Site

  • You can test the project using the guarding branch of https://github.com/mgfeller/aablog-foundation.

  • Clone into /home/vagrant/awestruct-project (this is defined as server root in the provided nginx configuration file).

  • Run bundle install to install all required gems.

  • Generate the site using awestruct -g, the result is generated in the _site folder.

  • On your host, open a browser and go to the IP address specified in the Vagrant file, port 90 (as specified in the nginx default configuration file).

Tooling Tips

Using the Screen Tool

From the man-page: "Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells)."

See also the Linux Screen Tutorial.

On your box, in the site folder, start screen by executing screen.

Help is available with screen --help and man screen.

The one command to remember is C-a ?. This will display a list of the available screen commands and their bindings.

Assume you want to run guard in one screen, and edit your files with vim in another:

  • Start screen if you haven’t done so already.

  • In the site directory, start guard using bundle exec guard, you should see some information that guard has been started.

  • Create another screen window by typing C-a c.

  • Start vim and edit your files.

  • Switch between windows by using C-a n (next) or C-a p (previous).

  • Close (aka kill) a window by C-a k and confirming.

  • Closing all windows closes the screen as well.

Useful Vim Commands

Some commands I found useful:

  • Setting current language to asciidoc for syntax colouring: :set syntax=asciidoc

  • Open file explorer on current folder: :e .

  • Open file explorer in a vertical split: :vs.

  • Swap splits, left-right: Ctrl-W r

  • Goto next tab (normal mode): gt

  • Goto buffer by number (e.g. 3): :b 3

  • Goto buffer by name (e.g. doc), supports tab-completion: :b doc

  • Exit insert mode: Ctrl-C

  • Save all open buffers at once: :wa

  • Wrap line (normal mode): gq

  • Setting line width: :set textwidth=132

  • Copy/paste (normal mode): mark the area using v and the cursor, then y to copy. Insert before cursor with P, after cursor with p

Conclusion

Using Vagrant and Puppet makes it easy to create a Linux-based development environment that can be used to provide the tooling necessary for Awestruct/Asciidoctor based sites. At the same time, it serves also as documentation for setting up and configuring an environment on another (Ubuntu-based) machine, if so desired. The possibility to use the proper Linux command line and Vim on a Windows box is a big plus.