Quattor | Starting a Site with Aquilon

This page documents how to configure a site with Aquilon, once the Aquilon broker is running, i.e. how to describe the hardware present in the site and how to configure hosts and services. For Aquilon installation, refer to the dedicate page

This document explains the steps required to set up the clusters for the Daily Planet, Superman’s employer. Feel free to replace the names used here by those relevant to your organisation but be sure to remain consistent.

Note: this documentation assumes that you have an aq command available in your path. If this is not the case, define an aq alias to /opt/aquilon/bin/aq.py.

Note: the commands provided in this documentation can be be copy/pasted and should result in a working configuration. If you are not familiar with Aquilon, it is recommended to execute the tutorial once with the proposed parameters and then to redo it changing them to values relevant for your organisation.

Aquilon terminology

These are the basic terms in Aquilon operations:

archetype: The highest possible grouping of hosts into distinctly separate types. Archetypes are a bundle that expresses how to build something. It defines what set of templates to use (for example, what operating systems are available, etc). Hosts therefore require an archetype to define how they are compiled.
broker: The back-end which the aq client communicates with and the owner of all object and production templates. We’ll refer to it also as aqd.
cluster: A group of hosts related in some way, different to an archetype in that hosts may or may not be in a cluster. When grouping hosts into a cluster, an object profile is also built for the cluster, as well as the hosts. Clusters go through a completely different schema and build process to how hosts are built and therefore have a different archetypes.
domain: A set of shared Quattor templates. Entities to be configured live in exactly one domain. A domain is currently implemented as a branch in the template-king git repository.
feature: A reusable configuration template that configures a specific thing but does not in itself describe a complete system. A feature may be included in a personality or anywhere else that PAN template code can be included.
personality: describes the services required but not the instance (selected using plenary template information) specific configuration (hardware used, OS version…).
plenary: Template generated by the broker, on the fly, from an external source such as the Aquilon database.
sandbox: A development area for making changes to Pan templates. Each sandbox must be started from a domain and hosts or clusters may be managed into the sandbox to allow for predeployment testing. Once the template code is fully tested, it must be published back to the broker and then deployhttp://quattor-pan.readthedocs.org/en/latested to the shared domain.
service: A network-based service with well-known clients and servers such as DNS, DHCP, NTP, etc. Each service can have multiple instances. Service instances must be mapped which allows the administrator to model client hosts connecting to different instances based on location and optionally personality.

For more details on these concepts, see the the dedicated page.

Before proceeding

Some understanding of Quattor architecture and the Pan language are expected before reading this document.

We’ll interact with the broker using the aq command, which has lots of sub-commands. Each sub-command has a detailed help. aq sub-commands are generally made of several words that can be separated either by a space or an _. The latter (_) is preferred as it allows to use tab completion when entering command. This is the form used throughout this document. In aq, everything before the first option (introduced by --) is considered as a sub-command word. Options can be abbreviated as long as they are non ambiguous but sub-command words cannot.

In this stage we will be adding a site, so we will use add_* commands. Most of the Aquilon commands have a show_, update_, search_ and del_ counterparts. search_ commands are similar to show_ commands but provide more options to select the objects to show. You are recommended to read the help of each command before actually running it.

Before starting using the aq command, you need get a Kerberos ticket. Use the kinit command to get it. If the Kerberos principal to use is different from the current Linux user, pass its name as a parameter to the kinit command.

Domain and archetypes

Domains

Domains allow to implement a staged deployment workflow. In our example, we will use two domains:

test: used for for testing configuration changes before wide deployment
```
  aq add_domain --domain 'test'
```
prod: used for managing all the production services. During the initialisation of Aquilon database, the prod domain was created in the database but the matching directory was not. Also the Pan compiler path must be fixed to the one declared in /etc/aqd.conf. Below are the command to fix this:
```
  # Replace the directory by what is used at your site
  cd /var/quattor/domains
  cp -R test prod
  cd prod
  git checkout prod
  git branch -d test
  # Fix Pan compiler path
  aq update domain --domain prod --compiler_version prod
```

Archetypes

Archetypes are used to group hosts with common configuration parameters/requirements (e.g. all your network switches) or with similar purposes (e.g. all your cloud hosts). In our example, we will start with one archetype, web_servers.

aq add_archetype --archetype web_servers --compilable

Configuring the Template Library

The template library is a set of generic templates that help to build host descriptions. It provides many building blocks, in particular features and hardware-related templates, ready to use to configure the host hardware, standard OS services and a few more specific middleware, like OpenStack for clouds or UMD for grid.

Importing the Template Library

The template library must be imported in the plenary template area using the plenary_template_library.py script. Download the script and execute it with option --help to display all the options available. A typical execution of this script, to download the template library version 18.3.0, is:

plenary_template_library.py --releases 18.3.0 /var/quattor/cfg/plenary/template-library

The /var/quattor/cfg/plenary/template-library directory will contain a directory corresponding to the version downloaded, making easy to have different versions of the template library. In this directory, a sub-directory is created for each component of the template library: core, os, standard, openstack and grid. As the template library is not supposed to be modified, it is intentionally placed into the plenary templates directory rather than in the template-king Git repository.

Enabling the Template Library

A template library version is enabled in the context of an archetype, using archetype/declarations.pan This template is typically placed in the archetype directory for which you want to enable this version of the template library. Temporarily, for the purposes of this tutorial, we’ll add it to the plenary templates area, under /var/quattor/cfg/plenary/web_servers, where web_servers is the archetype we’ll use in our examples.

The typical contents for declarations.pan can be created with:

mkdir -p /var/quattor/cfg/plenary/web_servers/archetype
cat > web_servers/archetype/declarations.pan <<EOF
# It is VERY IMPORTANT not to use this template for anything else than the initial
# LOADPATH configuration.
#
# This is the first template loaded by an object, before / is populated. This must
# be a declaration template because any change to / will be removed by a subsequent
# line in the object template.

declaration template archetype/declarations;

# Replace by the template library version you want to use
final variable QUATTOR_RELEASE = '18.3.0';

variable LOADPATH = append(SELF, format('template-library/%s/core', QUATTOR_RELEASE));
variable LOADPATH = append(SELF, format('template-library/%s/standard', QUATTOR_RELEASE));

variable DEBUG = debug(format('%s: (template=%S) LOADPATH=%s', OBJECT, TEMPLATE, to_string(LOADPATH)));
EOF

Storing your inventory

The Aquilon database contains the entire inventory of your infrastructure. You may use it as your asset management database, or feed it from your existing one. You will have to store all your hardware, its characteristics and locations.

Geographical locations

Before starting, you have to add the geographical locations of your infrastructure. The meanings of the commands to run should be obvious. We list here an example:

aq add_organization --organization 'daily_planet'
aq add_hub --hub 'pacific' --organization 'daily_planet'
aq add_continent --continent 'america' --hub 'pacific'
aq add_country --country 'us' --continent 'america'
aq add_city --city 'metropolis' --fullname 'Metropolis' --country 'us' --timezone 'dct'
aq add_building --building 'hq' --city 'metropolis' --address '355, 1000 Broadway'
aq add_room --room 'supersecret' --building 'hq' --floor 1

List your own addresses, rooms… but remember that each step depends on the previous ones. If you want to add European countries, you have to insert Europe as a continent first! Also note that object names should not start with a digit and cannot contain spaces.

Your hardware inventory

Next comes your hardware. At the very least, your racks, chassis and servers will be stored here. It is a good place to store other hardware such as switches. Many building blocks, like NIC and hard disks, are available in the template library that we configured in a previous step. The examples here uses them but, if necessary, you can define new hardware components: see the dedicated section.

For racks, you have to specify on which row and column inside that row they are, the room they are located into and an optional fullname. A rack ID will be allocated automatically with a prefix who is the building name and a sequential number in this building.

# 'aq add_rack` returns the rack ID to use in further commands
aq add_rack --fullname forlexluthor --row 3 --column C --room supersecret

Next come servers, switches, routers… we’ll go for servers, machine in Aquilon terms. To be able to define a server, we need to first define its vendor and its CPU model. Models are used to build an actual machine and they cover every component used in the machine. A model is identified by its name and its vendor. Initial Aquilon database already has several vendors and models defines: use aq show_vendor --all and aq show_model --all to see them.

Here we’ll use a simple rack-mounted server but Aquilon also allows to define other types of servers, in particular chassis and blades, and to use them to build a machine. See aq help add_model.

# --comment is optional and is valid also in 'aq add_model'
aq add_vendor --vendor luthorindustries --comments "My preferred vendor"
# xeon-e5-2670v3 is one of the many CPU models provided by the template library
aq add_model --model xeon_e5_2670v3 --vendor intel --type cpu
aq add_model --model thegreatserver --vendor luthorindustries --type rackmount --cpuname xeon_e5_2670v3 \
             --cpunum 2 --memory 98304 --disktype local --diskcontroller sas --disksize 1000 \
             --nicmodel bcm57711
rackid=$(aq search rack --fullname forlexluthor)
aq add_machine --machine testhw --model thegreatserver --rack ${rackid}

Network interfaces

Each machine has a number of network interfaces, with their MAC addresses. They need to be added to the machine object previously created. Aquilon supports different types of interfaces (public, management, VLAN…): see aq help add_interface for details. The MAC address can be explicitly defined for a standard machine or auto-generated for a virtual machine. The interface can belong to a port group. One of the interface must be marked as bootable to be able to define an IP address when adding the host.

In our example, we’ll use a standard public interface and mark it as bootable.

aq add_interface --interface eth0 --machine 'testhw' --mac 'AA:BB:CC:DD:EE:FF'
aq update_interface --interface eth0 --machine 'testhw' --boot

Declaring networks

Once we have the hardware in place, it’s time to declare the networks our hosts live in.

We’ll use the add_network command:

aq add_network --network 'leaks' --ip '192.168.1.0' --netmask '255.255.255.0' --city 'metropolis'
aq add_network --network 'reporters' --ip '192.168.100.0' --netmask '255.255.255.0' --city 'metropolis'

And now the broker knows about two networks, each with its own separate gateways. Networks can live in different network environments (for instance, internal, public…) on which separate policies may apply. Check the help for the aq add_network_environment command.

DNS domains

Our network has some DNS domains. We have to add them:

aq add_dns_domain --dns_domain 'dailyplanet.com'

Declaring hosts

Adding a personality

Before being able to add a host, we need to define a personality. Personalities are collections of smaller chunks called features. A feature is re-usable in many contexts, by different personalities, hardware models…

At this stage we’ll create an empty personality, without associated features. Indeed, a feature definition implies writing some pan templates, something that cannot be done at this stage.

Before creating the personality, we need to define a GRN. A GRN is an organisation group or entity name that will own features and personalities. You can create as many GRNs as you want. Each GRN has a name and a unique ID, called an EON_ID. They are respectively specified with option --grn and --eon_id in aq commands that need them: both are equivalent. To create our first GRN, use the following command:

```bash
aq add_grn --grn test --eon_id 1
```

When creating a personality, here test, it must be attached to an already existing archetype (here web_servers) and to be usable by a host, it must be marked as current with aq promote:

```bash
aq add_personality --personality test --archetype web_servers \
                   --grn test --host_environment dev
aq promote --personality test --archetype web_servers
```

Adding the operating system

Aquilon must also know about the valid OS for each archetype (you can declare several). An OS has a name and a version.

To create an OS object for CentOS 7.x for the archetype web_servers, use the following command:

```bash
aq add_os --osname centos --osversion 7.x --archetype web_servers
```

Once the OS object has been created, it is necessary to create a template associated with it. We will temporarily create this in the plenary template area normally reserved for templates managed by the database. The templates will be moved to their proper location in the next section. Create a file called config.pan in /var/quattor/cfg/plenary/$archetype/os/$osname/$osversion, i.e. /var/quattor/cfg/plenary/web_servers/os/centos/7.x/config.pan. At this stage we will create an empty template:

Currently, we’ll create it in the plenary template area, /var/quattor/cfg/plenary. The template must be called config.pan, under the directory web_servers/os/centos/7.x (archetype, osname, osversion). At this stage we’ll create an empty template:

cd /var/quattor/cfg/plenary/web_servers
template=os/centos/7.x/config
template_ns=$(dirname ${template})
template_file=${template}.pan
mkdir -p ${template_ns}
echo "unique template ${template};" > ${template_file}

Every model object must have a matching template that must be created manually. Templates for the building blocks we used are already provided in the template library, except for the machine model. Again, we’ll temporarily create them in the plenary template area, /var/quattor/cfg/plenary, under the hardware directory. For machines, the template must exist but can be empty.

Note: you need a version later than 18.3.0 for the template library in order to get the templates for the machine building blocks. If your version of the template library is older, update the hardware part from master branch of template-library-standard repository.

cd /var/quattor/cfg/plenary
template=hardware/machine/luthorindustries/thegreatserver
template_ns=$(dirname ${template})
template_file=${template}.pan
mkdir -p ${template_ns}
echo "structure template ${template};" > ${template_file}

Creating the archetype/base and archetype/final templates

In addition to the declarations.pan template described above, each archetype requires 2 further templates located in a directory called <<name-of-archetype>>/archetype, in our example web_servers/archetype, in the plenary template area, /var/quattor/cfg/plenary:

base.pan: this template is the first executed template after the definition of the hardware.
final.pan: it will be the very last one executed in the host profile. It will be created empty at this stage.

We will create empty templates at this stage. To define these templates, use the following commands:

cd /var/quattor/cfg/plenary/web_servers
for template in archetype/base archetype/final
do
    template_file=${template}.pan
    echo "Creating template ${template_file}..."
    echo "unique template ${template};" > ${template_file}
done

Adding and Building the First Host

Now we can declare a host on our testhw machine. A host needs to receive a personality when it is first registered in Aquilon, that can be later changed if necessary.

Now, we use all that information to add a testsrv.dailyplanet.com host:

aq add_host --hostname testsrv.dailyplanet.com --machine testhw --ip 192.168.1.3 \
             --domain test --archetype web_servers --personality test --osname centos --osversion 7.x

Once the host has been added to the Aquilon database, it is possible to build its configuration. This step involves generating the plenary templates describing the host configuration (machine used, OS, personality…) and compiling those templates. This is done with the command aq reconfigure:

aq reconfigure --hostname testsrv.dailyplanet.com

The execution of aq reconfigure will build the host profile in /var/quattor/cfg/domains/test/profiles and compile it. The compilation should succeed: if it fails, review the previous steps. Failure to compile a profile causes it to be removed from the profiles directory.

We now have a host profile compiled but it’s configuration is almost empty. Next steps will show how to describe the target configuration for the host testsrv.dailyplanet.com, using the template library. But before, we need to setup a sandbox which is the place where the configuration changes will be developed.

Creating a Sandbox

Up to now, the templates have been placed into the plenary templates area but this is not a suitable approach for several reasons, in particular:

Plenary templates can be edited only by the Aquilon administrator(s), not by every Aquilon users.
Plenary templates are not versioned
Plenary templates cannot be edited by several users in parallel

The purpose of sandboxes it to allow each Aquilon user to develop its changes without impacting others, until the changes are ready to be published, reviewed and eventually merged into the production ones. Some hosts can be associated with the sandbox to test the changes.

In the following steps, we’ll create a sandbox for the future changes and move into it the templates we wrote until now.

Add an Aquilon user for the current user

A sandbox is associated with an Aquilon user. The Aquilon user is derived from the Kerberos principal and is associated with an existing Linux account via the uid and gid. The Aquilon user name doesn’t have to match the Linux user ID. It must be created with the aq add_user command if the user doesn’t exist already. To list existing users, use:

aq show_user --all

If you are using the aquilon Linux user as suggested, use the following commands:

# Retrieve the Linux `uid` and `gid` for this account
id aquilon
# Create the Aquilon account
aq add_user --user aquilon --uid uid_retrieved --gid gid_retrieved \
            --home aquilon_account_dir --fullname "Aquilon administrator"

To be able to add a sandbox for the user, the Kerberos realm used to authenticate Aquilon users must be trusted. This is not the default when a realm is added. To declare a realm as trusted:

aq update_realm --realm dailyplanet.com --trusted

Note: the actual realm used is based on the Kerberos configuration that was done during Aquilon installation. Use klist command to get it, if you don’t know it.

Sandbox creation

The following command will create the sandbox object and the Git repository associated in /var/quattor/templates/user/sandbox_name with user the Aquilon user matching the Kerberos principal used and sandbox_name the name of the sandbox create. The sandbox is a Git repository created as a clone of the template-king repository (template-king is the Git remote origin for the sandbox repository):

aq add_sandbox --sandbox tutorial

Once the sandbox is created, it is necessary to associate with it the host we want to manage with the sandbox. In (almost) all the Aquilon commands requiring a --sandbox option (except the xxx_sandbox commands), the sandbox name is user/sandbox.

# The following command assumes that the Aquilon user has the same name as the current Linux user
aq manage --sandbox $USER/tutorial --hostname testsrv.dailyplanet.com

You can check that the host testsrv.dailyplanet.com still compiles successfully after its move to the sandbox, with the command used previously:

aq reconfigure --hostname testsrv.dailyplanet.com

Moving Added Templates to the Sandbox

We can now move the templates created at previous steps in the sandbox. Only the template library has been intentionally placed into the plenary templates as this is component that is not intended to be modified by a site. Use the following commands, customising the paths to your configuration if you didn’t adopt the standard layout. In the destination directories, aquilon is the Aquilon user, tutorial is the sandbox name and web_servers is the archetype name.

cd /var/quattor
# Move the hardware templates
mv cfg/plenary/hardware templates/$USER/tutorial
# Create the archetype directory in the sandbox
mkdir templates/$USER/tutorial/web_servers
# Move archetype/base.pan and archetype/final.pan
mv cfg/plenary/web_servers/archetype templates/$USER/tutorial/web_servers
# Move the OS template
mv cfg/plenary/web_servers/os templates/$USER/tutorial/web_servers

Again, check that the host testsrv.dailyplanet.com still compiles successfully after moving to the templates to the sandbox, as previously:

aq reconfigure --hostname testsrv.dailyplanet.com

After checking the successful compile, commit your changes in the sandbox:

cd /var/quattor/templates/$USER/tutorial
git add hardware web_servers
git commit -m 'Initial templates moved to sandbox'

Publishing and Deploying the Sandbox Changes

To illustrate the Aquilon deployment workflow, we can now publish these changes so that they can be reviewed by others and then deploy them into a domain.

To publish the changes so that it is possible to deploy them later:

aq publish --branch tutorial

This pushes the changes to the tutorial branch of the template-king repository. Once the changes are considered ready to be deployed in the test domain, use aq deploy:

# You can add option --dryrun if you want to see what will be done before
# actually doing it
aq deploy --source tutorial --target test

To test that everything is correct, we need to add a second host in the test domain (testsrv is now the tutorial sandbox) and compile it:

rackid=$(aq search rack --fullname forlexluthor)
aq add_machine --machine testhw2 --model thegreatserver --rack ${rackid}
aq add_interface --interface eth0 --machine 'testhw2' --mac 'AA:BB:CC:DD:FF:EE'
aq update_interface --interface eth0 --machine 'testhw2' --bootable
aq add_host --hostname preprodsrv.dailyplanet.com --machine testhw2 --ip 192.168.1.4 \
             --domain test --archetype web_servers --personality test --osname centos --osversion 7.x
aq reconfigure --hostname preprodsrv.dailyplanet.com

At this point we can also check that everything is properly configured with the domain prod and that we can deploy to it, after the validation in domain test. For the sake of simplicity, we’ll use the same host after moving it to the domain prod:

aq deploy --source test --target prod
aq manage --hostname preprodsrv.dailyplanet.com --domain prod

aq manage, when moving a host from between domains or between a sandbox and a domain, is trying to compile the host profile. If it fails, the host is not moved to the new domain/sandbox.

Configuring the base OS

This section shows how to leverage the template library to do the host basic configuration, i.e. the base OS and the Quattor client configuration

Defining the site global variables

It is a good practice to declare the main site variables in one template so that it is easy to call them during the initialisation of every archetype. The choice made in this example is to create a site/config directory at the top-level in the sandbox and to create a template global_variables.pan in it with the following typical contents:

cd /var/quattor/templates/$USER/tutorial
template=site/config/global_variables
template_ns=$(dirname ${template})
template_file=${template}.pan
mkdir -p ${template_ns}
cat > ${template_file} <<EOF
# This templlate is expected to contain only variable definitions that
# apply to the whole site

declaration template ${template};

# OS_FLAVOUR_ENABLED activates generic major version
# i.e sl6.x el7.x ... instead of fixed version (sl6.8, el7.2, ...)
variable OS_FLAVOUR_ENABLED ?= true;

# By default, do not remove postfix service
variable OS_CORE_POSTFIX ?= true;

# Define some variable used on many templates based on the current object template
variable HOSTNAME = hostname_from_object();
variable DOMAIN = domain_from_object('no.default');
variable FULL_HOSTNAME= OBJECT;

# Network parameters are defined by the Aquilon broker
final variable NETWORK_PARAMS = {
    SELF['ip'] = value('/system/network/primary_ip');
    SELF['gateway'] = value('/system/network/default_gateway');
    SELF;
};

# Define the nameservers to use for the site.
variable NAMESERVERS = list("192.168.1.250", "192.168.1.250");

# Define NTP servers
variable NTP_SERVERS = list("ntp.dailyplanet.com");

# Location and format of quattor profile.
variable QUATTOR_PROFILE_URL ?= "http://aquilon-broker.dailyplanet.com/profiles";
variable QUATTOR_PROFILE_FORMAT ?= 'json';

# Location of repository SNAPSHOT
variable YUM_SNAPSHOT_ROOT_URL ?= 'http://yum.dailyplanet.com/snapshots';

# Variable defining the Pan namespace for YUM snapshot templates
variable YUM_SNAPSHOT_NS ?= 'site/repository/snapshot';

# AII specific parameters
variable AII_OSINSTALL_SRV ?= "aii-server.dailyplanet.com";
variable AII_OSINSTALL_ROOT = '/packages/os';

# SELinux state at installation time
variable AII_OSINSTALL_SELINUX ?= 'permissive';
EOF

Note: when migrating from SCDB, you can reuse the contents of the SCDB template site/global_variables.pan in your site directory.

Adding the base information for the archetype

web_servers/archetype/base.pan, currently empty, is the place for all the actions that must be done before configuring the OS and the other components (e.g. personalities) of the host.

A typical web_servers/archetype/base.pan is:

cd /var/quattor/templates/$USER/tutorial
cat > web_servers/archetype/base.pan <<EOF
unique template archetype/base;

# Define the default suffix for YUM-based variant of RPMS configuration templates
# Defined too late by the standard templates in the context of Aquilon
variable RPMS_CONFIG_SUFFIX ?= '-yd';

## This variable will allow to control the YUM snapshot used, if you are using YUM
# snapshots as recommended
variable YUM_SNAPSHOT_DATE ?= '20180409';

# OS distribution used, e.g. centos7 used to build the actual repository name
# Must match the first part of the YUM repository actual name
variable YUM_OS_DISTRIBUTION_NAME ?= 'centos7';

# Timezone definition until the broker put it in the plenary templates
# See https://github.com/quattor/aquilon/issues/91
variable TIMEZONE ?= 'US/Pacific';

# Basic stuff from Quattor core that will be needed in many places
include 'quattor/functions/network';
include 'components/spma/config';

# Template used to define filesystem partition (optional)
variable FILESYSTEM_LAYOUT_CONFIG_SITE ?= "site/filesystems/layout-example";
include "filesystem/config";

# Load Quattor profile schema
include 'quattor/profile_base';

# Required information for every machine
"/system/rootmail" = "admins@dailyplanet.com";
include 'components/accounts/config';
"/software/components/accounts/rootpwd" = "hash_for_very_secret_password";

include 'site/config/global_variables';
EOF

Adding the package repositories

In order to configure packages, we need to define package (YUM) repositories. This involves several steps :

One template is needed to describe each repository available. The choice made in this example is to place them in the directory site/repository/snapshot. After creating this directory, use the following templates as an example: they declare a repository for the base OS, el7x_x86_64, and one for the errata, el7x_x86_64_errata.

cd /var/quattor/templates/$USER/tutorial
template_dir=site/repository/snapshot
mkdir -p ${template_dir}
for repository in el7x_x86_64 el7x_x86_64_errata
do
    template=${template_dir}/${repository}
    template_file=${template}.pan
    repository_suffix=$(echo ${repository} | sed -e 's/^el7x_//')
    cat > ${template_file} <<EOF
structure template ${template};

"name" = "${repository}";
"owner" = "aquilon.support@dailyplanet.com";
"protocols" = list(
    nlist("name","http",
          "url",YUM_SNAPSHOT_ROOT_URL+"/"+YUM_SNAPSHOT_DATE+"/"+YUM_OS_DISTRIBUTION_NAME+"-${repository_suffix}")
);
EOF
done

Create a template to add all the required repository to the profile. In this example, we’ll use site/repository/config/base.pan. First create the directory, then create the template with the following typical content is:

cd /var/quattor/templates/$USER/tutorial
template=site/repository/config/base
template_ns=$(dirname ${template})
template_file=${template}.pan
mkdir -p ${template_ns}
cat > ${template_file} <<EOF
# NOTE: This template should be the LAST thing included in a
# machine profile as other parts of the configuration could modify
# some variables used to configure the package repositories.

unique template site/repository/config/base;

# Repositories related to base OS and quattor client (should be first)
include {'repository/config/os'};

variable DEBUG = debug('OS_REPOSITORY_LIST = ' + to_string(OS_REPOSITORY_LIST) + "\n" +
                       'QUATTOR_REPOSITORY_LIST = ' + to_string(QUATTOR_REPOSITORY_LIST) + "\n");

# Cleanup repository information
include { 'components/spma/repository_cleanup' };
EOF

Adding the final information for the archetype

The very last template to get included into the host configuration is archetype/final.pan, in the archetype directory (here web_servers). This is the place where is typically added the configuration about the Quattor client, the initial installation (AII) and the package repositories, as they tend to depend from several things done during the other part of the configuration. A typical contents for web_servers/archetype/final.pan (currently empty) is:

cd /var/quattor/templates/$USER/tutorial
cat > web_servers/archetype/final.pan <<EOF
unique template archetype/final;

# AII (initial installation) configuration
variable AII_OSINSTALL_EXTRAPKGS ?= list();
variable AII_OSINSTALL_OPTION_TIMEZONE = TIMEZONE;

# Configure Quattor client
include 'quattor/client/config';

# Must be included after AII_OSINSTALL_EXTRAPKGS is populated
include "config/quattor/aii";           # Part of the OS templates

# Configure YUM repositories
# Must be done last
include 'site/repository/config/base';
EOF

Configuring the OS

Aquilon expects the main configuration template for the OS to be in the template os/centos/7.x/config.pan (where centos is the OS name specified for the host and 7.x is the OS version), typically in the archetype directory (in our example, web_servers/os/centos/7.x/config.pan).

A typical template content is:

cd /var/quattor/templates/$USER/tutorial/web_servers
template=os/centos/7.x/config
template_ns=$(dirname ${template})
template_file=${template}.pan
mkdir -p ${template_ns}
cat > ${template_file} <<EOF
unique template ${template};

# It doesn't make sense to modify the variables in this template
# at a later stage, thus they are marked final

final variable NODE_OS_VERSION = 'el7.x-x86_64';

# Bind interface names defined in the profile to the machine interface
# with the matching MAC address
'/system/network/set_hwaddr' = true;

include 'os/config/declarations';
include 'config/core/base';
EOF

This template is calling a template independent of the particular OS name and version to add to the LOADPATH the template library for the selected OS version. In the example, this template is called os/config/declarations.pan (in the archetype directory, i.e. web_servers/os/config/declarations.pan). A typical content for this template is;

cd /var/quattor/templates/$USER/tutorial/web_servers
template=os/config/declarations
template_ns=$(dirname ${template})
template_file=${template}.pan
mkdir -p ${template_ns}
cat > ${template_file} <<EOF
declaration template ${template};

variable LOADPATH = append(SELF, format('template-library/%s/os/%s', QUATTOR_RELEASE, NODE_OS_VERSION));

variable DEBUG = debug(format('%s: (template=%S) LOADPATH=%s', OBJECT, TEMPLATE, to_string(LOADPATH)));
EOF

Compiling the host profile

As usual, we can compile the profile for the host testsrv.dailyplanet.com with all the changes made:

aq reconfigure --hostname testsrv.dailyplanet.com

It should compile successfully. If not, please review carefully all the previous steps.

Publishing and deploying the changes

Once the profile has been successfully rebuilt, it is necessary o publish the changes so that they become visible to other users and they can be used by other domains:

aq publish --branch tutorial

This pushes the changes to the tutorial branch of the template-king repository. Once the changes are considered ready to be deployed in the test domain, use aq deploy:

# You can add option --dryrun if you want to see what will be done before
# actually doing it
aq deploy --source tutorial --target test

We can check that the host in the test domain added previously also compile successfully:

# preprodsrv should be in the prod domain after the previous step
aq manage --hostname preprodsrv.dailyplanet.com --domain test
aq reconfigure --hostname preprodsrv.dailyplanet.com

And when everything is correct, we can deploy the changes to the prod domain and test them:

aq deploy --source test --target prod
aq manage --hostname preprodsrv.dailyplanet.com --domain prod

Summary

We know have a two hosts successfully configured with a configuration that allows to deploy and manage a basic OS configuration. Thanks to the template library, it has been the matter of adding around ten Pan templates that are mainly variable definitions and includes.

Defining the Personality

Personality configuration is staged. That means that when a personality is updated, the change is not visible to the hosts using it until it is promoted as the current version with the aq promote command. When a personality is created and when it is updated, its stage is defined to next (configuration that will be applied to the personality when the new configuration is promoted as the production (current) one.

Personality configuration consists mainly of adding or removing features to it with the aq update_personoality command. More details can be found in the documentation on advanced management tasks, in particular the section related to features.

Adding a feature to test personality

In our example, we will add the feature pakiti) to the personality test used with the host testsrv.dailyplanet.com. Pakiti is a service to collect and display the patch status of hosts against known CVEs. Its configuration is part of the template library, meaning that we don’t need to write the Pan templates associated with the feature.

The pakiti feature is easily added to the Aquilon database with:

aq add feature --feature pakiti --type host --activation dispatch --deactivation reboot \
               --grn test --comment "This feature deploys de Pakiti software"

--comment is optional but it is a good practice to use it to describe what the feature does.

Once added it must be bound to the web-servers personality:

aq bind_feature --feature pakiti --personality test --archetype web_servers

After modifying a personality and before promoting the new configuration as the current one, it is possible to see the differences between the current and the new configuration with aq show_diff:

aq show_diff --archetype web_servers --personality test --personality_stage current --other_stage next

The resulting new personality configuration must now be promoted as current so that testsrv.dailyplanet.com can use the updated personality:

aq promote --personality test --archetype web_servers

This feature requires one variable to be added to our site configuration to define the name of the Pakiti server. Based on whether the current host is the pakiti server or not, the server or the client will be configured. Presently, we will configure the Pakiti client as part of the test personality. To achieve this, add the following variable definition to site/config/global_variables.pan:

# Pakiti server
variable PAKITI_SERVER ?= 'pakiti.dailyplanet.com';

The new host configuration can now be compiled, published an deployed with the usual commands:

aq reconfigure --hostname testsrv.dailyplanet.com
aq publish --branch tutorial
aq deploy --source tutorial --target test

This demonstrates again the power of the template library, we we can rely on features already defined by another site. This is particularly true when configuring hosts with middleware like OpenStack for a cloud or like UMD for the European Grid. Nevertheless some features are site specific and in this case, adding a feature will require to write the associated Pan configuration.

Troubleshooting

Error “branch already exists” when creating a domain

This may happen when using the command aq add_domain if a previous execution of the same domain failed after the corresponding Git branch in template-king repository has been created. Once the problem that caused the command to fail has been fixed, to be able to create the domain it is necessary to first delete this branch.

To do so, go to the template-king repository and delete the branch with the following commands

# Replace the directory by your actual path, if different from the default configuration
cd /var/quattor/template-king
# Replace 'domain' by the actual domain name that you are trying to create
git branch -D domain

Failure to run `aq compile`

If aq compile fails to run properly (not if the compilation results in errors) because of something wrong with the compilation command, look for string ant in /var/quattor/logs/aqd.log and execute the command found in the log as the user who run the aq compile command, adding the option --execdebug, after defining the ANT_HOME environment variable with the value of the ant_home parameter in /etc/aqd.conf.

This allows to see the Java command executed with all its options and to identify the wrong one.

In particular if you get the error Could not find or load main class org.apache.tools.ant.launch.Launcher and if ant has been installed, you should suspect an incorrect definition of ant_home parameter in /etc/aqd.conf. The standard value should be /usr/share/ant and result in /usr/share/ant/lib/ant-launcher.jar being added to the class path.

Starting a Site with Aquilon