The metaconfig component

Large fraction of all system administration (and thus also the quattor components) is creating correct configuration files, and if required, restart/reload/… the accompanying service in case of changes.

Generating structured text is the bread and butter of TextRender, as described in the 1st part of this series, and the metaconfig component (together with Test::Quattor::TextRender::Metaconfig unittest class) provides a high-level interface to the creation and testing of typical configuration files.

This document contains a very brief introduction to ncm-metaconfig and a more detailed step-by-step tutorial how to add a new service under metaconfig control.

The usage of TextRender in other components or tools as described in the 3rd part of this series.

ncm-metaconfig

The metaconfig component is a component that has a number of services, each service is the file to generate.

The metaconfig component contains a dict called service where the keys of the dict are the names of the files to be generated.

The pan configuration of such a service can be

include 'metaconfig/myown/schema';
bind "/software/components/metaconfig/services/{/etc/myown.config}/contents" = my_own_type;
prefix "/software/components/metaconfig/services/{/etc/myown.config}";
"module" = "myown/main";
"contents" = dict("some", "data");
"daemons" = dict("myownservice", "reload");

Here /etc/myown.config is the metaconfig service and the file that will be generated by TextRender. The contents is passed to CCM::TextRender as a Element instance, together with the module.

The daemons element is an optional dict of service(s) (of the sysvinit/systemd variety) and their actions. The actions are handled by CAF::Service and these are triggered when the generated text is different from the current one.

Other optional attributes are convert, mode, owner, group, backup and preamble (a fixed header).

`convert` option

As of 16.2, ncm-metaconfig supports the convert option. This option provides access to the predefined conversions from the CCM::TextRender element option and can be used with any module to perform basic and/or common conversions that would otherwise require a TT file (altough a trivial one).

It is important to keep in mind that the conversion is done on the element level of the services contents, and is applied to all relevant elements.

Example usage with builtin module tiny:

prefix "/software/components/metaconfig/services/{/etc/sysconfig/example}";
"module" = "tiny";
"contents" = dict(
    "ABC", "data with spaces",
    "IS_EXAMPLE", true,
);
"convert/yesno" = true;
"convert/singlequote" = true;

will create file /etc/sysconfig/example with content

ABC='data with spaces'
IS_EXAMPLE=yes

Development example

What follows is a step-by-step guide how to add a new service to ncm-metaconfig, which consists of creating

a pan schema for the service
the Template::Toolkit file(s) (TT for short) to generate the text (assuming the builtin TextRender modules are not sufficient)
unittests to verify the expected format

Prepare environment

Start with forking the upstream configuration-modules-core repository, and clone your personal fork in your workspace (replace stdweird with your own github username in the example below). Also add the upstream repository (using https protocol).

GHLOGIN=stdweird
git clone git@github.com:$GHLOGIN/configuration-modules-core.git
cd configuration-modules-core
git remote add upstream https://github.com/quattor/configuration-modules-core

Other requirements are:

recent CAF and CCM installed, and the PERL5LIB environment variable set to the quattor install path so the CAF and CCM perl modules are found. (The installation should take care of the dependencies, including the installation of the Template::Toolkit framework)
recent clone of the template library core repository to be able to use already existing pan types and functions in the schema. Set the environment variable QUATTOR_TEST_TEMPLATE_LIBRARY_CORE to this path (by default, the test framework looks in the parent path of configuration-modules-core).

You can verify your environment by running the metaconfig unittests. No tests should fail when the environment is setup properly.

cd ncm-metaconfig
mvn clean test

Adding a new service

Pick a good and relevant name for the service (in this case we will add the imaginary example service).

Set the variable service in your shell (it is used in further command-line examples).

service=example

Target

Our imaginary example service requires a text configuration file with path /etc/example/exampled.conf and has following structure

name = {
    hosts = server1,server2
    port = 800
    master = FALSE
    description = "My example"
}

where following fields are mandatory:

hosts: a comma separated list of hostnames
port: an integer
master: boolean with possible values TRUE or FALSE
description: a quoted string

The service has also an optional fields option, also a quoted string.

Upon changes of the config file, the exampled service needs to be restarted.

This type of configuration is ideally suited for metaconfig and TT.

Initial structure

Make a new branch where you will work in and which you will use to create the github pull-request (PR) when finished

git checkout -b ${service}_service

Create the initial directory structure (from the ncm-metaconfig path).

cd src/main/metaconfig
mkdir -p $service/tests/{profiles,regexps} $service/pan

Add some typical files (some of the files are not mandatory, but are simply best practice).

cd $service

echo -e "declaration template metaconfig/$service/schema;\n" > pan/schema.pan
echo -e "unique template metaconfig/$service/config;\n\ninclude 'metaconfig/$service/schema';" > pan/config.pan

echo -e "object template config;\n\ninclude 'metaconfig/$service/config';\n" > tests/profiles/config.pan
mkdir tests/regexps/config
echo -e 'Base test for config\n---\nmultiline\n---\n$wontmatch^\n' > tests/regexps/config/base

Commit this initial structure

git add ./
git commit -a -m "initial structure for service $service"

Create the schema

The schema needs to be created in the pan subdirectory of the service directory src/main/metaconfig/$service. The file should be called schema.pan and is a declaration template.

declaration template metaconfig/example/schema;

include 'pan/types';

type example_service = {
    'hosts' :  type_hostname[]
    'port' : type_port
    'master' : boolean
    'description' : string
    'option' ? string
};

long, boolean and string are pan builtin types (see the panbook for more info)
type_hostname is a type that is available from the main pan/types template as part of the core template library.
the template namespace metaconfig/example does not match the location of the file, but this is intentional and is resolved by the unittests. During the tests, the ncm-metaconfig/target/pan directory will be created with correct sub-structure.

Create config template for metaconfig component (optional)

A reference config file can now also be created, with e.g. the type binding to the correct path and configuration of the restart action and the TT module to load. The file config.pan should be created in the same pan directory as schema.pan.

unique template metaconfig/example/config;

include 'metaconfig/example/schema';

bind "/software/components/metaconfig/services/{/etc/example/exampled.conf}/contents" = example_service;

prefix "/software/components/metaconfig/services/{/etc/example/exampled.conf}";
"daemons" = dict(
    "exampled", "restart",
);
"module" = "example/main";

This will expect the TT module with relative filename example/main.tt.

Make TT file to match desired output

Create the main.tt file with content in the src/main/metaconfig/$service directory

name = {
[% FILTER indent -%]
hosts = [% hosts.join(',') %]
port = [% port %]
master = [% master ? "TRUE" : "FALSE" %]
description = "[% description %]"
[%     IF option.defined -%]
option = "[% option %]"
[%     END -%]
[% END -%]
}

FILTER indent creates the indentation
TT can easily introduce newline issues, so be careful if the config files are sensitive to this.

Add unittests

Each unittest consists of 3 parts:

an object template (in src/main/metaconfig/<servicename>/tests/profile)
the resulting profile has the required attributes (ideally including the schema and type bindings)
the test profiles (should try to) use the same path as actual ncm-metaconfig usage wrt to the profile structure
one or more RegexpTests that contain regular expressions that will be tested against the output produced by the TT module and the profile.
multiple files in directory src/main/metaconfig/<servicename>/tests/regexps/<name_of_object_template>
a single file src/main/metaconfig/<servicename>/tests/regexps/<name_of_object_template>
a perl unittest to run all unittests for this service as part of the metaconfig unittests.

The object template is compiled in JSON format using the pan-compiler.

The testsuite takes care of the compilation, TT output generation, and running the tests.

Only test templates with .pan suffix that are either unique, structure or object templates are considered, all other will get an (non-fatal) error message. Subdirectories will not be checked for object templates.

The object template and the (one or more) corresponding RegexpTest(s) together form a single unittest for the service; all unittests for this single service are ran via in a single perl src/test/perl/service-example.t unittest, and should look like

use Test::More;
use Test::Quattor::TextRender::Metaconfig;
my $u = Test::Quattor::TextRender::Metaconfig->new(
    service => 'example',
    )->test();
done_testing;

(To be complete, the ncm-metaconfig component has lots of perl service unittests, together with the perl unittests for the component itself).

simple unittest

The easiest example is a single object template with a single regexp file.

profile

The default pan basepath for the TextRender attributes like module and contents is /metaconfig.

Create the profile tests/profiles/simple.pan as follows:

object template simple;

"/metaconfig/module" = "example/main";
prefix "/metaconfig/contents";
"hosts" = list("server1", "server2");
"port" = 800;
"master" = false;
"description" = "My example";

the schema is not validated in this simple template, but it can easily be done by adding

include 'metaconfig/example/schema';
bind "/metaconfig/example/contents" = example_service;

But the preferred way is to create a proper config.pan file and use that (see config example below).

regular expression

Make a 3 block text file tests/regexps/simple, with --- as block separator as follows

Simple test
---
unordered
nomultiline
---
name
hosts
port
master
description

This will search the output for the words name, hosts, port, master and description.

This is good for illustrating the principle, but is a lousy unittest. Check the config unittest below for proper testing.

The filename simple has to match the object template you want to test with (in this case the simple.pan template).

Location flags in the RegexpTest

The required attributes for ncm-metaconfig (e.g. module and contents) are retrieved from the location default /metaconfig (e.g. the contents will $cfg->getElement('/metaconfig/contents')->getTree().

To select another path, 2 additional location flags are supproted:

an absolute path starting with a single / is interpreted as a metaconfig service (/etc/config will result in contents from $cfg->getElement('/software/componentes/metaconfig/service/{/etc/config}/contents')->getTree();
an absolute path starting with 2 /s is interpreted as an absolute panpath (e.g. //some/path will look for contents from $cfg->getElement('/some/path/contents')->getTree().

verify

You can verify this single unittest for the example service using

QUATTOR_TEST_SUITE_FILTER=simple mvn clean test -Dunittest=service-example.t

The QUATTOR_TEST_SUITE_FILTER environment variable is a regular expression pattern that will filter the tests to run (matching tests are run).

(Run this from the configuration-modules-core/ncm-metaconfig directory)

config based unittest

It is better to use a full blown template as will be used in the actual profiles. The added advantage here is the config.pan and schema.pan from the pan directory are tested as well.

profile

The profile tests/profiles/config.pan is similar to the simple one (after all, we want to set the same values), but by targetting metaconfig usage, a different prefix is required.

object template config;

include 'metaconfig/example/config';

prefix "/software/components/metaconfig/services/{/etc/example/exampled.conf}/contents";
"hosts" = list("server1", "server2");
"port" = 800;
"master" = false;
"description" = "My example";

The type binding and definition of the TT module are part of the pan/config.pan template, and this usage is very close to actual usage.

regular expressions

We will now make several regexptests, each in their own file and grouped in a directory called config (matching the object profile name). The filenames in the directory are not relevant (but no addiditional directory structure is allowed).

We need to set the location flag to point to the test infrastructure which metaconfig-controlled file this is supposed to test.

In principle only one of the regexp tests should set this flag (and if multiple ones are set, they all have to be equal). You cannot test different metaconfig file paths from the same profile.

Lets start with a regexptest identical to the simple test above, tests/regexps/config/base:

Simple base test
---
/etc/example/exampled.conf
unordered
nomultiline
---
name
hosts
port
master
description

A 2nd improved regexptest tests/regexps/config/not_so_simple uses the default flags multiline and ordered, where the regular expressions are all interpreted as multiline regular expressions.

Basic multiline test
---
/etc/example/exampled.conf
---
^name
^\s{4}hosts
^\s{4}port
^\s{4}master
^\s{4}description
= ### COUNT 5

This test also uses the directive ### COUNT X (with leading space; X is number, can be 0 or more), where this regular expression is expected to occur exactly X times (in this case, we expect 5 = characters). The COUNT directive ignores the ordering; itsimply is the total number of matches.

A 3rd regexptest tests/regexps/config/neg checks if certain regular expression do not match using the negate flag.

Basic negate test
---
/etc/example/exampled.conf
negate
---
^hosts
^port
^master
^description

This tests that the expected fields can’t start at the beginning of the line, whitespace must be inserted before. (The FILTER indent TT inserts 4 spaces, as tested with the \s{4} in the multiline regexp above.)

If one only needs to check that a single regular expression does not occur, one can also use ### COUNT X , without having to make a separate regexp test with the negate flag.

(Setting the negate flag silently ignores the order flag).

A 4th regexp test tests/regexps/config/value uses full value checks, which is interesting to have, but harder to maintain and review.

Basic value test
---
/etc/example/exampled.conf
---
^name\s=\s\{
^\s{4}hosts\s=\sserver1,server2$
^\s{4}port\s=\s800$
^\s{4}master\s=\sFALSE$
^\s{4}description\s=\s"My example"$
^}$

verify

You can verify this single unittest for the example service using

QUATTOR_TEST_SUITE_FILTER=config mvn clean test -Dunittest=service-example.t

(this will run all 4 regexptest files)

other possible tests

a test for the optional option field: a new test profile is required that has the optional field configured, and it also requires one or more regexp tests to verify at least the option field in the output, and possibly also the quoted value.

Generic TT files

The example service is small and simple enough to create and maintain using all keys and their expected output format.

However, the inserted CCM namespace allows a more generic approach, by simply looping over all possible key/value pairs via CCM.contents.pairs with the format based on the (base)type in the schema (e.g. whether the value is a list or a boolean).

example service with generic TT files

The main.tt can be refactored in 2 TT files:

main.tt with basic structure looping over each element

name = {
[%- FOREACH pair IN CCM.contents.pairs %]
[%     INCLUDE metaconfig/example/element.tt key=pair.key value=pair.value FILTER indent %]
[%- END %]
}

element.tt generating the format for each element based on the type of the value

[% key %] = [% # force space after = -%]
[%- IF CCM.is_list(value) -%]
[%-      value.join(',') -%]
[%- ELSIF value.is_boolean %]
[%-      value ? "TRUE" : "FALSE" -%]
[%- ELSE -%]
[%-      value.is_string ? '"' _ value _ '"' : value -%]
[%- END -%]

(_ is the TT string concatenation, taken from Perl6 syntax)

This mainly avoids duplication of the metadata held (and enforced) by the schema. E.g. one doesn’t need to repeat all possible booleans in the TT file. You can trust that the profile has the values (and metadata) to render the correct final configuration file via the schema definition and the binding.

But it is very important in this approach that you have a correct schema, that is as specific as possible because the basetype metadata will decide the format (rather then e.g. the name of the attribute).

In order to couple the validity of schema and TT files, you should use value based unittests to raise any unexpected changes to the schema and/or the TT files.

Caveats

There are a few attention points:

the pairs VMethod sorts alphabetically on the key, making the output reproducible. But this makes it harder to control the order of the output with the generic TT approach. Luckily, this is hardly ever an issue for real configuration files.
requires xml profiles or CCM typed JSON (i.e. json_typed=1 set in ccm.cfg). E.g. in case of JSON with json_typed disabled, long type is encoded as a string in the CCM configuration database, and port would become a quoted integer in the output.

Result filestructure

Generating all files as discussed above generates following file tree in the configuration-modules-core

ncm-metaconfig/src/main/metaconfig/example/main.tt
ncm-metaconfig/src/main/metaconfig/example/pan/config.pan
ncm-metaconfig/src/main/metaconfig/example/pan/schema.pan
ncm-metaconfig/src/main/metaconfig/example/tests/profiles/config.pan
ncm-metaconfig/src/main/metaconfig/example/tests/profiles/simple.pan
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/config/base
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/config/neg
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/config/not_so_simple
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/config/value
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/simple
ncm-metaconfig/src/test/perl/service-example.t

Also see the PR for this example service (and the generic example).

metaconfig, the TextRender meta-component

The metaconfig component

ncm-metaconfig

convert option

Development example

Prepare environment

Adding a new service

Target

Initial structure

Create the schema

Create config template for metaconfig component (optional)

Make TT file to match desired output

Add unittests

simple unittest

profile

regular expression

Location flags in the RegexpTest

verify

config based unittest

profile

regular expressions

verify

other possible tests

Generic TT files

example service with generic TT files

Caveats

Result filestructure

`convert` option