The metaconfig component
Large fraction of all system administration (and thus also the quattor
components) is
creating correct configuration files, and if required, restart/reload/… the accompanying
service in case of changes.
Generating structured text is the bread and butter of TextRender
, as described in the
1st part of this series, and the metaconfig
component
(together with Test::Quattor::TextRender::Metaconfig
unittest class) provides a
high-level interface to the creation and testing of typical configuration files.
This document contains a very brief introduction to ncm-metaconfig
and a more detailed
step-by-step tutorial how to add a new service
under metaconfig
control.
The usage of TextRender
in other components or tools as described in the 3rd part of
this series.
ncm-metaconfig
The metaconfig component is a component that has a number of service
s,
each service is the file to generate.
The metaconfig component contains a dict
called service
where the
keys of the dict
are the names of the files to be generated.
The pan configuration of such a service can be
include 'metaconfig/myown/schema';
bind "/software/components/metaconfig/services/{/etc/myown.config}/contents" = my_own_type;
prefix "/software/components/metaconfig/services/{/etc/myown.config}";
"module" = "myown/main";
"contents" = dict("some", "data");
"daemons" = dict("myownservice", "reload");
Here /etc/myown.config
is the metaconfig service and the file that will be generated by TextRender
.
The contents
is passed to CCM::TextRender
as a Element
instance, together with the module
.
The daemons
element is an optional dict of service(s) (of the sysvinit/systemd variety) and their actions.
The actions are handled by CAF::Service
and these are triggered when
the generated text is different from the current one.
Other optional attributes are convert
, mode
, owner
, group
, backup
and preamble
(a fixed header).
convert
option
As of 16.2, ncm-metaconfig
supports the convert
option.
This option provides access to the predefined conversions
from the CCM::TextRender
element
option
and can be used with any module to perform basic and/or common conversions
that would otherwise require a TT file (altough a trivial one).
It is important to keep in mind that the conversion is done on the element level
of the services contents
, and is applied to all relevant elements.
Example usage with builtin module tiny
:
prefix "/software/components/metaconfig/services/{/etc/sysconfig/example}";
"module" = "tiny";
"contents" = dict(
"ABC", "data with spaces",
"IS_EXAMPLE", true,
);
"convert/yesno" = true;
"convert/singlequote" = true;
will create file /etc/sysconfig/example
with content
ABC='data with spaces'
IS_EXAMPLE=yes
Development example
What follows is a step-by-step guide how to add a new service to ncm-metaconfig
,
which consists of creating
- a pan schema for the service
- the Template::Toolkit file(s) (TT for short) to generate the text (assuming the builtin
TextRender
modules are not sufficient) - unittests to verify the expected format
Prepare environment
Start with forking the upstream configuration-modules-core repository,
and clone your personal fork in your workspace
(replace stdweird
with your own github username in the example below).
Also add the upstream
repository (using https
protocol).
GHLOGIN=stdweird
git clone git@github.com:$GHLOGIN/configuration-modules-core.git
cd configuration-modules-core
git remote add upstream https://github.com/quattor/configuration-modules-core
Other requirements are:
- recent
CAF
andCCM
installed, and thePERL5LIB
environment variable set to the quattor install path so theCAF
andCCM
perl modules are found. (The installation should take care of the dependencies, including the installation of theTemplate::Toolkit
framework) - recent clone of the template library core repository
to be able to use already existing pan types and functions in the schema.
Set the environment variable
QUATTOR_TEST_TEMPLATE_LIBRARY_CORE
to this path (by default, the test framework looks in the parent path ofconfiguration-modules-core
).
You can verify your environment by running the metaconfig unittests. No tests should fail when the environment is setup properly.
cd ncm-metaconfig
mvn clean test
Adding a new service
Pick a good and relevant name for the service
(in this case we will add the imaginary example
service).
Set the variable service
in your shell (it is used in further command-line examples).
service=example
Target
Our imaginary example
service requires a text configuration file with path
/etc/example/exampled.conf
and has following structure
name = {
hosts = server1,server2
port = 800
master = FALSE
description = "My example"
}
where following fields are mandatory:
hosts
: a comma separated list of hostnamesport
: an integermaster
: boolean with possible valuesTRUE
orFALSE
description
: a quoted string
The service has also an optional fields option
, also a quoted string.
Upon changes of the config file, the exampled
service needs to be restarted.
This type of configuration is ideally suited for metaconfig and TT.
Initial structure
Make a new branch where you will work in and which you will use to create the github pull-request (PR) when finished
git checkout -b ${service}_service
Create the initial directory structure (from the ncm-metaconfig
path).
cd src/main/metaconfig
mkdir -p $service/tests/{profiles,regexps} $service/pan
Add some typical files (some of the files are not mandatory, but are simply best practice).
cd $service
echo -e "declaration template metaconfig/$service/schema;\n" > pan/schema.pan
echo -e "unique template metaconfig/$service/config;\n\ninclude 'metaconfig/$service/schema';" > pan/config.pan
echo -e "object template config;\n\ninclude 'metaconfig/$service/config';\n" > tests/profiles/config.pan
mkdir tests/regexps/config
echo -e 'Base test for config\n---\nmultiline\n---\n$wontmatch^\n' > tests/regexps/config/base
Commit this initial structure
git add ./
git commit -a -m "initial structure for service $service"
Create the schema
The schema needs to be created in the pan
subdirectory of the service directory src/main/metaconfig/$service
.
The file should be called schema.pan
and is a declaration template
.
declaration template metaconfig/example/schema;
include 'pan/types';
type example_service = {
'hosts' : type_hostname[]
'port' : type_port
'master' : boolean
'description' : string
'option' ? string
};
long
,boolean
andstring
are pan builtin types (see the panbook for more info)type_hostname
is a type that is available from the mainpan/types
template as part of the core template library.- the template namespace
metaconfig/example
does not match the location of the file, but this is intentional and is resolved by the unittests. During the tests, thencm-metaconfig/target/pan
directory will be created with correct sub-structure.
Create config template for metaconfig component (optional)
A reference config file can now also be created, with e.g. the type binding to the correct path and configuration of the
restart action and the TT module to load. The file config.pan
should be created in the same pan
directory as schema.pan
.
unique template metaconfig/example/config;
include 'metaconfig/example/schema';
bind "/software/components/metaconfig/services/{/etc/example/exampled.conf}/contents" = example_service;
prefix "/software/components/metaconfig/services/{/etc/example/exampled.conf}";
"daemons" = dict(
"exampled", "restart",
);
"module" = "example/main";
This will expect the TT module with relative filename example/main.tt
.
Make TT file to match desired output
Create the main.tt
file with content in the src/main/metaconfig/$service
directory
name = {
[% FILTER indent -%]
hosts = [% hosts.join(',') %]
port = [% port %]
master = [% master ? "TRUE" : "FALSE" %]
description = "[% description %]"
[% IF option.defined -%]
option = "[% option %]"
[% END -%]
[% END -%]
}
FILTER indent
creates the indentation- TT can easily introduce newline issues, so be careful if the config files are sensitive to this.
Add unittests
Each unittest consists of 3 parts:
-
an object template (in
src/main/metaconfig/<servicename>/tests/profile
) - the resulting profile has the required attributes (ideally including the schema and type bindings)
-
the test profiles (should try to) use the same path as actual
ncm-metaconfig
usage wrt to the profile structure -
one or more RegexpTests that contain regular expressions that will be tested against the output produced by the TT module and the profile.
- multiple files in directory
src/main/metaconfig/<servicename>/tests/regexps/<name_of_object_template>
-
a single file
src/main/metaconfig/<servicename>/tests/regexps/<name_of_object_template>
- a perl unittest to run all unittests for this service as part of the
metaconfig
unittests.
The object template is compiled in JSON format using the pan-compiler.
The testsuite takes care of the compilation, TT output generation, and running the tests.
Only test templates with .pan
suffix that are either unique
, structure
or object
templates are considered,
all other will get an (non-fatal) error message. Subdirectories will not be checked for object templates.
The object template and the (one or more) corresponding RegexpTest(s) together form a
single unittest for the service; all unittests for this single service
are ran via in a single perl src/test/perl/service-example.t
unittest, and should look like
use Test::More;
use Test::Quattor::TextRender::Metaconfig;
my $u = Test::Quattor::TextRender::Metaconfig->new(
service => 'example',
)->test();
done_testing;
(To be complete, the ncm-metaconfig
component has lots of perl service unittests,
together with the perl unittests for the component itself).
simple unittest
The easiest example is a single object template with a single regexp file.
profile
The default pan basepath for the TextRender
attributes like module
and contents
is /metaconfig
.
Create the profile tests/profiles/simple.pan
as follows:
object template simple;
"/metaconfig/module" = "example/main";
prefix "/metaconfig/contents";
"hosts" = list("server1", "server2");
"port" = 800;
"master" = false;
"description" = "My example";
- the schema is not validated in this
simple
template, but it can easily be done by adding
include 'metaconfig/example/schema';
bind "/metaconfig/example/contents" = example_service;
But the preferred way is to create a proper config.pan
file and use that
(see config example below).
regular expression
Make a 3 block text file tests/regexps/simple
, with ---
as block separator as follows
Simple test
---
unordered
nomultiline
---
name
hosts
port
master
description
This will search the output for the words name
, hosts
, port
, master
and description
.
This is good for illustrating the principle, but is a lousy unittest. Check the config
unittest below for proper testing.
The filename simple
has to match the object template you want to test with (in this case the simple.pan
template).
Location flags in the RegexpTest
The required attributes for ncm-metaconfig
(e.g. module and contents) are retrieved from the location default /metaconfig
(e.g. the contents will $cfg->getElement('/metaconfig/contents')->getTree()
.
To select another path, 2 additional location flags are supproted:
- an absolute path starting with a single
/
is interpreted as a metaconfig service (/etc/config
will result in contents from$cfg->getElement('/software/componentes/metaconfig/service/{/etc/config}/contents')->getTree();
- an absolute path starting with 2
/
s is interpreted as an absolute panpath (e.g.//some/path
will look for contents from$cfg->getElement('/some/path/contents')->getTree()
.
verify
You can verify this single unittest for the example
service using
QUATTOR_TEST_SUITE_FILTER=simple mvn clean test -Dunittest=service-example.t
The QUATTOR_TEST_SUITE_FILTER
environment variable is a regular expression pattern that will
filter the tests to run (matching tests are run).
(Run this from the configuration-modules-core/ncm-metaconfig
directory)
config based unittest
It is better to use a full blown template as will be used in the actual profiles. The added
advantage here is the config.pan
and schema.pan
from the pan
directory are tested as well.
profile
The profile tests/profiles/config.pan
is similar to the simple one
(after all, we want to set the same values),
but by targetting metaconfig
usage, a different prefix is required.
object template config;
include 'metaconfig/example/config';
prefix "/software/components/metaconfig/services/{/etc/example/exampled.conf}/contents";
"hosts" = list("server1", "server2");
"port" = 800;
"master" = false;
"description" = "My example";
The type binding and definition of the TT module are part of the pan/config.pan
template, and this usage is very
close to actual usage.
regular expressions
We will now make several regexptests, each in their own file and
grouped in a directory called config
(matching the object profile name).
The filenames in the directory are not relevant (but no addiditional directory structure is allowed).
We need to set the location flag to point to the test infrastructure which metaconfig-controlled file this is supposed to test.
In principle only one of the regexp tests should set this flag (and if multiple ones are set, they all have to be equal). You cannot test different metaconfig file paths from the same profile.
Lets start with a regexptest identical to the simple
test above, tests/regexps/config/base
:
Simple base test
---
/etc/example/exampled.conf
unordered
nomultiline
---
name
hosts
port
master
description
A 2nd improved regexptest tests/regexps/config/not_so_simple
uses the default flags multiline
and ordered
,
where the regular expressions are all interpreted as multiline regular expressions.
Basic multiline test
---
/etc/example/exampled.conf
---
^name
^\s{4}hosts
^\s{4}port
^\s{4}master
^\s{4}description
= ### COUNT 5
This test also uses the directive ### COUNT X
(with leading space; X is number, can be 0 or more), where this regular
expression is expected to occur exactly X times (in this case, we expect 5 =
characters).
The COUNT
directive ignores the ordering; itsimply is the total number of matches.
A 3rd regexptest tests/regexps/config/neg
checks if certain regular expression do not match using the negate
flag.
Basic negate test
---
/etc/example/exampled.conf
negate
---
^hosts
^port
^master
^description
This tests that the expected fields can’t start at the beginning of the line,
whitespace must be inserted before.
(The FILTER indent
TT inserts 4 spaces, as tested with the \s{4}
in the multiline regexp above.)
If one only needs to check that a single regular expression does not occur, one can also use ### COUNT X
, without
having to make a separate regexp test with the negate flag.
(Setting the negate
flag silently ignores the order flag).
A 4th regexp test tests/regexps/config/value
uses full value checks, which is interesting to have, but harder to maintain and review.
Basic value test
---
/etc/example/exampled.conf
---
^name\s=\s\{
^\s{4}hosts\s=\sserver1,server2$
^\s{4}port\s=\s800$
^\s{4}master\s=\sFALSE$
^\s{4}description\s=\s"My example"$
^}$
verify
You can verify this single unittest for the example
service using
QUATTOR_TEST_SUITE_FILTER=config mvn clean test -Dunittest=service-example.t
(this will run all 4 regexptest files)
other possible tests
- a test for the optional
option
field: a new test profile is required that has the optional field configured, and it also requires one or more regexp tests to verify at least theoption
field in the output, and possibly also the quoted value.
Generic TT files
The example
service is small and simple enough to create and maintain
using all keys and their expected output format.
However, the inserted CCM
namespace allows a more generic approach,
by simply looping over
all possible key/value pairs via CCM.contents.pairs
with the format based
on the (base)type in the schema (e.g. whether
the value is a list or a boolean).
example service with generic TT files
The main.tt
can be refactored in 2 TT files:
main.tt
with basic structure looping over each element
name = {
[%- FOREACH pair IN CCM.contents.pairs %]
[% INCLUDE metaconfig/example/element.tt key=pair.key value=pair.value FILTER indent %]
[%- END %]
}
element.tt
generating the format for each element based on the type of the value
[% key %] = [% # force space after = -%]
[%- IF CCM.is_list(value) -%]
[%- value.join(',') -%]
[%- ELSIF value.is_boolean %]
[%- value ? "TRUE" : "FALSE" -%]
[%- ELSE -%]
[%- value.is_string ? '"' _ value _ '"' : value -%]
[%- END -%]
(_
is the TT string concatenation, taken from Perl6 syntax)
This mainly avoids duplication of the metadata held (and enforced) by the schema.
E.g. one doesn’t need to repeat all possible booleans in the TT file.
You can trust that the profile has the values (and metadata) to render
the correct final configuration file via the schema definition and the bind
ing.
But it is very important in this approach that you have a correct schema, that is as specific as possible because the basetype metadata will decide the format (rather then e.g. the name of the attribute).
In order to couple the validity of schema and TT files, you should use value based unittests to raise any unexpected changes to the schema and/or the TT files.
Caveats
There are a few attention points:
-
the
pairs
VMethod sorts alphabetically on the key, making the output reproducible. But this makes it harder to control the order of the output with the generic TT approach. Luckily, this is hardly ever an issue for real configuration files. -
requires
xml
profiles or CCM typedJSON
(i.e.json_typed=1
set inccm.cfg
). E.g. in case of JSON withjson_typed
disabled,long
type is encoded as a string in the CCM configuration database, andport
would become a quoted integer in the output.
Result filestructure
Generating all files as discussed above generates following file tree in the configuration-modules-core
ncm-metaconfig/src/main/metaconfig/example/main.tt
ncm-metaconfig/src/main/metaconfig/example/pan/config.pan
ncm-metaconfig/src/main/metaconfig/example/pan/schema.pan
ncm-metaconfig/src/main/metaconfig/example/tests/profiles/config.pan
ncm-metaconfig/src/main/metaconfig/example/tests/profiles/simple.pan
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/config/base
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/config/neg
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/config/not_so_simple
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/config/value
ncm-metaconfig/src/main/metaconfig/example/tests/regexps/simple
ncm-metaconfig/src/test/perl/service-example.t
Also see the PR for this example service (and the generic example).