Note: SCDB, the second generation of Quattor databases, is now deprecated in favor of [Aquilon][aquilon_intro] which provides more flexible workflows and an improved scalability.. As SCDB is still used as some sites, the information here is kept for reference but if you are a new Quattor user, condider using Aquilon rather than SCDB.
SCDB Architecture
SCDB configuration database is based on Subversion (SVN): everything is store in a Subversion repository. Any SVN client can be used to checkout/update/commit changes from/to the configuration database and all the SVN features can be used to manage the history or to roll back changes.
Profile compilation and deployment is done with ant, a
tool available on any platform (Unix, OSX, Windows). With SCDB, all the work is done
on the administrator machine. Except for the deployment, no network connection is required.
In SCDB, deployment of a new configuration is a 3-phase process :
- Template modification with your preferred editor and compile with
ant. This phase requires no network connection. - Once compilation is successful, it can be committed with your preferred SVN
client. And you can synchronize changes that were done by other administrators
with
svn update. - After committing the changes, the new configuration can be deployed with
antdeploy. Deploy phase starts by recompiling the configuration locally: this is almost immediate if no change happened since first step. It creates a SVN tag for the deployed configuration for an easier roll back.
ant enforces the following prerequisites for a successful deployment:
- A successful local compilation of the new configuration
- All the local changes must have been committed to the repository
- All the changes made to the repository by others must have been imported in the
local copy (
svn update).
Template Library
In SCDB all the templates are stored in the cfg/ directory of the configuration database/repository.
Under this directory, a site typically import the template library. In addition,
two site specific directories are available: clusters and site.
-
clusters: this directory contains mainly host profiles organized by clusters. A cluster is an arbitrary group of hosts that share some common parameters, typically defined in thesitedirectory of the cluster. There is one directory per cluster under theclustersdirectory. Each cluster has one filecluster.build.propertiesin its top directory that defines the other template parts that are used in the cluster (every directory specified is relative tocfgdirectory) withcluster.pan.includesproperty. See SCDB examples on GitHub for more details. -
site: a set of configuration templates and parameters shared by several clusters. There is typically one directory per site under thesitedirectory. A configuration database can contain as many site as needed. The clustercluster.build.propertiesfile defines which sites are used by a cluster
Cluster groups
Clusters in the clusters directory can be organized in groups. To enable the cluster
group features, is is necessary to define the property clusters.groups.enable
in the file quattor.build.properties (in the top directory of the configuration database):
clusters.groups.enable=true
Using SCDB
Configuring SCDB Ant Tools
The main configuration file for SCDB and its Ant Tools is quattor.build.properties in
the top directory of the configuration database. This file allows to customize default
values defined in quattor.build.xml, the ant configuration file used by Quattor.
Lines in this file must have the format
key=value, with # as the comment delimiter. Not that everything
after = is considered part of the value, including leading spaces,
quotes and doubles quotes (quotes are not interpreted as a string
delimiter).
The configuration options (called properties) available are documented
at the beginning of quattor.build.xml. They allow to customize
the layout used to organize and locate templates. It is recommended to
avoid unnecessary changes as it may become more difficult to
troubleshoot problems. The most common options used are:
# Define to true if you want to use cluster groups
clusters.groups.enable=true
# Define profile format to be json (recommended)
pan.formats=json,dep
# Disable pan deprecation warnings (not recommended in production)
pan.warnings=off
Displaying Resulting Changes in Profile
When doing changes in configuration, this is often useful to check the resulting profiles before deploying. There is a small tool to help doing that, [source:SCDB/trunk/utils/profiles/compare_xml utils/profiles/compare_xml]. This tool is part of [source:SCDB/trunk SCDB base].
To use this tool, before compiling your changes, you need to copy your
current (or reference) build directory to something else, by example
build.saved (default name used by the script). After the compilation,
if you run the script without arguments, it will display the list of all
profiles that have changed. You can inspect the changes (require a bit
of knowledge of profile contents) using option -v. Or you can restrict
the check to one specific profile with option -p. By example :
# utils/profiles/compare_xml
Comparing XML files (*.xml)...
Differences found in build/xml/profile_grid05.xml
Differences found in build/xml/profile_grid06.xml
Differences found in build/xml/profile_grid07.xml
# utils/profiles/compare_xml -v -p profile_grid05
Comparing XML files (profile_grid05.xml)...
Differences found in build/xml/profile_grid05.xml
--- build.saved/xml/profile_grid05.xml 2006-07-28 11:41:45.000000000 +0200
+++ build/xml/profile_grid05.xml 2006-07-28 11:50:58.000000000 +0200
@@ -960,7 +960,6 @@
<options>
<dpm>
<db>
- <adminpwd type="string">MyPrefPwd</adminpwd>
<adminuser type="string">root</adminuser>
<configfile type="string">/opt/lcg/etc/DPMCONFIG</configfile>
<password type="string">DpmMgrPwd</password>
Another option available, -d, extends the comparaison to profile
dependencies. This is generally not useful but can be used to do a
regression test with a new version of the compiler, for example.
Using SCDB with an IDE
An IDE can be used to manage SCDB contents and trigger deployments as long as the IDE
has an integrated support for ant and for executing Java applications.
If this is the case, just add the build.xml file
to your IDE Ant interface. Eclipse is such an IDE that has been used successfully by
several sites.
If you are interested by this integration, contact the Quattor community.
Frequently Asked Questions about SCDB
ant usage
java.lang.NoClassDefFoundError: org/apache/tools/ant/launch/Launcher (11/2/07)
If you get this error when trying to use ant, that probably means you
have a version conflict between ant version distributed as part of SCDB
(in external/) and another version installed on the same machine. This
happens if you have a file /etc/ant.conf created by the non SCDB
version.
To clear the problem, remove /etc/ant.conf or upgrade the non SCDB
ant to the same version as SCDB.
java.lang.NullPointerException during the deploy phase (11/2/07)
One possible reason for this error is an incompatibility between the
version of JavaSVN used by ant deploy and the version of the SVN
client you use to commit your changes. This happens is one of them is
older than SVN 1.4 and the other is greater or equal to 1.4, because of
a metadata format change in workspace introduced in SVN 1.4. As a result
a workspace cannot be read and modified by both pre 1.4 and 1.4 or later
SVN clients (this is clearly stated in SVN 1.4 release notes).
Deploy failure
Failed to switch SVN cache to new tag: url1 is not the same repository as url2
This error means you have an existing working copy of your Quattor
repository on the Quattor deployment server that was checked out from a
repository different from the one defined in /etc/quattor-deploy.conf
(option repository_url in [scdb] section). A common mistake is to
use http in one case and https in the other one which leads SVN to
consider they are different repositories.
Depending on the URL you really want to use, either fix the configuration file or remove the existing working copy. If removing the working copy, be sure to remove the directory, not only its contents (as you may leave some hidden directories).
The location of the working copy is defined in
/etc/quattor-deploy.conf (svn_cache option in [build-tag]
section). If not defined, the default location is svncache directory
in the parent directory of build-tag.py location
(/root/quattor/svncache by default).
Unable to get information on tags branch
If you get this error at the beginning of the deploy phase when
executing ant deploy and you have no compatibility problem between the
SVN client version you use and the one embedded in ant tools(
external/svnkit), this means you don’t have valid cached credentials
to access the tags branch in your configuration database that can be
used by svnkit, the SVN client used by ant. To fix the problem, try
to write to tags branch using jsvn utility provided in
external/svnkit. You can use the following commands as an example,
changing the URLs to match your configuration database:
external/svnkit/jsvn cp htpp://quattorsrv.example.org/quattor/trunk htpp://quattorsrv.example.org/quattor/tags/test -m 'Deploy test'
When executing this command, you should be asked to enter your
credentials (userid and password). Note that depending on your exact
actual configuration, you may have to define environment variable
JAVA_HOME to reflect the directory containing bin/java.
After doing the previous step, check it works by trying to remove the
previously created branch. This should work without jsvn asking you
your credentials.
external/svnkit/jsvn rm htpp://quattorsrv.example.org/quattor/tags/test -m 'Remove test'
AII Issues
AII is the Quattor subsystem handling initial installation of machines. It has nothing specific to either SCDB or QWG templates, even though it is well integrated with both of them (see [/wiki/Doc/OS/Installation OS Templates] documentation about how to configure it). Entries in this fact are only about interaction problems between SCDB and AII.
aii-shellfe: No node matches xxx
When using aii-shellfe to configure a machine for initial
installation, you may get an error message similar to the following one:
bash> aii-shellfe --configure mynode.example.org
[ERROR] No node matches mynode(.example.org)?
This usually means either that you forgot to deploy your configuration before
running the command or that you made a spelling error with the machine name.
You can check if a .xml or .json file for the machine name you entered exists
in the directory referred to by variable osinstalldir in
/etc/aii/aii-shellfe.conf on your deployment server.
Under very rare circumstances, it may happen the .xml or .json file exists but
you still get the error. In this particular case, try to remove the file
profiles-info.xml in the same directory and redeploy your
configuration with the command:
external/ant/bin/ant deploy