2nd Quattor Worshop - DESY - 18-20/10/06

Agenda

Site Reports

CERN

Mainly increase in the number of nodes managed by Quattor : #4000

Quattor Solaris support dropped

CEN specific activities :

CDB profile authentication/encryption : not yet deployed, need to solve the issue of a host cert expiring or in a CRL to avoid dead lock. Idea is to use 2 URLs for downloading the config, with an automatic failover it first one fails.
Manage Xen and VirtualPC with Quattor
Namespaces : urgently need, want to agree on namespace layout before. Will take time to implement in 20K templates
test, new and pro area to be provided using loadpaths and ACLs
Integration of Quattor with SL (Service Level Status)
Impact of SL5 (using FC5/6) on ncm-components
SINDES maintenance : not really part of Quattor but mainly used in this context.

DESY

200+ systems in production in the GRID infrastructure

Local HERA experiments
CMS and Atlas (DESY T2)
VOs that are part of EGGE

Still using Quattor 1.1

CDB, AII, SWREP, SPMA
Still interest for SCDB but no time yet…
Middleware installation is done using YAIM (yaimexec ?)

Current issues :

How to keep CA and middleware RPMs up to date ? How to use YAIM and SPMA in conjunction ?
PAN compile time increasing with the number of machines (currently 5-6 minutes for 200 nodes)
Time to get fixes for YAIM bugs reported in Savanah
Infrequent cdispd aborts : seems to be known at CERN, related to CCM

BEGrid

Central SCDB + SWrep

SCDB : Certs+ACLs. Not everybody allowed to edit everything.
Goal :

Current configuration :

4 sites with LCG, 2 with gLite
Restarted from scratch (new repository) with gLite

Additions to QWG templates :

SE_dCache
Lemon server with Oracle
Use of IPMI for /system/hardware
Ganglia server and client
All pwd and sensitive information into one template

Problems using SLC with Quattor/QWG

Tests done with WNs in VMware under WXP, managed by Quattor.

Changes to AII :

SINDES/AII integration : some templates needed to be changed
Install kernel in ks rather than ks-post-install : easier to use alternative kernels

Work on improving bulk compile of a large number of worker nodes

Compile a dummy WN
In real WNs, include the compiled profile and redo part of the configuration
Doesn’t fit well with QWG templates, difficult to say wich part to reinclude and which part to ignore, probably need something inside PANC

CNAF

Early adopter of Quattor : both experienced QWG templates and YAIM

Currently using Quattor for initial installation (AII)

Use http repositories instead of SWrep
Since LCG 2.7, moved to YAIM with ncm-yaim

Pretty well accepted by farm managers

Especially storage guys

Started to implement our own namespaces

Mainly to achieve machine category segmentation, in particular to control access to templates.
Would like some discussion on use of namespaces in standard templates

Would be useful to have all templates needed to install a basic SL system.

More of documentation on basic Quattor components (PAN, LC libs…) would help dissemination.

NIKHEF

T1 for LHC and involved in several national projects (BIG GRID, VL-e

2 “sites” :

Production : ~180 nodes. Significant increase expected.
Installation testbed : ~15 nodes

Quattor usage : CVS + ant/panc

OS : CentOS 3
panc 5.0.6
Only generic components used

gLite : moved to ncm-yaim

Initial installation via Quattor

Issues :

64-bit SW installation
Compiler performance

Philips Eindhoven

Part of research division uses grid

A few test systems installed with Quattor
Links with NIKHEF

PIC

250 nodes running SL3 + 2 CASTOR nodes

Use ncm-yaim.

Problem with the hardcoded list of variables that can be configured with ncm-yaim. Made some changes, not filled into Savanah. Pb fixed 6 months ago by Ronald.

Deploying Quattor 1.2 + SCDB

Irish Grid

Entire irish grid managed with Quattor

18 sites, 200 nodes : all the nodes centrally managed from 1 site (Dublin)
1 Quattor database
CVS SCDB, HTTP RPMs
95% of nodes are Xen VMs
Had our own hierchical model : EGEE->GI->Site. Started to move to standard QWG layout/hierarchy soon with gLite 3.0.2.
Expertise is spreading throughout the group

Moving non Grid servers to Quattor : 64-bit, SL4, Xen…

Integration with automatic VM creation tool for building testbeds.

Have spent a lot of time keeping up to date with changes in QWG structure.

UAM

Quattor used for installation + configuration of 3 clusters, using 3 Quattor servers.

Use CDB with last QWG templates

Cluster UAM-LCG2

Part of a distributed T2
130 WNs
QWG LCG templates
Issue with update snc with other sites

Cluster GVMUAM-LCG2

500 nodes, mainly PCs for student lectures
Used for different topic
Installed with QWG templates
Must preserve existing partitions
No full control of DHCP
Network pretty slow

Cluster WS

User desktops : #40
Template layout based on organization : departement, group…
Home made templates
Software for desktops
Components to configure desktop services like printers, X11

Experience with ncm-yaim - R. Starink

QWG templates until 2.6.0 : several difficulties, mainly due to the lack of genericity

Took 4-8 weeks to incorporate local changes to a new QWG release
Backward compatibility between release
Complex structure

Move to YAIM with ncm-yaim with LCG 2.7

YAIM used only for config, install with SPMA
YAIM variables created from templates
Activate YAIM on each machine

Setting YAIM variables into templates very similar to writing a pro_lcg2_config_site.tpl

Issue with new version of YAIM requiring variables not supported by ncm-yaim
Explicit list of supported variables comes from the ncm-yaim schema that bring the advantage of validation (instead of using a plain filecopy).

Building RPMs list. Several solutions attempted :

Dependencies from gLite meta packages
From an APT repository

Need for YAIM local functions required :

Security : no shared pool accounts for SGM users
Central db for DPM and LFC
Shared gridmapdir, shared home dirs, accounts on LDAP : not supported by YAIM
Developped nikhef-yaim-local to be installed (via SPMA) or updated after gLite-yaim. Close contacts with YAIM developpers.

Satisfied by the change :

Easier to maintain, less overhead, less dependent on external party (QWG)
Still some surprise with YAIM…
Shorter time for deployment of a new release (1 week)
No experience in changing a node type without reinstalling

RPM Dependency Hell - Stijn Weirdt

RPM management by Quattor involves several steps :

Create/update repository with tools in cvs/utils
Create base repository contents with cvs/utils
Keep repository up to date
Test deployment : nothing to help here

Other tools existing :

apt : no bi-arch support
yum : the best presently, not supporting all RPM options
smart : try to support everything, still buggy
All these tools share in common the idea of RPMs metadata stored in some db that can be accessed without accessing RPMs.

SPMA : works well but some limitation : in particular inability to test deployement without installing RPMs as there is no metadata.

RPM repositories :

SWrep : should have a “rsync url” option to keep local repository up to date.
OS distros : could use existing mirrors to avoid duplication locally. Would require some kind of metadata to produce the local templates required.
Initial loading of OS templates : may rely on comps.xml to find what is needed or on some metadata.
Keeping repository up to date : OS update metadata parsing ?

RPM testing before deployment : test all the dependency from the administrator machine before deployment

Should be very fast (30-60s per machine) : require metadata
Fake installation not fast enough
Injection of all RPMs in a rpmdb + rpm -i
Problems with first tests on 64-bit : yum fails for unknown reasons, rpmdb doesn’t support correctly bi-arch.

Specific Use Cases

Diskless Systems - M. Shroeder

2 possible setups :

RAM disk : the whole system in a large file loaded at boot time
NFS mount : a small image loaded from network at boot, other FS through NFS, mainly readonly (and shareable)

Read Hat’s way

PXE + NFS mount
Clone the server system
One snapshot for each client : non shared files, writable files

Quattor usage in this context :

Configure RH tools (pxeos, pseboot) via quattor templates and components : ncm-diskless_server
Kickstart for server install and cloning : only install base system
server and its clone configured separatly : chrooted for clients
Client configuration cloned in 2 parts : 1 common to all clients (done on the clone), 1 specific for each client (1 profile / client)

Clone is not really a real machine : cannot receive CDB modifications notification

Has to fetch new profiles via cron
ncm components run on the server but must not impact the server
Client filesystem is read-only but to run a component on a client, need to create some files and the ability to modify existing ones
Not clear if we want to support several clones (several configurations) per server

Current experience : 2 test clusters (2 and 8 clients)

Clients in a private network without access to CDB/SWrep
SPMA cannot be run on the client : establishing a matrix of components that can run on the clients

Quattor and XEN - S. Child

Main problem is the grub component :

Need support for multiple boot
XEN is the kernel, Linux kernel and initrd are “modules”
New version with this support now checked in but problem found at CERN ?

Started ncm-xen :

Write configuration files for individual VMs
Should also write base Xen configuration
Will set up links for automatic start of domains
Will check in 0.1 soon… Still not mature !

GridBuilder : web based interface (Ajax based) for creating and managing VMs

http://gridbuilder.sourceforge.net, developper by Dublin Trinity College (Author : S. Childs)
LVM allows fast creation of COW FS images
Database of VMs and images
Quattor used for configuration. Still small amount of pre-configuration :
Configure network on filesystem images
Fetch Quattor profile

Possible improvements in Quattor integration

Automatic generation of node profiles from user supplied description in GridBuilder
Support for Colinux, a version cooperation with Windows for sharing of HW ressources (memory…)
Condor pool

PAN Compiler Update

C version : implementation frozen, only major bugfixes, v6.0.3

Performance improvements in last version : compression removed, defaults processing improved, speed and memory consumption as good or better than before
Added “session” directory to improve interface for CDB

Java version : still in development, all major parts functionning

Limited alpha available, first beta mid December, Production January
Main part missing : built-in functions
Validation suite is complete
License : probably Apache2 to be consistent with EGEE-II
Source in QWG SVN repository
Backward-compatibility : as much as possible, may be some incompatibilities for some very unused features
Require Java 1.5+
Compilation and packaging : ant
Parser (build) : JavaCC 4.0
Unit testing : JUnit 4.1
Base64 encoding/decoding (build & run) : classes available from Apache/W3C, probably incorporate them directly in the code base.

Syntax changes :

Bit operators
Unary plus for symmetry with unary -
Octal, hex accepted everywhere : ranges, path…
Limits allowed on record statements
‘bind’ statement added for binding a path to a type, in replacement of one form of ‘type’ (this ‘type’ usage will be deprecated)
‘return’ allowed where functions are (not very useful, grammar simplification)
Warning could be issued for deprecated usage

Other changes :

Stricter syntax checking at compile time
Generation of “object” files (binary form of a syntax checked template) to avoid recompilation of an unchanged template
ant tasks will be the primary interface to the compiler (no “binary”) but wrapper scripts will be provided for command line

Incompatibilities :

‘bind’ is now a keyword
OBJECT, SELF, ARGV ARGC defined to conform to best practices for global varaible and avoid conflict (at grammar level) between object keyword and object variable
No pointers to properties : now ‘x = y[0] = 0; y[0] = 1’ now leaves x==0 (currently x is also set to 1)

Emphasis for first release : verifying functionality and measuring performances

In particular evaluate cost/benefit for object files

Future changes after initial release :

Parallelization : compilation of templates, building of configuration trees
Remove deprecated features : lowercase global variable, deprecated form of ‘type’, ‘define’ keyword
Addition of string functions : uppercasing, lowercasing, push, pop
More default types : XMLSchema, port…

Missing features/pending bug reports :

Unescape strings in traceback produced in error msg
Add a file existence test operator
Add an argument to matches() to allow to pass global options (like in Perl)

QWG Templates

From discussion : need to think about explicit support for CDB

Probably mainly the matter of defining load paths in an optional template in replacement for cluster.build.properties

CDB/SCDB Update

CDB Update

New features since last workshop :

Namespaces
X509 and Krb authentication
ACLs with namespace support
Client/server improvement through session metadata

State management through metadata :

Problem was session directories used for both data and state
Clear separation with specific metadata for state control : better and earlier detection of commits

Parallel compilation of templates :

All profiles compiled with one command doesn’t scale : too long, to much memory
Whole set of templates divided into several subsets (without dependencies) compiled separatly and in parallel on several processor/machines

Smarter rollback and commit :

Currently rolling requires rollback all the mods in the sesion, commit can screw up previous modifications committed but not in the session directory
Now allow selective rollback and interactive commit

Handling of dead revisions : problem related to CVS backend

A removed template is no longer in CVS, restoring from backup requires a lot of manual cleanup. Look at SVN as a new backend ?

New authentication for CDB moved to a separate library and now used in all components (SWrep in particular)

Doesn’t require to install CDB to use other components, only the library

Other project status :

CDB as a Web service : still any interest ? Not sure…
Fine-grained CDB locking with faire queuing : really required
Concurrent compilation of (non object) templates : too complex, wait for new compiler..
mod_perl : no further investigation, mod_fastcgi probably a better solution.

Open issues :

CVS desn’t scale : a problem with 24k templates. Possible solutions

perl based CVS ? Subversion ? XML database ?
Relocatability : difficult to port to other systems, testing requires a full installation or specific privileges

SCDB Update

See slides.

Suggestion :

To avoid a full rebuild after repository templates update, ignore these templates when evaluation if a node profile must be recompiled (this is how it is handled within CDB).

Quattor Core Modules Update

Several ongoing developpement at CERN, of general interest.

CDB2SQL :

Rewrite with multithreaded Python and a fast XML parsing library. Sould have no CERN dependency, only Oracle dependency (but should not be difficult to add support for other RBMS)

autoconf :

In contact with ETICS to use their framework for configuration and automatic build
Will remain possible to run Quattor build tools outside of ETICS (mainly need to define –LOCALDIR)

CCM :

Add support for a failover profile, in case the URL in –profile is not available
Problems observed causing ncm-cdispd to crash

wassh2 : improvement of wassh (parallel ssh), interfacing with CDB

CDB plugin done using a plugin : CERN uses Oracle/cdb2sql
Will be part of Quattor 1.3
Non CERN testers welcome

Notification systems : have the ability to trig execution of a NCM component on one or several nodes without logging to them (even with wassh)

Basically one command : notify_host myhost component
‘component’ is a keyword translated on the target host via a configuration file
Current version with lots of CERN dependencies, plan to reengineer it and release it as part of Quattor

AII

Since last workshop, various bug fixes and some enhancements :

Support for rescue images
Support for alternative device naming schemes

Work in progress :

Separate site configuration from component configuration
Error handling
Complete partitioning scheme
Documentation
Schema change for block devices

On the todo list :

SPMA proxy start
Support for SINDES

Separating site and component configuration :

Idea if to have pro_software/declaration_component_aii really only related to AII component. Will provide a function aiiconfig(name,disk) to return actual configuration
pro_config_aii_OSNAME : OS / arch specific configuration
pro_aii_config_site : site specific information

Expected incompatibilities in new version :

Separation of configuration
Change in schema for generic block devices

Remark : could add a property to /software/components/osinstall/options to select if initial installation is done with DHCP or final address and improve the template accordingly

== New Schema for Block Devices ===

Need to add support for SW RAID, new kinds of block devices, filesystem mount options.

Also need to align naming for HW Raid and SW Raid

New schema proposal : /system/blockdevices/[disk

lvm

hwraid]

disk : 1 entry per disk, almost all information optional, mainly partitions
md/hwraid : basically the same, add information about raid members, raid level, stripe size…
lvm : allow to use more sophisticated LVM scheme that current functions (LVM vg splitted over several HW raid…)

Seems ok but need to check :

AII compatibility : in particular KS template
Ability to represent multi-pathed devices
Is the md/hwraid distinction relevant : may be just keep a property in a common schema

Namespaces

See German presentation. Basically everybody agrees. Just a few details :

What’s in pan/ and what’s in quattor/
Have all type defined in one place as some will move to the compiler
Hardware : may be cards is not needed, for ram/ use bank.tpl
Components : upgrade quattor build tools to be able to automatically produce namespaced version from non namespaced source, call declaration.tpl schema.tpl and define default values there in the future, add the ability to build RPM-less components (automatically insert perl script into template).

Clusters :

CDB relies on clusters and subclusters
Could be added in SCDB
Rename site/ to config/
Difficult to agree on the whole layout between CDB/SCDB as the concepts are different

Sites :

Rename site/ to config/
Difficult to agree on the whole layout between CDB/SCDB as the concepts are different

OS templates :

Change rpmlist to rpms
Rename templates describing groups to groupname.tpl
Renamme os/ namespace to config/

Standard variables : no real need to agree on variables, need to agree only on the schema

OS/arch naming : originally os_arch, QWG use os-arch, no real need to agree as this is not in the schema

Wish List, Roadmap…

Documentation :

Provide user guides in addition to specification for all components
Move quattor.org to Twiki (except home page ?)
Have a short installation guide (not 80 pages…)
List of available components, with a very short explanation and the recommended production version (may come from CVS tags, updated manually if necessary)
Tutorial : differenciate between old and recent tutorials

Open Issues (from Savanah) :

Conclusions

Next meeting : Trinity College, target date : mid-march

TBD :

Add an option to osinstall for using DHCP at installation time and merge LAL and standard KS template

2nd Quattor Workshop Summary (October 2006)