Quattor Workshop - CNAF (Bologna) - 30/9/2013-2/10/2013

Agenda

Project Status - L. Munoz

Survey ran before the workshop

• Significant improvement compared to previous ones
• QWG : many were not able to move to QWG or forked a long time ago. We need to think more heavily at it.
• Aquilon adoption making progress (2-3 sites)
• Community support is perceived as good
• Documentation: still the weak point, were not able to make significant progresses
• LAL wiki: certificate problem still perceived as a barrier, a developer only resource

Core modules

Configuration Modules - L. Munoz

Releases

• Count : nine releases, 3 of them tagged as stable
• New releases are not mandatory
• Need a releaser manager backup for Luis

Config modules: main activity is on the core components

• Not much on grid components or on daemons
• ncm-openldap and ncm-cups were resurrected

New features

• Preliminary support for SSSD in authconfig
• YUM-based SPMA ready for production
• Hooks for ncm-ncd before and after execution of components

Metaconfig stuff

• Ugent has schemas and templates for many services: would like to share them
• But we can hardly promise backward compatibility

Work in progress

• Policy-based component dispatching in ncm-ncd
• Solaris port
• ncm-network: still not much progress

ncm-ncd hooks and policy-based component scheduling

See [/wiki/Development/PrePostActionsNCMNCD proposal].

Goal: execute some arbitrary actions before running all components and after.

• These actions will have no access to node profile
• Specified by run-init and run-finalizein /etc/ncm-ncd.conf
• An error during run-init will prevent executing components but run-finalize will run

Hooks may be used to implement site-specific policy for component scheduling, allowing to filter out some components from the initial list based on some local criteria

• But what to do with dependencies if one is filtered out? Can be very complicated
• Probably better/enough to implement a “everything or nothing” strategy: at least use case can be cleanly defined

Some use cases for executing only part of the configuration changes for a component. Proposal:

• Split component configuration into 2 different parts/subtree, e.g. /software/components/filecopy and /software/components/filecopy-magic
• Bind /software/components/filecopy-magic to type_filecopy
• Define a component alias name in filecopy-magic saying it will use component filecopy to execute the action
• New field in deps structure
• Filter out filecopy-magic in run-init hook
• See https://github.com/quattor/configuration-modules-core/issues/112 for discussion/implementation follow-up

CDBSQL - Z. Sebestyen

Used at MS to produce/update a SQLite db every 2 hours with the AQDB contents: used by users to query configuration rather than overloading the broker.

• Reused an older API from MS to query the DB

Code will be pushed to SF at some point

• Still some concerns with pushing directly to GitHub

Still need to review if it could be integrated with QuatView to replace the internal SQL generator in QuatView and deliver a generic web front-end.

Also, may look at using a NoSQL db rather than SQLite.

• May give some perf advantage for querying the db but may a price for the update

Maker Quattor Easier

Introduction - D. Zilaskos

Painful path for a new Quattor user

• Go to website and click download: not exactly what was being looked for…
• Try documentation: suggest to go to overview… just marketing blurb…

Should not dream of producing a polished documentation as we failed for so long… but we can make things much easier if not very easy. Goal should be to install a workable Quattor installation for testing in 30 mn!

• Install from Quattor YUM repo
• yum install quattor-server
• yum install quattor-client?
• Sample templates usable with minimum editing (host name, mac address) to install/bootstrap a first machine
• For the first step, could be take control of already existing machines without doing the initial installation to avoid AII/kickstart difficulties
• Panc book: improve the tutorial part

Documentation may be organized in 2 different parts:

• Getting started: build/describe an optimized procedure for a user to start with Quattor both for evaluating AND setting up a production infrastructure (could then be used to produce an appliance)
• Prerequisite hardware/site config
• Install server: prefer Aquilon if mature enough. We don’t want to attract new sites to SCDB.
• Take control of a client (already installed): should include a basic configuration description that is as less intrusive as possible for the existing configuration
• Mini-demonstration illustrating how to change some service configuration, for example ssh keys, account creations, add a RPM
• Next steps
• How to find docs and examples
• Initial installation: AII/kickstart customization
• Security
• Customizing node configurations: common services and recipes
• Document how to execute an arbitrary command with existing config modules like filecopy or metaconfig

Put documentation on GitHub: start with a non public (review) section

• Review how to improve the documentation structure on gitHub: see documentation discussion later during the workshop

Documentation: activate robot indexing for Quattor wiki hosted at LAL

Timeline

• Getting Started done at next workshop (April 2014)
• Next steps after

Encourage adoption of new YUM-based ncm-spma: make it clear that YUM is used in a completely standard way and that people may/must refer to YUM docs for details on advance YUM management (like mirroring).

QWG Status - G. Philippon

Recent changes

• No more release for more than 1 year
• No update to branches: all changes in the trunk
• standard/ almost stable: most things are in fact imported from other places

Current trunk status

• Partly based on Quattor 13.1: most components not yet updated (validation required)
• Support SL5 and SL6

3 different subsets shipped together

• “Standard” templates
• Provide a core node type without any grid dependency
• Should be really standard: can it be the case for a Web server?
• Grid templates
• Templates generated from other Quattor components like AII or config modules

StratusLab templates delivered separately (git repo)

• Support both ncm-spma v2 and v3

Work in progress

• Remove templates related other Quattor components from standard/
• Create a new namespace quattor/ with one sub-namespace per Quattor version: easier to move from one version to another one
• YUM-based ncm-spma
• Move to GitHub: do we keep the history?

To be done but not yet handled by lack of time

• Namespace structure of OS templates
• Integration of Aquilon
• NFS configuration independent of grid templates

GitHub migration based on several repositories, no history kept

• templ-library-standard: standard/ without the stuff imported from other places and without monitoring
• monitoring not migrated
• templ-library-core: templates imported from other Quattor components (config modules, AII, schema, Pan types…). Generated by the release process ideally, else some SCDB scripts may help (e.g. updateComponents)
• master branch + 13.1 branch (ncm-spma v2)
• Add standard/pan and standard/quattor other than AII
• templ-library-grid: grid templates
• 1 branch per major grid MW version
• templ-library-stratuslab: current strastuslab repository
• templ-library-examples: current clusters/ and sites in SVN
• Minor issue: how to keep a “unified view” of these separate repositories for pushing back from a unique place like in the past
• git submodule are not appropriate for this
• git subtree: requires additional options when pushing but may be explored
• Luis will try to find some examples

Core Quattor Schema

standard/pan and standard/quattor general templates were not migrated from SF SVN: need to be added to template-library-core

Some suff added/changed for Aquilon: need to merge asap

• Mostly additions but a few mods (network part in particular)

Actions

Documentation

• Getting Started section on quattor.org
• Allow indexing of Trac wiki by robots
• Move to quattor.org parts of general interest on QWG wiki

Configuration modules

• Implement ncm-ncd hooks
• Add support for component aliasing