Minutes of the CMT Workshop

1. Issues discussed in the context of the talk of JE Campagne
2. Issues discussed in the context of the talk of RD Schaffer
3. Issues discussed in the context of the talk of J. Hrivnac
4. Issues discussed in the context of the talk of P. Mato
5. Issues discussed in the context of the (remote) talk of Toby Burnett
6. Issues discussed in the context of the (remote) talk of David Quarrie
7. Discussion on SC2/RTAG issues
8. Specific question of the Interface packages
   1. Last minute brainstorming
9. Specific question of the directory structure
10. Various items

---

1 - Issues discussed in the context of the talk of JE Campagne

PPT slides - PDF slides

• A better control is needed over the the mechanisms involved when building shared libraries (typically through the cmt_make_shlib_common.sh script)
  • To obtain info on the current library being built (they were forced to use an internal make macro ($(lib)) not directly handled by CMT)
  • The transitive propagation of shared-lib linker options is not currently automatic (ie. the _shlibflags macro is not inherited similarly to _linkopts). In the discussion, it has been argued that most of the time, and due to very specific architectures (eg. for DLL-based components) it's difficult to automate this. Generally one can simply exploit $(use_linkopts) to setup _shlibflags
  • It should be possible to propagate the trace control (eg. using the QUIET=1 option) to display the internal activity of the cmt_make_shlib_common.sh script.
• It should be possible to exploit the GNUmake features (such as the $(filter-out ...) or other similar functions). Currently the "$(" syntax is captured by CMT if included in the CMT's macro definition.

  This request originally came from the need to filter-out the "-O" option specifically to one source file (or one constituent). It is currently possible to remove some substring from a macro at the package level, or to add specific options at any level (package, constituent, file), but what cannot be done is to remove a substring from a global macro (eg from cppflags) at a file level.

• The possibility to extend CMT to handle filter-out-like mechanisms (inspired from GNUmake) was felt as difficult at least in the short term, both because going that way could be seen as trying to replicate all of GNU make but also because this implies macro expansion to be performed by CMT rather than lately by make. And this needs some more evaluation of the consequences.
• Jean-Eric expressed to need to manipulate wild card expressions representing eg. the list of input sources to constituents (typically to exclude items, or to perform global replacements, text editing). This here again implies to perform by CMT the wild card expressions in the input source list (while it's given as it is to Make).

  One short solution could be here again to rely on GNUmake functions (and thus to permit them in macro specifications). Otherwise, it might be possible to extend CMT language to something like:

  macro_edit macro-name "regular-expression" "replacement" \
             tag        "regular-expression" "replacement"
            

• The tag memory (that source setup.sh -tag=xxx,yyy,zzz provides) was felt as something difficult to manage, especially due to the lack of clear option.

---

2 - Issues discussed in the context of the talk of RD Schaffer

PPT slides - PDF slides

• The concept of control flow versus data flow in CMT which enforces the need of private inheritance
• The architecture used to create the Database architecture (Federations, Schemas, DDL to C++ generations) implies a top package to hide all implementation packages and responsible of all global management operations.

---

3 - Issues discussed in the context of the talk of J. Hrivnac

PDF slides

• Julius shown that it is possible to let CMT handle a very specific and home made set of Make files, which offer somewhat different policies and conventions than what CMT offers by default. This was useful to investigate what should be the CMT support to Java development. In this approach, CMT was only used to get package environment properties (CLASSPATH etc..)
• What is missing now:
  • automated support for JavaDoc
  • support for the Ant mechanisms
  • wider use of Jar files to export different categories of files
  • support for JNI invokation
  • support for more advanced Java features (eg. bytecode engineering)

---

4 - Issues discussed in the context of the talk of P. Mato

PPT slides - PDF slides

• The emphasis put to Interface packages. A standardized set of Interface packages is missing. A specific and complete discussion on this point was conducted the second day.
• The need to clarify the usage of configuration tags, especially those defining the binary tags and the standard binary directories in the context of a project.

  In particular CMTCONFIG should be restored in its centric role of default configuration tag

  Also Pere claimed that the recent feature that memorizes the set of selected tags should be discarded, so as to always rely on the concept of default configuration tag specified though CMTCONFIG when no other precision is given

  This approach should also reactivate properly the possibility to switch at make time to another configuration settings using the gmake tag=xxx syntax

• the concept of package grouping both for the management purposes and to offer a simplified vision of a group of packages to the users.
• Pere described and discussed the LHCb work model based on a two dimensional structure, with 4 levels of packages Layers (Core, Event, Algorithms and applications) and 3 levels of package areas (private, integration and release) which permits them to manage a quasi-continuous integration process and partial incremental releases in a flexible manner.
• There was also a long discussion on the use of wild cards in the use statements which appeared to be complex to manipulate and master.

  It appears that wildcarding is basically required to avoid frequent changes in the requirements files, when strict binary compatibility cannot be enforced between packages (and their shared libraries).

  The ideal mechanism by which all packages present a pure abstract interface is probably the only real case were explicit versioning can be really exploited.

  ...

• Pere expressed its strong concerns on the future compatiliby of CMT itself. It was agreed that this user requirement should be given a very high priority in the future evolutions of CMT, and on the project management style.

---

5 - Issues discussed in the context of the (remote) talk of Toby Burnett

PDF slides

• This GUI was found very nice and we discussed the possibilities to extend its visibility towards non-windows platforms.

  Toby has plans for rewriting the tool using Perl (instead of JavaScript)

---

6 - Issues discussed in the context of the (remote) talk of David Quarrie

PPT slides - PDF slides

• Discussions on the scaling issues (360 packages in Atlas). There were opinions that we should evolve towards less packages (through groupings?).
• The David's concerns about the version directory lead to a very long discussion, and eventually we came up with a proposal for a profund refactoring of CMT (see later in the minutes)
• The problem of long LD_LIBRARY_PATH is discussed, in terms of possible mechanisms based on partial (or complete) installation procedures. Also related is the question of the too large number of environment variables.

  The mechanisms of package grouping may be used for local installations

• Issues on the work model: The role of CMTPATH should probably be increased (like for LHCb) to organize the software production around a small and stable number of package areas. If no versions are specified (i.e. fully wildcarded) does exposing the version (as a directory) for developer releases gain you anything?
• Babar makes heavy use of symlinks, to differentiate between package area (where versions exists in parallel) and user view. But of course symlinking is not portable to Windows (not yet an issue for Atlas but...)

---

7 - Discussion on SC2/RTAG issues

Fons describes the expectations of the SC2/RTAG activities, and the goals for common project management. The OpenSource approach should play a major role in all management practices for the common projects.

About configuration management tools, he distinguishes the activities for the common projects themselves from the experiment activities. Common projects will likely select one common tool (or likely a set of common tools) while they may simply advise a set of recommended tools for the experiments.

A set of standard questions will form the base of an evaluation process for the candidates (At least SCRAM and CMT) concerning the features and the support of the product.

---

8 - Specific question of the Interface packages

PPT slides - PDF slides

• Everybody agrees that having a standard set of Interface packages would be useful. Yet remains to understand how to organize conventions and policies to nicely adapt them to client projects.

  The basic questions that remain to cover:

  • the naming convention for the package themselves: should their name exactly follow the interfaced package, or do we prefix it (possible choices are I<package> or CMT_<package>)
  • were to store the interface packages.
  • how to name the version tag (is it or not related with the native version tag?)
  • how to structure the commonalities or the regularities between interfaced packages (eg. how to describe all CERN packages).
  • do we need a uniform pattern of macros/sets?
• The location. It is proposed to provide the standard interface packages close to CMT itself. For instance in /afs/cern.ch/sw/contrib/CMT/Interfaces
• The naming convention. It seems that prefixing the interfaced packages is preferred. Two options are still possible : I<package> or CMT_<package>
• the version tag convention. Two basic options exist so far:
  1. to follow as much as possible the native convention
  2. to have a version tag only describing the evolutions of the interface itself
  Generally pros for 1) are cons for 2) and vice-versa.
  • in favor of 1):
    • the native version tag is naturally exposed and understood by users
  • in favor of 2):
    • more accurate decription of the evolutions, in terms of compatibility constraints since the evolution is a mixture of the evolutions of the interfaced package and of the interface itself
    • the native version tag generally needs to be transformed to fit the CMT conventions
• the organization of Interfaces packages and associated structuring packages

  It si possible to factor out commonalities of a set of Interfaces into a configuration top packages (LHCb has Externallibs, Atlas has Anaphe, CERNJavaInstallation, AtlasExternalArea, etc). These packages hold the common root definitions and policy.

  The goal is to simplify and centralize as much as possible the customization procedure for the global settings, when Interfaces are exported towards laptops or regional centers. To obtain this, one expects that individual Interfaces can be always constructed without explicit dependency to CMTSITE while transporting this dependency into the common part(s). Unfortunately due to non-homogeheous installation policies this is not always guaranteed.

An action plan is settled down which includes:

• Organizing a CVS repository (with anpnymous access)
• An experimental installation of few examples Interfaces in /afs/cern.ch/sw/contrib/CMT/Interfaces
• An evaluation of the possible naming and patterning conventions
• Selecting one convention amongst the possible ones
• Applying few of these interfaces in real projects
• Evaluating the issues related with transportability to other sites, compatibility between projects, ease of customization

8.1 - Last minute brainstorming

It appears that after some discussions off-workshop, there might be a solution to the problem of the version naming scheme. This consists in introducing a new concept in CMT : the package semantics. The first example of application of this concept would be the Interface package. The implementation could take the form of a new keyword in requirements file such as (let's select one of these).

interface package ICLHEP

or

package -type=interface ICLHEP

or

package_type interface

            

Then we might introduce a standard macro or statement that would describe the native version tag. The point is that the "cmt show uses" would display this automatically, when the package is understood to be an Interface package..

package ICLHEP
native_version xxx

or

package ICLHEP
macro ICLHEP_native_version xxx

            

and:

> cmt show uses
...
#    use ICLHEP v1r2 native_version=xxx

            

---

9 - Specific question of the directory structure

It appears, from different sources (David Quarrie concerns, several customers requests in Atlas, discussions reported by Fons, etc...) that the current directory structure enforced by CMT and especially the extra layer of the version directory is perceived as useless or unnatural, resulting in misunderstanding and extra work load.

Long discussions showed that although considering these remarks was absolutely required, many use cases were in favour of the current structuring style. One typical is the Interface Package area were for obvious reasons, it's requested to be able to present all interface packages in different versions simultaneously and individually. In this use case it is probably impossible to organize one global versioning scheme which would characterize an unlikely synchronous evolution of ALL interface packages for ALL projects at once.

Therefore we had a brainstorming session to understand whether a possible solution can be found to offer a mixture the two styles, while maintaining one of the key principles of CMT, which is the specification of the dependency constraints (through the use statement).

The result of this discussion was positive since it seems that we can make the two structuring styles to coexist peacefully. Moreover, it seems that no conceptual nor technical feature of the current CMT should come up to forbid the implementation.

Here are the proposed mechanisms:

• (unchanged) A package is still identified by a name and a version specification (this concerns the use statement which is unchanged)
• (unchanged) Packages are still localized in package areas described in the CMTPATH search list
• (new) A package may be structured in three different styles:
  1. current style: the version information is maintained along the directory structure

     package/version/src/...
     package/version/cmt/requirements
                       

  2. CVS-based style: the version information comes from the last check-out operation and is maintained in the CVS/Tag file:

     package/src/...
     package/cmt/requirements
     package/CVS/Tag
                       

  3. new CMT-based style: the version information is manually maintained in the file cmt/version.cmt

     package/src/...
     package/cmt/requirements
     package/cmt/version.cmt
                   

• According to the discovered style, the effective version is checked against the specification, using the same compatibility rules as currently.
• The only difference is that only ONE version of a given package may be installed in every package area when the new styles are used.
• Package creation actions ("cmt create" or "cmt checkout") will HAVE to specify the selected structuring style (eg "cmt create A v1 -no_version_directory".
• A priority must be defined when both CVS/Tag and cmt/version.cmt exist
• Each visible package will know its structuring style, and all operations (for building, generating standard macros etc...) will be parameterized by this style.

Now remains to construct a proper action plan to

1. express the detailed specifications of the new mechanisms
2. propose a new design, finding what has to be changed in the current CMT design
3. propose a temptative implementation
4. experiment against exisiting projects

According to the available manpower, my very rough evaluation (having rapidly looked at internal technical implications within CMT) results in one man-month for getting a first temptative implementation to test

---

10 - Various items

PPT slides - PDF slides These less critical points were addressed during the discussion. Most of them were easily agreed on, and should not be hard to implement.

• The naming convention for the library constituents is currently frozen (lib.). Some environments (such as Python glue modules) may require different conventions. Therefore it should be possible to customize it (similarly to the recent possibility to customize the extension for application through the macro $(application_suffix) )
• The macro_append, set_append and path_append statements do not implicitly insert space characters. This is on purpose since there are use cases where space characters would be unwanted. However, it might be nice for end users to have another set of statements (such as macro_extend, set_extend, ...) that would automatically insert a space character.
• There is a graphical tool (cmtgrapher) currently available for generating dependency graphs (it is based on the Grafviz tool). This utility, should be made publicly available from the CMT distribution kit, in particular for using it in the README builder facility so as to insert graphs in Web pages.
• Make fragments should be sensitive to private areas in requirements files.
• CMT should become aware of the new .NET syntax of the MSVC++ configuration files (it is now XML based)