What you need to know to understand this document

The purpose of this document: index of options

To help you read other's work I've distilled some of the uses of the various options, and examples for them into an easy-to-search list.

The index itself

-a c -- xapply

Change the expander escape character from % to c. If you don't plan to use the ampersand (&) in a shell command where you were going to call date with a format specification you could avoid the double-precent madness with:

$ xapply -f -c"&" 'processIt -d `date +%Y/%j` -L &1.log &1' todo.cl

-A -- ptbw; xapply

Append resources to the client argument list. This is covered in the HTML document for ptbw (as well as the manual page).

-b -- mmsrc

Output the list of words in each given make macro. For example to output the value of INTO in the recipe file Msrc.mk:

$ mmsrc -b -f Msrc.mk INTO
/usr/src/opt/ksb

While you can specify multiple macros on the command line, there is no sure way to tell where the output from any macro ends when you do that.

-B macro -- hxmd; msrc, mmsrc, efmd

The specified macro must be defined for a host to be included in the selection logic or results. For example to select only hosts that have the attribute LOST defined:

hxmd -B LOST -C domain.cf ...

Since each host must match all such checks we can find hosts that are both lost and found with:

hxmd -B LOST -B FOUND -C domain.cf ...

A negation prefix of exclamation mark (!) is allowed as well. So to find elements that were neither lost nor found:

hxmd -B !LOST -B !FOUND -C domain.cf ...

-B count -- hxmd; msrc, mmsrc, efmd

This is a variant of the check for a definition, but in the case of an integer specification this is a check for the count of the number of configuration files that defined the element. For example if we specify 3 configuration files, we can pick the hosts that occurred in any 2 of them:

$ hxmd -C huey.cf:dewey.cf:louie.cf -B2 ...

As in the previous case a complement operator selects those elements that were defined in any number but the specified value. So you select elements defined in 2 or 3 files one would code:

$ hxmd -C huey.cf:dewey.cf:louie.cf -B!1 ...

It is customary to group the integer with the option specification, for no good reason.

-c cmd -- hxmd; msrc, mmsrc

Specify a custom command for xapply, rather than %+. This option is used to control the command-line protocol between hxmd and xapply. I've only ever used it to debug target commands. One could use it to add a nice command around each target command:

$ hxmd -c 'nice %+' -C site.cf ... 

-C configs -- hxmd; msrc, mmsrc, efmd

This is a key option. It selects the potential population of hosts that may be selected to participate in this task. Any host never listed in such a configuration file (or in one given under -Z below) cannot be targeted. The default configuration file is the name of the program with .cf added to the end. So for the common name "hxmd" it would be "hxmd.cf". You can force every command to include an explicit population specification by never building a file with that name.

The specification of a directory looks for the default filename in that directory. Also the default directory, usually /usr/local/lib/hxmd, is available as -- (double dash).

The specification allows multiple filenames separated by a colon (:). So the specification -C --:huey.cf is a request for the default configuration plus the huey configuration.

-d -- ptbw, xclate; xapply

Do not publish this wrapper level in the linked environment. The new diversion is instead recorded in an environment variable with the letter 'd' where the link number would be. Only use this (as msrc does) to hide your use of a wrapper from other programs. In general you don't need this in scripts.

-D name=value -- m4; hxmd, msrc, mmsrc, efmd

Filter defines passed to every m4. This is largely used to pass a preference down to some marked-up file. The markup in the file controls the expected values.

Note that this never satisfies a -B check: that is to say that forcing a definition at the m4 layer does not force a configuration-level definition. That means that any markup for a host that doesn't hold a definition for the attribute expands to the value presented under this option.

For example if some hosts define a COLOR and you want a default of "white" for the rest:

hxmd -DCOLOR=white -Chuey.cf 'echo HOST COLOR'

-D!name=value -- m4; hxmd, msrc, mmsrc, efmd

The exclamation mark flag is removed from the specification before it is passed on to m4. That is to say it only changes the behavior of hxmd, and only under the -o specification.

Usually all defines (under -D) specified on the command-line are included in any merged configuration file generated under -o. Any specification marked with a leading exclamation mark is not included in the merged configuration file, see below. In the color example above any merged configuration file would explicitly have the COLOR attribute set to "white" for any host which didn't otherwise have a COLOR defined. This is facilitated by putting a default value near the top of the output file (in HXMD_U_MERGED):

$ hxmd -DCOLOR=white -Chuey.cf -o "COLOR" -K 'cat HXMD_U_MERGED' :
COLOR=white
%HOST COLOR
w01.example.com black
w02.example.com .
sulaco.example.com grey
...

In the output above the default definition of COLOR changes the color of "w02.example.com", but with the phrasing below it is left undefined:

$ hxmd -D!COLOR=white -Chuey.cf -o "COLOR" -K 'cat HXMD_U_MERGED' :
#COLOR
%HOST COLOR
w01.example.com black
w02.example.com .
sulaco.example.com grey
...

-e var=dicer -- xapply; hxmd, msrc

Set an environment variable (var) to the expansion of dicer for each process. This option often forces part of the requested resources or parameters into the environment for processing by an indirect task (a descendant process of cmd). For example the test-ppp script takes a phone number, and reads $MODEM for the path to the dial-out modem device:

xapply -f -P4 -t modem.cl -R1 -E MODEM=%t1 './test-ppp %1' modem-bank.1

The process limit of 4 threads and the count of the modem devices (taken from modem.cl) both limit the parallelism, while a list of end-points to test (modem-bank.1) limits the total iterations. The resource count of one (-R1) is the default, but is good form when sometimes the same resource file is taken with other values. Other parts of this structure may need two modems at a time, this option and -J often serve as reminders of purpose.

-E compares -- hxmd; msrc, mmsrc, efmd

Macro value must satisfy given relation to select hosts. The compares expression allows only a single comparison with any of the common math operators: ==, !=, <, <=, > or >=: For example the integer comparison (which uses m4's eval, and is therefore limited to 32 bit math):

hxmd -C dewey.cf -E "1==ROW%2" -E "1==COL%2" 'echo HOST is in both an odd row, and an odd column'

For a string comparison use the shorter operators (equal as = and not-equal as !). For example to find all the hosts that at not blue:

hxmd -C dewey.cf -E blue!COLOR ... 

The expression may be negated with a prefix of !. I'm not sure it is ever is more clear to phrase the restriction as a negative, but it is allowed:

hxmd ... -E "!AGE<21" ... 

-f makefile -- make; msrc, mmsrc

Specify a make recipe file. This might be used to pun a master source directory. That is to say offer more than one control recipe to change the meaning (interpretation) of the structure.

If I were to implement such a thing I would encode the rules in a site policy script that tested for the existence of the punned recipe file, then run msrc with the options I wanted added to the command line options offered:

#!/bin/sh
...
exec msrc -f Reflect.mk "${1+$@}"

If the alternate interpretation is generic, a template recipe file could be offered by default. That improves code reuse and removes a lot of files that would otherwise clutter the name space. Use mmsrc -b to copy values from the current recipe to the template.

-f -- xapply

Treat parameters as files to read for input arguments rather than the arguments themselves. This is covered quite well in the xapply HTML document. The only special feature here that is less clear is that the dash (-) for stdin is special in that multiple specifications on the command line are treated as references to the same stream, where multiple references to the same path are not. For example to read pairs of lines from /etc/group:

xapply -2 -f 'echo %[1:1] %[2:1]' - - </etc/group

While this reads 2 copies of the same line:

xapply -2 -f 'echo %[1:1] %[2:1]' /etc/group /etc/group

-F literal -- hxmd; msrc, mmsrc, efmd

Specify the number of parameters which are marked-up text, as opposed to the remaining parameters which are the names of files which contain marked-up text. This specification allows several styles of parameter specification:

-F 1 control [files]

The default, specify 1 text parameter with subsequent parameters taken as filenames.

-F 0 [files]

A specification of zero forces every parameter to be taken as a filename. Thus the program acts like any plain filter. The first file should be a marked-up executable script (which is run through m4 before it is executed).

-F -1 [files] control

Move the fixed text parameter to the right-most position. Each parameter left of that is taken as a filename. I almost never use this, but if it makes it more clear in your local domain, go for it.

-F 2 control control [files]

Take more than one text parameter (two in this case) from the left most of the parameters specified. This style allows any expression in the first parameters to leverage the xapply dicer against the text of the subsequent literal parameters.

The last form is, in my experience, the most useful. For example when we need to pull part of the RACK attribute out for a local script:

hxmd -F2 ... 'placement HOST %[1:2] HEIGHT' 'RACK'

-g -- hxmd; msrc

Assure that this instance of hxmd is wrapped by an instance of gtfw. While gtfw is not manditory to run this chain it is quite useful later. This option won't work until you install the sshw, gtfw, and wrapw wrapper chain, aka. unix-madness. It is also not supported by mmsrc.

-G guard -- hxmd; msrc, mmsrc, efmd

Expressions to select hosts by name. The guard is usually an m4 macro function that produces a hostname, or a macro attribute tied to each host that names a related host. For example we can produce a list of all of the NFS file servers:

efmd -Cserver.cf -G "ifdef(\`NFS',\`NFS')" HOST

The ifdef markup prevents the host "NFS" from being included by a host without that attribute defined, and is generally not really needed if every hostname is an FQDN (so "-G NFS" would be fine).

-h -- ptbw, xclate, xapply, hxmd, msrc, mmsrc, efmd

This option always provides on-line help for any of my tools.

-H hr -- xclate

Provides a "horizontal rule" or footer after each diversion output. This is used in the environment variable xclate reads at a given level, or in an explicit wrapper around xapply. See below for a spell to put such options in XCLATE_level

-i input -- xapply; hxmd, msrc

Change stdin for the child processes. Common values might be /dev/tty or /dev/null. Usually other files are less than useful as the order subcommands read from them might not be the order expected due to parallel processing.

For example to edit a list of files taken on stdin with xapply:

find-targets |xapply -f -i /dev/tty "${VISUAL:-vi} %1" -

Since the same file descriptor is passed to each child the position in the stream is common to all children, thus they can read the file to advance through it with whatever locking they choose.

-I dirname -- m4; hxmd, msrc, mmsrc, efmd

Include the directory named dirname in the specification passed to every m4 filter. The double-dash specification (--) is replaced with the double-dash directory (usually /usr/local/lib/hxmd). Use this to locate common template files (such as recipe files or shell script templates) that are marked-up with site specific macros. For example to run the marked-up template script report-stat.sh with the msrc entry login as the only parameter:

hxmd -I -- -C heuy.cf -F -1 report-stat.sh ENTRY_LOGIN

-j m4prep -- hxmd; msrc, mmsrc, efmd

The specified m4prep file contains m4 markup which defines macros, but doesn't output any text. Any included markup is used to produce output from the context of host selection, processing of files, redo or filter markup. This is similar to -Y and -Q but applied to all contexts, and must be a file (rather than the aforesaid option's literal strings).

For example to map the name of a host to a class we may have a CLASSOF macro in the file class.m4 in hxmd's double-dash directory (usually /usr/local/lib/hxmd). To use that function in a spell:

hxmd -I -- -j class.m4 ...

This factors the common function out of many configuration files into a single-purpose file that can be separately regression tested and maintained.

-J jobs -- ptbw; xapply, hxmd, msrc, mmsrc

Provide a guess as to the best number of tasks/clients/jobs to limit resource consumption; this allows ptbw to share the available resources more evenly. Normally this is not presented when resources are allocated one-at-a-time, so it is hardly ever used in the Real World™. Like -R this option usually serves more as a declaration of purpose than an formal constraint (more resources may be allocated than suggested when more are available, given large requests from few clients).

The specification is intended to prevent starvation when many clients compete for scarce resources. It is most often applied to a global diversion started on a well-known socket to manage some resource common to a large structure -- not in an ad hoc manner. For example a local machines attached wireless cards may be bound to "wlan" devices. We can pool those with ptbw:

(
umask 027
date +"# wlan devices as of %s" >/var/tmp/wlan.cl
glob -s /dev/wlan[0-9] wlan[0-9][0-9] >>/var/tmp/wlan.cl
daemon -c ptbw -m -J3 -R1 -t /var/tmp/wlan.cl -N /var/run/wlans :
)

Given that process (started at boot) any subsequent process needing a "wlan" device could ask ptwb for one with:

ptbw -t /var/run/wlans -R1 -A -- use-wlan

For the shutdown spell see -Q below.

-k key -- hxmd; msrc, mmsrc, efmd

Change the merge attribute macro from HOST to key. This can have strange side-effects given that most configuration files are nonsense when viewed with a less unique key attribute. For example to find all the unique NFS servers one might use:

efmd -k NFS -L -C server.cf

But that's going to produce errors on stderr for each definition that doesn't provide a definition of "NFS". So you can really only use it when you are sure that every defined element has key defined.

-K filter -- hxmd, msrc

Activate the retry logic structure with filter as the controlling process. There are 2 forms of this option.

-K "|filter options"

In this case the redo status stream generated under -r is available as stdin to the filter process.

-K "script options"

In this case the script should be passed the macro HXMD_0 in its command-line specification so it may consult that stream for the key bits of status information.

For example the pager less makes a pretty good filter in this context:

hxmd -K "|${PAGER:-less}" -C huey.cf 'exit %(u,$)'

or:

hxmd -K "${PAGER:-less} HXMD_0" -C huey.cf 'exit %(u,$)'

I used the mixer/dicer to set the last character of the sequence number as the exit code of each process. Think of is as a cheater's "modulo 10".

Given the default value of -r that outputs each value of HOST followed by the last digit of the sequence number, and the full sequence number (aka. HOST HXMD_STATUS HXMD_U).

-l -- msrc; mmsrc

Set "local mode" in which the platform copy of the master source directory is built on the local host, then the utility command executed from the directory local (rather than on the target host). Use this to access other transport mechanisms (viz. rsync) or some packaging structure.

A local network service might even build platform copies for any client willing to ask for them. A getpeername could provide the address of the client, then a reverse DNS lookup maps that back to a client hostname, which we locate in a local configuration file. If the client passes the reverse DNS and is listed in the local configuration file we can assume that it is safe to expose a configured version of the master directory to them. All that remains is to push it to a temporary directory, then give them back an archive of the results.

For example:

#!/bin/sh
...
# fetched parameters from command-line specification, output the archive
cd $REQUESTED_MSRC
exec msrc -l -C dewey.cf -E HOST=$PEER_REVERSE -- tar cf - .

-L -- efmd

An optimization to allow efmd to avoid any m4 processing. In this mode it just outputs the key value for each element without any filtering. For example to convert three configuration files into a plain-text list of unique hosts:

efmd -L -C huey.cf:dewey.cf:louie.cf |...

-L cap -- xclate

Expand (or limit) the in memory buffer capacity of each diversion buffer. This is hardly ever used, as the 64k default is usually large enough for simple applications. But it is a good example of the mkcmd "bytes" type, so it takes the common bytes suffixes (w, l, b, k, m, g):

xclate -m -L 512k -O /tmp/log.$$ -W /tmp/widow.$$ report-gen.ksh

-m -- ptbw, xclate; xapply

Set a new diversion level for any wrapper. Under xapply a new xclate diversion is started only when none encloses the process.

-m prereq[:postreq] -- msrc, mmsrc

Specify a different make target recipe to gather prerequisite files before an update. The default is __msrc (double underscore plus the name of the program).

When a postreq is also specified it is updated after the push is complete for all hosts. Use this to cleanup any cache directories, the default for an empty specification is __clean.

-M prefix -- hxmd; msrc, mmsrc, efmd

Undocumented option to change the common macro prefix from HXMD_ to prefix. From trying this myself I can tell you it is hard to manage a clean change -- the code supports it but putting this option in every script and command-line is problematic at best. Don't do this unless someone pulls a gun on you.

-n -- xapply, xclate; hxmd, msrc

Trace commands as they would be run, but don't execute them. This option is less than useless in combination with ptbw as it produces output that may violate the exclusive use policy.

-N else -- xapply; hxmd, msrc

This shell command is expanded and executed when no other commands would be executed. That means not enough files or args were specified to build any. In the latest version of xapply the percent positional expander is bound to the whole of the parameter list cmd templates to run. For example to explain there were no files to compress in a list:

xapply -f -N 'echo "No files to compress in %1."' 'gzip -9' /dev/null

If you'd rather fail the command in that case:

xapply -f -N 'echo "No files to compress in %1." 1>&2; exit 65' 'gzip -9' /dev/null

Strangely without the next option you can't see the DATAERR exit code above. That's because xapply succeeded to launch the processes it was asked to launch.

-N notify -- xclate

Specifies a file, process, or socket which receives exit notifications for each completed task, including any else specification.

For example to see the "65" exit code from the last example:

$ xclate -mr -N/dev/tty -- xapply -m -f -N 'echo "No files to compress in %*." 1>&2; exit 65' 'gzip -9' /dev/null
No files to compress in /dev/null.
65,00

That output the exit code (65) and the xid of the process that produced it ("00"). The -N argument could be any file, socket, process, or currently open file descriptor. This is used in my automation to check the exit codes of many parallel tasks -- but you should almost never stop the running machine as a result of a single failure. Let the processes continue to run, then use retry logic to cleanup.

-o config -- hxmd; msrc, mmsrc, efmd

A temporary output file is created with a configuration that represents the selected hosts with each attribute listed in config, and with all the defines specified under -D (unless they were marked with !). The temporary filename is provided via the m4 macro HXMD_U_MERGED.

This is used in recursive calls to msrc and hxmd and is described in detail in the msrc HTML documents and in the efmd HTML document.

-O output -- xclate

The new diversion outputs to this file, socket or process rather than stdout. This option is only used when you were given a stdout that you have to keep while you run a job in background, or to a socket (since the shell won't connect to a unix domain socket for you).

-p pad -- xapply; hxmd, msrc

Fill in missing positional percent parameters with the pad string rather than the empty string. Some common values of pad:

/dev/null

Used under -f when the list of files to read might not be the same length and we opened for read.

0 or -1

Used to even out size computations, or as a sentinel value to mark missing data.

'$PAD'

Used to delay the value of the sentinel until run-time.

/nonexistent

Used to force an error for a missing element used as a path.

In each case the control cmd may contain code to check the parameter for the pad value we are using as a sentinel, or may be designed to do something harmless with that value.

For example let's diff the files listed in 2 manifests, any new files on the end are compared to /dev/null.

$ xapply -2 -f -p/dev/null 'diff -u2 %1 %2' manifest.prev manifest.cur | less

Note that hxmd supports the option, but there should be no way to get that tool to provide mismatched length input streams. Never provide the option to hxmd or msrc, as it may be reused in the future.

-Pjobs -- xapply; hxmd, msrc

Run tasks in parallel (default $PARALLEL). Note that the value and the option specification must be abutted.

This is a key option: with it the whole master source tool-chain works many times faster than single threaded. Because it is so important hxmd sets a default it believes is optimal for the machine on which it was built. If the hxmd binary has been moved to another host, or the tasks provided are largely remote commands, it might be a good idea to specify a better value.

The sleep program offers an easy to understand example, because it stalls for a fixed time (representing any task that takes wall-clock time). The command below takes 30 seconds:

$ time xapply -x 'sleep %1' 7 5 3 7 5 3
sleep 7
sleep 5
sleep 3
sleep 7
sleep 5
sleep 3
       30.05 real         0.01 user         0.03 sys

While the parallel version takes only 7:

$ time xapply -P8 -x 'sleep %1' 7 5 3 7 5 3
sleep 7
sleep 5
sleep 3
sleep 7
sleep 5
sleep 3
        7.02 real         0.01 user         0.07 sys

This is most important when managing complex configuration task's critical resources. For example network bandwidth to/from a backup server might be represented as a process limit, or a ptbw resource limit (each token might represent 100Mb of bandwidth). The peer backup instances can use the resources allocated to them with the certainty that xapply only starts a sane number of them.

-q -- ptbw

Do not complain about resource requests that are impossible; make best effort to process clients requests with the limited resources available. This quiet option is largely used in scripted automation to prevent noise on stderr.

-Q over -- hxmd; msrc

The m4 markup specified as over is placed at the top of the redo stream. This is parallel to the -Y markup (top) placed at the top of the host selection stream. Note that any m4prep files are included before this markup. For example to add the date to the top of the backup report I have to include the output of a shell command, so I can't put the option in Msrc.hxmd, I include the option in the crontab:

7 3 * * *  cd ... && msrc -Q "Backup run for `date +%Y/%j`" -Cdumps.cf -- make daily

-Q -- xclate, ptbw

Every wrapper client accepts this option as a request to tell a master diversion to quit after this client disconnects. Presently only locally coded persistent diversions use this option (later chains use it more).

For example, to shutdown the persistent wlan diversion started under -J above:

ptbw -R1 -t /tmp/run/wlans -AQ -- :

-r redo -- hxmd; msrc

This m4 markup is expanded for each command executed in the context of the target host. The stream generated from this input is fed to -K's filter command (which is also generated from markup).

In the example above we reported the default values. In this example let's change what we report to the pager:

hxmd -r "HOST iteration HXMD_U exit code HXMD_STATUS" -K "|${PAGER:-less}" -C huey.cf 'exit %(u,$)'

-r -- xclate

Report exit codes in the notify stream. Used by hxmd for any retry logic. See the example above.

-R req -- ptbw; hxmd, msrc

Request more resources at a time. For example if this host had a modem pool and needed to connect to 2 remote hosts, one to pull data from and 1 to push that data to, we might ask for that with:

xapply -P3 -f -t modem.cl -R2 -e MODEM1=%t1 -E MODEM2=%t2 'poll2push %1' update.cl

(assuming that the lines in update.cl has both the source and destination information on the same line.

Or to be more explicit by building the ptbw master diversion and passing the modems as command-line options:

ptbw -m -t modem.cl -R2 \
	xapply -P3 -f -R2 'poll2push -I%t1 -O%t2 %1' update.cl

A fine point here is that each resource doesn't have to represent a whole object: a resource could represent a seventh of a network interface's bandwidth, or 21 transactions per second on a disk controller -- the abstraction you apply is up to the implementation you provide. I like to use rsync's --bwlimit=KBPS option as an example: use the count of the number of tokens as a value, or the sum of the tokens taken as numbers, or read a file selected by the token to set the limit for the current task. It is up to you to find the method that works best in your environment and for your coding style.

-s -- hxmd

Turn off any slow-start logic. This can really hurt rate limited processes (like ssh-agent). I never use it.

-S shell -- xapply; hxmd, msrc

Change the shell for each command to shell rather than $SHELL or the default /bin/sh. The shell perl changes the command-line option that specifies shell's mode from -c to -e.

-t tags -- ptbw; xapply

Specify a file (or socket) that contains a list of tokens, and maybe comments. The number of token lines is the count of the available resources. The resources are the lines, but the order they are allocated may not be the order they are presented in the file. Any specified socket should be bound to an instance of ptbw, or be compatible.

For example if I have a list of USB devices in /tmp/usb.278  I could loop though each with:

sed -e '/^#/d' /tmp/usb.278 |xapply -f ... -

After they are all setup with that spell I might use them to hold temporary data for a large process with:

$ xapply -R1 -t /tmp/usb.278 -f 'dd of=%t1 ...' copy.list

In the first command I am iterating over the devices, in the second I am iterating over the copy.list file, but in each iteration I have exclusive access to a USB device I can use to hold my large working file.

-T title -- xclate

Start each new diversion with a the expansion of title. This usually is a banner that identifies the provoking item/resource. It is only possible to insert into an xapply instance via the environment, in which case you must know the present diversion (in xcl_link).

$ x=`expr 1 + ${xcl_link:-0}` ; eval XCLATE_$x="'-T \"%x begins\"'" \; export XCLATE_$x
$ xapply -mu 'echo %1' a b c
0 begins
a
1 begins
b
2 begins
c

-u -- xapply

Force the expansion of %u as the xid for each diversion. This is most useful when the enclosing xclate is sending exit notifications, so they will be matched back to the command that issued the task. This is how hxmd keeps track of the exit code for each task for any redo logic. To see that structure built ask hxmd to track execution under -d:

$ hxmd -d X -K 'date' -Csome.cf -E SHORTHOST=lv426 'echo HOST'
hxmd: xclate -m -r -N >&6 -- xapply -fzmu -sP6 %+ -
lv426.example.com
Fri Oct 12 14:00:00 CDT 2009

From that output you can see that hxmd has made arrangements to leave file descriptor 6 open for the xclate notification stream. And no, that's not pseudo-code, that's really the argument vector as it was passed.

-u unix -- xclate

Bind the new instance of xclate to this unix domain socket. Foreknowledge of this specification makes it far easier for unrelated processes to find the socket. The example in the manual page is good.

-U name -- m4; hxmd, msrc, mmsrc, efmd

Passed as given to m4 to undefine a macro. For example:

$ hxmd -Uunix -Dmacos ...

-v -- xclate

Show more details (under -V) and some internal actions as pseudo-shell commands. For example the xclate wrapper shows the supported protocol version of the current diversions under this option:

$ xclate -m xclate -vV
xclate: $Id: xclate.m,v 2.59 2009/01/04 19:31:12 ksb Exp $
xclate: environment prefix "xcl"
xclate: environment tags: "link", "list", "d", "1"
xclate: protocol version 0.8
xclate: safe directory template: xclXXXXXX
xclate:  1 /tmp/xclVQumqX/1: version 0.8 [target]

-V -- ptbw, xclate, xapply, hxmd, msrc, mmsrc, efmd

All of my tools accept this option to output just a version information page. Use this to safely check the versions of the installed tools against a known-good list. I do that, every week. Programs can regress due to filesystem recovery, and you don't want to regress to broken versions of important tools (like these).

-w -- xclate

Redirect the output of this process to the widows stream, rather than the main diversion. Used to annotate the widows stream with debugging information, progress reports, or other milestones.

$ xclate -m xclate -w bust date +"We lost our minds on %Y/%j" </dev/null

Normally the enclosing xclate is started by some other process (viz. hxmd), and the notification is formatted for a log file, available on the widow stream.

-W widow -- xclate

Send any unmanaged output to this file, process, or socket. This is called the "widow stream" not as a reference to the type-setting meme, but as a reference to loss or loosing something. This is the shortest example I could code:

$ hxmd -Csome.cf -E SHORTHOST=w02 -W /tmp/ksb.widow 'xclate -w %u date +"HOST lost our minds on %%Y/%%j" </dev/null'
$ cat /tmp/ksb.widow
w02.example.com lost our minds on 2009/278

In a real application the widow message would come from a conditional statement in a script, so the double-percent quoting of the date format specification would be unnecessary.

-x -- xapply

Trace commands as they are run on stderr. This is a replacement for the old -v option that is deprecated (which traces on stdout which ends up in the widows stream when xclate is managing output). For example

$ hxmd -x -Csome.cf 'echo HOST' >/dev/null
echo w01.example.com
echo w02.example.com
...

-X ex-configs -- hxmd; msrc, mmsrc, efmd

Add attribute macros from ex-configs to any hosts that have already been defined. Any hosts which have not already been defined are ignored. For example to output all the hosts with a serial number in hardware.cf, that are owned by huey:

$ hxmd -Chuey.cf -X hardware.cf -BSNUMBER 'echo HOST SNUMBER'
w01.example.com 2ZRKN11
w02.example.com 2UA6480K3Q
...

Note that we added the boolean check to omit hosts without a serial number, the complementary optation is:

$ hxmd -Chuey.cf -X hardware.cf -B!SNUMBER 'echo HOST'
sulaco.example.com
...

-y yoke -- msrc, mmsrc

Each yoke is added to the make command-line used to plunder the control recipe for macro values. While any option could be passed on, this is intended to allow name value pair definitions as overrides on the command-line. It is poor form to pass other options to make this way. For example:

$ msrc -y INTO=/tmp/myspace ...
$ cd /tmp/myspace
$ ...

-Y top -- hxmd, msrc, mmsrc, efmd

The m4 expression top is inserted at the top of the host selection stream. This is after any m4prep files, and before any host definitions and guard logic. This allows some markup to setup ordering logic with divert, for example.

To put all the hosts with the attribute Happy first:

$ hxmd -Y "divert(-1)dnl" -G "ifdef(\`SNUMBER',\`divert(3)',\`divert(7)')HOST" -Csome.cf -Xhardware.cf 'echo HOST SNUMBER'
w01.example.com 2ZRKN11
w02.example.com 2UA6480K3Q
...
gawk.example.com SNUMBER
xray.example.com SNUMBER

The diversion markup under -Y assures that if the guard is not applied no hosts will be selected.

-z -- xapply

Process files with NUL as the end-of-line character rather than NL. Sadly this applies to all file input, not just one. See the manual page for example. This option is always used when hxmd starts an xapply, because the command strings passed to that process might have embedded newlines.

-z -- hxmd, msrc, mmsrc

This option suppresses the inclusion of extra options taken from someplace other than the explicit command-line specification. Each of the listed programs interpolates options from some out-of-band source to help make the command-line use of the program less cumbersome:

hxmd normally HXMD, extra from the HXMD_PASS environment variable

Under -z the value of HXMD_PASS is set to the empty string after it is read for options.

msrc normally MSRC, extra read from HXINCLUDE file or from the MMSRC_PASS environment variable

Under -z files included in the HXINCLUDE make macro are ignored (not read for any options).

In each case the -z option prevents acting on the out-of-band specification. An example master source directory includes Msrc.hxmd to limit its effect to test hosts:

# Limit this spell to test host
# Find the level function
-I -- -j level.m4
# Make sure this host is a test host based on level
-E test=LEVEL

If we want to use that on a `beta' level host the spell will silently ignore our request. To override the restriction we can use two tactics (without removing the file).

We can yoke (under -y) HXINCLUDE to the value dot (.), which is a sentinel value that askes for the empty string without reading any files:

$ msrc -y 'HXINCLUDE=.' ...

Or we can apply -z to ask msrc to ignore the files for us, so they won't be sent to the remote host either.

Both of them get the job done, but the -z style works under hxmd as well, and doesn't send noise to the remote host.

$ msrc -z -C beta.cf -E HOST=...

-Z zero-config -- hxmd, msrc, mmsrc, efmd

Each configuration file is read as under -C, but any macro assignments in-scope at the end of the file are taken as default values for any host which doesn't have a more explicit definition of that macro. Host definitions are allowed in such as file, but they might be considered poor form under a sane site policy.

The shorthand double-dash (--) asks for a file named for the program name from the default directory (see above). For example

$ hxmd -Z -- -C some.cf ...
hxmd: stat: /usr/local/lib/hxmd/hxmd.cf: No such file or directory

But if we were to build a symbolic link to hxmd with the basename of a file in /usr/local/lib/hxmd then we could use that shorthand to select that zero configuration file. This is exactly the type of logic msrc employs to force the default configuration filename to be based on its name when it uses hxmd (by replacement of arvg[0] in the call to execvp).

-preload -- hxmd

[This option is undocumented in the manual page, but I'll change that soon.] The slow start code tries to keep 2 slots ahead by default. This allows you to tune that up by specification of a larger preload. I've never needed to do this, but that doesn't mean someone else won't need it.

While msrc doesn't accept it on the command-line, it will pass it when specified from Msrc.hxmd (or any file specified in HXINCLUDE).

Macro details

The HOST macro (aka. the key macro)

If hosts don't show up in the list as you think they should you should add "-d L" to the hxmd command-line. For example this attempt at hostname compression won't work:

$ cat broken.cf
DNQ=`example.com'
%HOST		COLOR
w01.DNQ		red
w02.DNQ		blue
nostromo.DNQ	yellow
...

$ hxmd -d L -C ./broken.cf -E HOST=w01.example.com 'echo found HOST'
w01.example.com

$ grep -f 'w01.example.com' broken.cf

That's why I always list the host key as a FQDN without any trailing dot. That's not to say that the name used is the only name for the host, it might be a CNAME or an interface alias, but it is not compressed or unqualified (use a site policy SHORTNAME for that).

Other hxmd attribute macros

For quicker reference I'll list them here with some example uses. Each is listed by name below:

HXMD_0

Defined in the context of a redo or redo filter, this specifies the name of the file that contains (or will contain) the m4 processed redo test for each element. This is used in the filter context to scan the results for any task we need to process again. See the example above.

HXMD_n

This is defined in the context of processing each parameter to the path to all the parameters output files. For example a processed script may recursively call itself, if it knows which parameter it is on the command-line:

$ echo 'echo "I am HXMD_1 next is HXMD_2"' >test.m4
$ hxmd -C dewey.cf -E SHORTHOST=w02 -x 'cat' test.m4 empty
cat /tmp/hxtf7e8pRW/uP/test.m4 /tmp/hxtf7e8pRW/uP/empty
echo "I am /tmp/hxtf7e8pRW/uP/test.m4 next is /tmp/hxtf7e8pRW/uP/empty"

There is a lot of process here. Note that the test.m4 file knows the location of itself and the subsequent files while it is being built. This is allows for cross references between the files and control commands no matter which order they are presented on the command-line.

I also used the empty file to show the above. That file starts out empty for each iteration, often used to keep state while we process the element, then closed out at the end of processing (rolled up to the next level, so to speak).

HXMD_B

The number of configuration files (under -C or -Z) that define this element. This is the same number against which -B compares. If you need to write any "or" logic then you can use this in a guard.

HXMD_C

The count of the command-line parameters (control and files). This is sometimes used to find the last file specified, which is otherwise hard in m4.

HXMD_OPT_C

HXMD_OPT_X

HXMD_OPT_Z

Each of these records all files read under the corresponding configuration specification option. The names are in a colon (:) separated list. Any reference to stdin is replaced by a temporary file with a copy of the contents presented. For example:

$ hxmd -C - <dewey.cf -E w01=SHORTHOST 'echo HXMD_OPT_C'
/tmp/hxtfIowCLE/cachezK9bx4

This allows recursive calls to hxmd to process the same configuration files as the level above.

HXMD_U

This is a prediction of the value xapply assigned to %u for the loop-iteration for the current host, in every context where a host is being processed (selection, iteration, and retry). For example to check that this is synchronized with xapply this command should never execute the echo command:

$ hxmd -x -Chuey.cf '[ HXMD_U == %u ] || echo HOST'
[ 0 == 0 ] || echo w01.example.com
[ 1 == 1 ] || echo w02.example.com
...

I used the -x option to show the expanded commands as they were run.

HXMD_U_COUNT

The count of the total number of total elements that were defined in any configuration file. This might be used to compute a percentage of the population you are about to touch. It has no intrinsic value to hxmd, other than for client driven statistics. It is defined in all m4 contexts.

HXMD_U_MERGED

Defined in every context, under -o, this is the name of a temporary file which contains the requested extract from the specified configuration. It is a value configuration file for recursive calls to hxmd (or msrc), and is used for that purpose.

If you want to produce a new file from a given file with some attributes combined -o is the option for you. For example let's look at two ways to combine three attributes OS VERSION, and CPU into a single attribute (that someone else needs) HOSTTYPE. The first way is to use a macro definition to combine them into the mapped name for the current command, then output the file we want. (We are assuming that the input macros don't contain double-quotes.)

$ cat source.cf
%HOST		CPU	OS	VERSION
w10.example.com i486	fiction	801

$ echo "%HOST HOSTTYPE" >$OUTPUT
$ hxmd -C source.cf 'echo HOST \"OS-VERSION-CPU\"' >>$OUTPUT
$ cat $OUTPUT
%HOST HOSTTYPE
w10.example.com "fiction-801-i486"

The second way would be to make an aggregation of the configuration with -o to give to your down-stream partner. This file might be better quoted, as hxmd takes care to quote the fields:

$ hxmd -C source.cf -D"HOSTTYPE=OS_VERSION_CPU" -o "OS-VERSION-CPU" -K "cp HXMD_U_MERGED $OUTPUT" :
$ cat $OUTPUT
HOSTTYPE=OS_VERSION_CPU
%HOST OS_VERSION_CPU
w10.example.com fiction-801-i486

There is no good way to make hxmd rename the output column itself, but we could use sed with knowledge that the only line which starts with a percent (%) is the header line. This has the advantage that hxmd does a good job of quoting any white-space or quotes in every value.

$ hxmd -C source.cf -o "OS-VERSION-CPU" -K "sed -e '/^%/s/OS_VERSION_CPU\$/HOSTTYPE/' <HXMD_U_MERGED >$OUTPUT" :
$ cat $OUTPUT
%HOST HOSTTYPE
w10.example.com fiction-801-i486

In any case you get insanity if a source macro is not defined. In the first you get the macro name as the field value, in the other two you get dot (.) as the field value. Use -B to eliminate hosts that lack any required fields.

HXMD_U_SELECTED

The number of hosts selected from the total population. For example:

$ hxmd -E RACK=E5-4 -Chuey.cf 'if [ 0 == %u ] ; then
		set _ $(echo "scale=2; 100*HXMD_U_SELECTED/HXMD_U_COUNT" |bc -l)
		echo "Selected $2 percent of hosts"
	fi
	echo "HOST is $((1+HXMD_U))/HXMD_U_SELECTED"'
Selected 2.70 percent of hosts
nostromo.example.com is 1/5
sulaco.example.com is 2/5
...

HXMD_STATUS

Defined in the context of the redo processing, this is the exit code returned for the iteration of each host.

The msrc attribute macros

SSH

A command that acts like ssh, or the path to a script that is command-line compatible with ssh. This may include some option specification prefix as well as the name of the program.

RSH_CMD

The path to a program rdist can use as transport-path under -P. Note that rdist is not cool enough to do a path search for this program, so you should use an absolute path to be sure you get the right program.

RDIST_PATH

A command to rdist or a command the emulates rdist. This might be different for some hosts that have older (newer) versions of rdistd installed, this may include some additional specification, but that would be unusual.

RDISTD_PATH

The path to a program that rdist can use as rdistd-path under -p. If you have a homogeneous population of hosts you might not need to specify this, or you might just have to force it to a version the you've installed locally.

SDIST

The arrangement of the above macros to build an invocation of rdist up to the specification of a distfile. If you provide a definition it overrides the internal default, because it is defined with the markup:

ifdef(`SDIST',`',`define(SDIST,`RDIST_PATH `'ifdef(`RSH_PATH',`-P`'RSH_PATH') ifdef(`RDISTD_PATH',`-p`'RDISTD_PATH') -f')')

Building your own value for SDIST is almost never the right thing to do because msrc uses the command several times.

ENTRY_DEFS

The absolute path to a shell script sourced with the . (dot) command on the target host before we begin the remote utility, if defined. To be backwards compatible with the 2004 (or 1995) version you'd set it to /usr/local/lib/distrib/local.defs, but most people don't need to put it there unless they also have older distrib-based source structures.

ENTRY_LOGIN

The login that accepts incoming builds. By default none is specified, but command-line option -u is a shorthand for definition of this attribute.

INIT_CMD

PRE_CMD

POST_CMD

These are hooks into the update process designed to support recursion. See the next sections.

How this is put together

The markup starts with a descriptive comment and defaults for the key macros:

dnl remote per-host update script for hxmd
ifdef(`SSH',`',`define(SSH,`ssh')')dnl
ifdef(`RSH_PATH',`',`define(RSH_PATH,`SSH')')dnl
ifdef(`RDIST_PATH',`',`define(RDIST_PATH,`rdist')')dnl
ifdef(`SDIST',`',`define(SDIST,`RDIST_PATH `'ifdef(`RSH_PATH',`-P`'RSH_PATH') ifdef(`RDISTD_PATH',`-p`'RDISTD_PATH') -f')')dnl

Then a hook to take any per-host actions you need on the local host:

ifdef(`INIT_CMD',`INIT_CMD
')dnl

Then up to three calls to rdist to build the remote directory:

dnl myself, subdirs, then files:
SDIST HXMD_2 myself || exit $?
SDIST HXMD_2 subdir || exit $?
SDIST HXMD_2

Then a hook to fail if rdist failed, or do any recursion you need:

ifdef(`PRE_CMD',`PRE_CMD
')dnl

Then the command to run the specified utility in the target directory on the target host with any local definitions set:

SSH ifdef(`ENTRY_LOGIN',`ENTRY_LOGIN@')HOST ifdef(`ENTRY_DEFS',`. defn(`ENTRY_DEFS') \&\& ')`cd into \&\& utility'

Then a chance to cleanup any files you made here, but not ones the control recipe made (because we might have more elements to do):

ifdef(`POST_CMD',`POST_CMD
')dnl

The script doesn't have an explicit exit at the end; thus it will use the exit code from the last command. When you need to force one put it in POST_CMD.

A detailed example of a useful PRE_CMD

For some files we could put a MAP file in that just include's the file we want to send. That works when every host has a file to include. For the time being let's take the case where we have a file to send, later we'll look at the other case.

The best place to hook-in to do this is at PRE_CMD. We have just sent the rest of the payload, and we can hook in our host specific file (or build one and send that) just before we ssh over to make it rain. Because the logic to send has many linkages to the msrc code itself it is pretty complex. For example we must handle both local and remote modes. So the best thing to do is to write once and reuse it.

Here is a blow-by-blow of sendfile.m4 from hxmd's double-dash directory (see the file): You might want to review the list of the five run-time parameters the provision script presented with.

Start by checking to see if we were called in a sane place, if not get hxmd to abort the host processing:

dnl SendFile(source,dest) msrc provision in either local or remote mode (ksb)
pushdef(`SendFile',
`ifdef(`RDIST_PATH',`',`errprint(`msrc: sendfile.m4: RDIST_PATH: definition missing, called outside of provision script?')m4exit(70)')dnl
ifelse($1,`',`errprint(`msrc: sendfile.m4: no file specification')m4exit(66)')dnl

Then we find the destination name. Either $2, or the basename of $1:

`# sending $1
RNAME='$2`
: ${RNAME:=$(basename '$1`)}

We can use the shell to do the work, using the parameters, best check those too

if [ -z "${5}" -o -z "${3}" -o -z "${1}" ] ; then
	echo "msrc: sendfile: no INTO or MODE set in \${5} and \${3}" 1>&2
	exit 76			 # PROTOCOL
fi

If the filename is not the empty string, pick the mode and send the file.

[ -z "'$1`" ] || case _${3} in
_local)
	cp '$1` ${1}/$RNAME ;;
_remote)
	'RDIST_PATH ifdef(`RSH_PATH',`-P`'RSH_PATH') ifdef(`RDISTD_PATH',`-p`'RDISTD_PATH') \
		`-c' $1 ifdef(`ENTRY_LOGIN',`ENTRY_LOGIN@')HOST:${5}/$RNAME` ;;
*)
	echo "msrc: sendfile: ${3}: unknown MODE" 1>&2
	exit 78 ;;		# CONFIG
esac

The standard finish, to stick the landing. We can't exit here because we might be called again, and we've not run the utility yet.

'')dnl

Use this in Msrc.hxmd as

# Fetch the SendFile macros
-I --
-j sendfile.m4
# send the data we built in $YOURS to the far end as "mine.cl"
-D PRE_CMD=SendFile($YOURS,mine.cl)

No file to send, or a variable list

dnl Explain why this is going on here.
include(sendfile.m4)dnl
`if [ -n "$YOURS" -a -f "$YOURS" ] ; then
'SendFile(`$YOURS',`mine.cl')dnl
`fi
'dnl

The macro may be called more than once, so a list of files could be sent -- and that list could be of variable length. This generates a long shell script which calls rdist multiple times, but it is automated. If you wanted to build a distfile and send more files at-a-time you could.

If you code a recursive macro you can loop through a variable list of filenames to send, as pairs, as many as you like.

The common header

dnl Arrange for the msrc provision script to send arbitrary files to the	(ksb)
dnl target cache directory.
dnl
dnl SendFiles(source1,dest1, source2,dest2, ...)  destN may be empty
pushdef(`SendFiles',`ifelse($1,`',`',
`SendFile($1,$2)dnl
SendFiles(shift(shift($*)))')')dnl

And for luck here is a macro to remove/restore all of these

dnl Undo our inclusion, like we were never here				(ksb)
dnl
pushdef(`SendFilePop',`popdef(`SendFilePop')popdef(`SendFiles')popdef(`SendFile')')dnl

Cleanup

When you need to change the diversion be sure to record the previous diversion so you can divert back to the original after your jaunt. For example:

pushdef(`my_div',divnum)divert(7)dnl
rm -f $MY_TEMP
divert(my_div)popdef(`my_div')dnl

Be aware that a POST_CMD could force an exit before the textual end of the file. You might prefer a shell trap on exit. Such a trap might have to refer to a variable holding the present list of files to remove, or match a glob that won't otherwise remove any other files.

See also

The hxmd HTML document and its manual page for lots of information about usage of that tool. Also the configuration file format from hxmd.5.

$Id: options.html,v 1.19 2010/08/14 19:15:10 ksb Exp $