To understand this document

This templates in this directory are used to make new master source structures. We also describe the common use of make macros across all the recipe files.

The canonical use of `make` macros

All the master source recipes use a common vocabulary of macro names. This helps multiple engineers work as one to build larger structures, since re-targeting and merging configurations is a whole lot easier.

Here is a list of the macros I've used, and how they fit together. We'll spell them as simply as possible first.

DESTDIR=: This allows an install to target an alternate root directory. It is used by automation to build packages for later deployment. Never set this to an non-empty value in a recipe file. It is only set by a build script, for example on the make/msrc/mmsrc/mk command-line.
TOP=/usr/local: This allows the migration of a product to deploy to a different hierarchy. For example, the common default of /usr/local might become /opt/npcguild. All of the non-variable files are placed under this directory. Never set TOP to the empty string. The shortest TOP would be /, which would be apropos for sh's platform recipe.
VTOP=/var/local: This allows the migration of the variable files (those updated by running an application, starting a service, or adding a new user).
BIN=${DESTDIR}${TOP}/bin: The location of applications listed in section 1 of the manual. The normal shell level interface to an application.
SBIN=${DESTDIR}${TOP}/sbin: The location of application (utilities) in section 8 of the manual. Usually used by system administrator or highly technical users.
CGIBIN=${DESTDIR}${TOP}/cgi-bin: The location of any HTTP services. While they are executable, they do not take standard shell options (usually).
DOC=${DESTDIR}${TOP}/man: The root of the manual page hierarchy. A subdirectory for each section is implied (viz. man1); so don't add a section subdir suffix.
INFODOC=${DESTDIR}${TOP}/info: The root of the info document library.
VAR=${VTOP}
VAR=${VTOP}/subdir: The root of the variable files for this product. If a subdirectory (subdir) is specified a rule should be included to build it with the correct modes.
LIB=${DESTDIR}${TOP}/lib
LIB=${DESTDIR}${TOP}/lib/subdir: The root of the read-only library files for this product. If a subdirectory (subdir) is specified a rule should be included to build it with the correct modes.
LIBEXEC=${DESTDIR}${TOP}/libexec
LIBEXEC=${DESTDIR}${TOP}/libexec/subdir: The root of any executable files which the application may execute a internal helper applications. These should have their own manual pages, if they could be at all useful to other similar applications or their usage needs to be documented for any other reason.
LIBDATA=${DESTDIR}${TOP}/libdata
LIBDATA=${DESTDIR}${TOP}/libdata/subdir: A replacement for LIBDATA, does not allow any executable files at all.
ETC=${DESTDIR}${TOP}/etc/
ETC=${DESTDIR}${TOP}/etc/subdir: This is the mess that was. Now days we use SBIN for any executables. Some things still live under /etc for historical compatibility, but nothing new should be placed there. If I can move to sbin, then so can you.
SHARE=${DESTDIR}${TOP}/share
SHARE=${DESTDIR}${TOP}/share/subdir: A place to put file that may be shared across platforms. Usually plain text files of reference data.

These macros are each only defined if the recipe file requires then. Never clutter a recipe file with dead macros.

N.B. MAN is not the same as DOC. MAN is a list of manual pages, DOC is their target installation hierarchy.

Run-time path requirements

When a recipe needs the run time path to any of those, then build it as:

RUN_BIN=${TOP}/bin
BIN=${DESTDIR}/${RUN_BIN}

This separates the run path from the (possibly forced into an installation forced into a chroot, or jail, or a RPM staging directory) path. You'll also need to pass that computed path down to the application, since it may need it to call their peer applications:

CDEFS=-DMY_RUNBIN=\"${RUN_BIN}\"

Never build any RUN_ variants that you don't need, or other macros you don't use. Always force recursive TOP and DESTDIR rather than forcing just what you think you need, like BIN or SHARE.

all install ...:
	cd subdir && make TOP=${TOP} DESTDIR=${DESTDIR} $@

That way later changes in the recipe to add SBIN won't break your relocation spells.

This also allows automation to use grep to find re-target macros.

In-line markup comments

I've placed in-line comments in all these recipe files. The comments are enclosed in double-percents, like this:

MAN=	%% the manual page for this tool %%

msync: .PHONY%% 3dd unless level1, then D this markup %%
	rcsdiff -q -r\$$ RCS/*,v

the text inside the double-percents is a mix of what to replace the markup with, and vi editor commands that would delete the feature being suggested. The update commands I suggest will be tuned for local site policy.

The master recipes

The master recipes all use the macros that msrc expects. See the HTML document.

Pick one of the master recipe files from the list below, copy it to the new directory as Makefile (or Msrc.mk). Then edit it to fill in the double-percent parts. Whole lines can be deleted or replaced, but partial lines need to be filled in with more care.

Makefile.msrc - gather and send sources with a MAP's recipe

The recipe used for most master source products. This is by far the most common organization. The master side sends all the work to the other side. Usually you'll want to build a `Makefile.host' file from one of the lower case recipe files below.

Since msrc figures out the disposition of each file automatically you don't need to set most of the configuration macros. We include them in the file just in case you need to set one, but for the most part you can just delete those lines.

Makefile.pun - send sources without MAP'ing the recipe

The recipe that is both the master and the platform recipe. I hardly ever use this, as it may lead to triggering the platform targets on the master host. This can be used for file that are installed on every host, like /etc/motd. The recipe must work from the msrc side exactly the same way as it would on each platform. This is really quite rare.

Makefile.squelch - prevent msrc from doing something untoward

When a source directory only represents a common management or ownership of sources we place this recipe in to prevent an accidental mass update of the otherwise unrelated projects.

Makefile.recurse - send ourself and subdirectories [include in another Makefile]

The recipe used to a collection of related source directories, built in a specific order. [The one requires Msrc.hxmd as well. See below.] This is not used as often as the common structure, because most tasks don't require multiple subdirectories to do their work. Large library structures (like mk's library or op's library) use this.

Msrc.hxmd - manage options to hxmd to provide recursive updates

This template is used to loop control back to the master recipe to project subdirectories to build platforms. It may be tuned to local site policy at your site.

It works by building a HXMD_U_MERGED file that records the attributes required to push to each target client. Then it provisions a PRE_CMD that calls back to the recipe file for either of 2 targets based on msrc's shell recursive support (either local_descend or remote_descend).

It passes some make macros from the command-line (DOWN_CFG, LINTO, THOST and TINTO) which provision the recursive call with all the information to get the goods to the correct directory on the correct host.

Capture the list of required configuration files.

DOWN_CFG=-C\ HXMD_U_MERGED`'
	ifdef(`HXMD_OPT_Z',`\ -Z\ HXMD_OPT_Z')`'
	ifdef(`HXMD_OPT_C',`\ -X\ HXMD_OPT_C')`'
	ifdef(`HXMD_OPT_X',`\ -X\ HXMD_OPT_X')

Capture the local cache directory.

LINTO=${1}

Capture the current target host.

THOST=HOST

Capture the remote cache directory, if any.

TINTO=${5}

The 2 commands execute an xapply loop over the directory list in SUBDIR, which looks like either:

remote_descend:

cd %1 && msrc ${DOWN_CFG} -E HOST=${THOST} -y INTO=${TINTO}/%1 -- \
	${MAKE} TOP=${TOP} DESTDIR=${DESTDIR} source

local_descend:

cd %1 && mmsrc ${DOWN_CFG} -E HOST=${THOST} -y INTO=${LINTO}/%1 -- \
	${MAKE} TOP=${TOP} DESTDIR=${DESTDIR} source

Note that in the remote case we target $TINTO (msrc's ${1}), while in the local case we target LINTO (msrc's ${5}). This selects the local cache directory for local revisions (via mmsrc, or msrcmux) and the remote cache for rdist-driven distribution (via msrc or hxmd).

Platform recipe files

The platform recipe files below are customized for every target host (or platform). This allows the build to include custom make macros, special recipe actions, and distinct dependencies. All of these are build with m4 markup, based in the attribute macros hxmd defines for each target host. Here are the recipe templates:

dead - prevent the build of this directory: The directory my not be installed anymore, but keeping it around for reference is harmless, as long it is it never installed.
gnu - install any standard GNU package: If you need to install a older GNU tool which doesn't have a current RPM, this is your best option.
src - send prerequisite files need to compile other products: The files may not get installed, but they are need on the remote host.
data - install files that need no compilation: One step up from src. Install data files for local programs, template files like these, or even web pages (CSS, HTML, graphic files, etc.).
script - install script and a manual page: Every script should have a manual page, and a source. Never imagine that scripts don't matter.
singlec - install a simple C program and a manual page: This could be modified to support any compiled language.
multiplec - install a product from C sources and a manual page: A little more complex than singlec, but not really bad.
genc - build a C program that might use lex, yacc, mkcmd or some other code generator: Another step up from multiplec. This is the template that most of the mkcmd driven ksb tools use.
libc - install a shared library: Totally over the top way to compile: old-school library files (.a), lint library files (.ln), and maybe even shared libraries (.so). The shared code is presently not included.

Every one of these recipe files may have been locally edited to conform to local site policy. The original template files contain some assumptions about how source is laid out in each directory, the common revision control structure, and the names of local tools (like maketd, explode, and calls) which your site my not install or support. I don't presently distribute mkcat, but I will again someday.

Package recipe files, layer 3

Package recipe files gather level 2 products together into a common staging directory. From there we may run a few cross-checks, then we build a source package, which might be SRPM for rpm, or source tar file, or the like. That source product has all the recipes and information required to build a binary (aka. distribution) package (RPM, pkgsrc directory, ports directory, depot file, or some local packaging system).

This completes the build from level 1 (files) to level 2 (products) to level 3 (packages). If your packages contain correct dependency meta-data, then the level 4 (running system) step is just package adds for the subscribing hosts. That leaves local site policy, which is totally your problem. See msrc base, if you forgot why you were here.

Note that these use some additional make macros:

TMP - a temporary directory, usually /tmp or /var/tmp: Often tuned to put the package being built in some other staging area.
RELEASE - the product's current release number, e.g. 5.7: The release number of the package, which should change for updated release. (How that happens is local site policy, of course.)
SUBDIR - provision-ordered product directories, in the package provision template: The list of product directories in the order in which they should be provisioned. If order doesn't matter, use the order in which they should be built (below).
SUBDIR - build-ordered product directories, in the package build template: The list of all the product directories to visit, in the correct order required to build each.

The package builder recipe template Package

The package builder template takes a while to construct, but it is well worth it. I aggregates a list of level 2 packages into a single build directory. When assembled the components become a consistent structure that makes a level 3 package.

Here are the related make targets:

stage:: This recipe constructs the package under /tmp as a source hierarchy. It may have a target to build a compressed tar archive of the resulting directory. It is a separate process to then construct the output from that package.
msync:: Used to check the msync status of the packaging recipe. This has been replaced by msync's -3 option, but some recipes still carry the logic.
sub_sync:: Used to check the msync status of all the level 2 products under this level 3 package.; Build a new package directory under ${TMP}. The doesn't create any compressed tar archive.
check: - ad hoc check: This target checks that the code in the staged package matches the code in each master directory. This is a last minute check I like to run before I send out a package build command, after I stage the package, but before I produce RPMs. It may be of little use to other people, it depends on a stable source tree with the current revisions visible.

There is an assumption in the recipe that the name of the recipe file is Makefile. If local site policy changes that name the called to mk (for example) will not work.

This recipe also has some mk markup for msync, level2s, and level3s:

# $Msync(group): - msync

# $Msync(target): - msync

These are documented in msync's manual page and HTML document. These tunes allow msync to accept a different group owner, or provide a make recipe to replace the default logic.

# $Level2s: - level3s

This is the hook that level3s uses to find the list of level 2 packages (by symbolic revision and path to the source directory).
Trigger with:

$ mk -mLevel2s Makefile
Eight /usr/Pkgs/install_base
Eight ${MSRC}/local/lib/install.cf
...

Ownership changed via `op` rules

Last, but not least, I package my sources up with a common owner (in the filesystem). To get that common owner I need to be able to chown the staged files (as the superuser). So there is an op rule installed that allows anyone in group source to chown a staging directory. See the update action under the clean and stage targets:

clean:
	-[ -d ${STAGE} ] && op -u $${USER:=$$LOGNAME} level2s-chown ${STAGE}
	rm -rf ${STAGE}

stage: ${STAGE} ${STAGE}/Pkgs/${PKG} %% add all the dirs you wanted %%
	addlic ${STAGE}
	find ${STAGE} -type d -exec chmod g+w '{}' \;
	op -u source level2s-chown ${STAGE}

Your local site policy may not required common ownership of source files. You may delete the 2 lines, or make the op rule do nothing but succeed.

The level 3 master template Package.meta

This recipe becomes the package's Makefile as part of the process above. That makes it the controlling recipe for the subsequent master build process for the package data, which is effectively just a recursive level 2 build.

Quite often this is triggered from a package system specific file, like the rpm specific "spec" file. If you need one of those, just include it as you would any other file. I always put markup in it (mk or the like) to remind me how to trigger the build process. And I always call my spec file ITO.spec, which is just local site policy.

The spec file does contain some mk markup for level2s, which knows how to build at least the source tar for most level 2 products. The markers are KeyFile, Level2, and Level3. See the level2s manual page.

The level 3 platform build template Package.host

This recipe is the platform build recipe for each target host. It is a loop through the products to build them in the correct order. Only the targets all, clean, and install are provisioned (unless you add more).

This is really the most trivial of recipes. The actual build logic is left to the product recipes, which is as it should be. Either the packaging system knows how to gather the built files from the source directory (which is bad), or their is a target in the recipes to install into a chroot-like environment, from which the files are recovered. This is where DESTDIR and TOP come in handy.

Evolution

These templates are not set in stone, they are not perfect, they are not even complete. I expect any local implementation to learn from them, then evolve them to fit local needs. I surely don't expect everyone to start using RCS as their revision control structure.

The point of all of this is to give you a boost above the noise: so your group will focus on a repeatable process that can build anything you need. To do that, your team must start with something that works, then move to something better. This works, and it is better than starting from nothing.

The obvious things I almost never use are the autoconf tools. That doesn't mean you shouldn't use them. I just have more things to configure that have nothing to do with building programs. (For example op and sudo rule-bases.) For my work-load msrc-driven m4 works far better than ./configure.

And once you have a powerful tool, you find more ways to use it.

Summary

Read, understand, adopt and adapt.

Any technology comes with good and bad elements, this is no exception. I've spent some time making this work for me and my peers. It has some "local flavor" which you might not think you'll like, but the power it enables is truly awesome.

You only have to follow a UNIX command learning curve, and most of the learning is incremental (like you don't have to learn a whole new way to spell or run chmod).

Stay calm, and carry on.

-- ksb, Jan 2013

$Id: master.html,v 1.15 2013/01/04 15:55:58 ksb Exp $