To understand this document

This document assumes you are familiar with the standard UNIX™ recipe builder, make(1), and have a working knowledge of rsync or scp.

The goal: you build a private copy of the `msrc` tools

To do that you'll need a local data source that has msrcmux installed (and configured under tcpmux), see the HTML document for details. I'm going to call that machine "msrc.example.com".

If you don't have access to a local unpacked copy of the tools you want to installed, then you are going to have to build them from the packages. I don't have a script for that, but one could be derived from the clues in the recipe file we use here. See Makefile.host The source web site has the instructions to use the package boot sequence to build the first local master source repository; use that.

To install msrcmux you're going to have to be a superuser someplace: even a virtual machine on your workstation should work. You need to setup the network service (msrcmux) on port 1, which is a privileged port. You might need to build a reverse IP map for the instances you want to configure. If you don't have configuration file for hxmd that describes your hosts, then you'll need to build one of those too.

Unsupported hosts

If you want this to work on HP-UX you're going to have to install some packages first: gcc, flex, bison, perl, rsync, and maybe others. I don't have any HP-UX hosts where I can test anymore. Sorry. --ksb

Example session

It would help a lot if you had rsync installed, else one might use scp or ftp to fetch the files needed from the local msrc repository.

I'll build the tools an a test host called "predator", which is configured in "ksb.cf" (you may have to use an absolute path to your configuration file -- it doesn't have to be under hxmd's library directory). I also assume that you already include your own ~/bin in your existing PATH, before any system directories.

First add the new bin and sbin directories to your shell's search path, and hook in a perl library in case we need that, and go ahead and update MANPATH:

predator /home/662/ksb 1  vi .profile	# add lines as you see fit
Add these lines near the end
	export PATH=$HOME/usr/local/bin:$HOME/usr/local/sbin:$PATH
	export PERL5LIB=$HOME/lib/perl5
	export MANPATH=$HOME/usr/local/man:${MANPATH:-/usr/man:/usr/local/man:/usr/share/man}
save and exit

predator /home/662/ksb 2  . ./.profile

Or login again, as you like. Next we'll setup the environment for the build. We need to set 2 variables. The first, MPS, points to the "master pull source" host (or IP address). The second, CFG, holds the name of the configuration file the RFC1078 msrcmux services should read to configure this host. Then we need to rsync down the recipe file to get started building tools.

predator /home/662/ksb 3  MPS=msrc.example.com CFG=ksb.cf

predator /home/662/ksb 4  export MPS CFG

predator /home/662/ksb 5  rsync $MPS::msrc/opt/ksb/home/Makefile.host Makefile

predator /home/662/ksb 6  make -e hosttype

predator /home/662/ksb 7  make -e msrc_base
make[1]: Leaving directory `/home/662/ksb'
echo 'next "make -e install_base"'
next "make -e install_base"

predator /home/662/ksb 8  make -e install_base
...lots more output...
*** Mortal install of installus skipped
...
*** Mortal install of op skipped
echo 'If you need it, "make -e entomb_base" or "make -e ad-hoc-list"'
If you need it, "make -e entomb_base" or "make -e ad-hoc-list"

Check that mk's %~ starts with your home directory path.

predator /home/662/ksb 9  mk -V
...

Next we'll build the stray level2 products that I have recipe rules. Some of these fail based on platform or installed library depends, but that's OK.

predator /home/662/ksb 10  make -e ~/usr/local/lib/libgdbm.a
standard GNU build spew...

predator /home/662/ksb 11  make -e ~/usr/local/sbin/level2s ~/usr/local/bin/oue
...
predator /home/662/ksb 12  oue -V
oue: $Id: oue.m,v 2....
oue: safe file template: oueXXXXXX

predator /home/662/ksb 13  make -s ad-hoc-list | xapply -fx 'make -e %1' -
lots of build and installtion output
Failure to build /home/662/ksb/usr/local/sbin/level3s

Now let's see what didn't get built (besides level3s):

predator /home/662/ksb 14  make -s ad-hoc-list 
List of things that did not build

predator /home/662/ksb 15

That session is pretty typical. It is possible to script the whole thing if all your hosts need to pull the same updates. It is also possible to push the pull recipe to the target host.

Installing the `setuid` applications

If you need op and installus, then you will have to su to the superuser to build them. You'll need to set HOME (which su resets):

predator /home/662/ksb 15  su
Password:
# export HOME=`pwd`
# make -e $HOME/usr/local/bin/op $HOME/usr/local/sbin/installus
# : if you have a local op rule-set master source installed, run this too
# make -e $HOME/usr/local/lib/op/class.cf
# exit

Alternatively op may be installed in your bin as a sentinel. See op's references HTML document for details, read past "Compile a sentinel copy of op". Then use something like:

predator /home/662/ksb ..   make -e GROUP=`id -gn` ${HOME}/bin/op

where you can substitute any group you are in for `id -gn`.

Test that by running the new binary under -V and then under -S. Then build your rule-base.

Sadly installus is not going to work as a mortal cannot read the password file. You can build it and install it, but all the password requests fail, and it is not safe to run without that authentication.

Manual page installation

If you want all the manual pages installed run

predator /home/662/ksb 15  make -e man-all

This you'll have to run $MANPATH through $HOME/usr/local/man to see the pages. Fix that in your profile, if you didn't in the first step.

Cleanup after the build

(N.B. I didn't renumber all the commands for optional steps above.) If you really do not need the source cache or the install backup files you should remove them.

predator /home/662/ksb 15  find ~/usr/src -name Makefile -print | \
	xapply -fx 'cd %[1/-$] && make clean' -

Or just remove the whole source cache, since the source cache is is easy to rebuild:

predator /home/662/ksb 15  rm -rf ~/usr/src

That will not cleanup anything you built as the superuser (you'd have to su to do that).

Then remove install's back-out files. I remove the pull recipe so we don't use it again without getting a fresh copy from the master source pull server. Also let's remove the version of muxcat in our bin, so it doesn't get stale:

predator /home/662/ksb 16  purge -vd0 ~/usr
predator /home/662/ksb 17  rm -f ~/Makefile ~/bin/muxcat

Lastly the values of MPS and CFG were recorded in auto.cf, let's look at them:

predator /home/662/ksb 18  grep MSRCMUX ~/usr/local/lib/hxmd/auto.cf
MSRCMUX="msrcmux"
MSRCMUX_MPS="msrc.example.com"
MSRCMUX_CFG="/usr/local/lib/hxmd/ksb.cf"

The information encoded in that file allows local automation to take action to see if it is available (-BMSRCMUX) then fallback to another tactic (-N else) when it is not available:

predator /home/662/ksb 19  hxmd -Cauto.cf -G localhost -BMSRCMUX -N '%0echo fail' \
	'echo MSRCMUX MSRCMUX_MPS MSRCMUX_CFG'
msrcmux msrc.example.com /usr/local/lib/hxmd/ksb.cf

This could be handy if you have automation to pull newer versions automatically. Because a push will clobber the information in auto.cf, which is not always a good thing. If you need a local policy file to keep these parameters, build one. (And pushes to your home directory are not commonly supported.)

There is a spell in the recipe file to update the copy in your home directory with one that has the values from auto.cf set as the default values:

predator /home/662/ksb 20  make reload
...
hxmd -Cauto.cf -Glocalhost -BMSRCMUX -N "%0echo fail; exit 69" \
	"sh %1 | install -c - your-home/Makefile" update.m4
predator /home/662/ksb 20

You might use it more as a reference for building your own update scripts. The recipe file is clever in that it works with or without being run through m4.

Bugs in the sauce

Some bugs I've noticed in this tactic.

We do not installed distrib's lib

We do not install distrib's library directory. Local policy still uses it, but we're trying to get off that really bad habit. If you want to build it (viz. for local.defs):

predator /home/662/ksb ..  make -e $HOME/usr/local/lib/distrib

msh doesn't work

The message shell (msh) has a RUN_LIB macro we could set, but you should install it someplace you can trust before you make it anyone's shell. This prevents the -V option from working, since it won't succeed without the message directory.

May not install hostlint's policy

The libexec/hostlint-policy directory won't work unless your host is marked with the policycache service. Which is a local site policy here, but mayhap not at your site.

But hostlint's versions.hlc takes an undocumented environment variable PSEUDO_ROOT to force it to check versions rooted below slash. So the policy is actually useful to you. Fetch it with an explicit rsync reference:

predator /home/662/ksb ..  rsync -v hostlint.example.com::hostlint/versions.hlc .
versions.hlc...
predator /home/662/ksb ..  rsync -v hostlint.example.com::hostlint/vercmp .
vercmp.pl...
predator /home/662/ksb ..  PSEUDO_ROOT=$HOME ./versions.hlc

Builds require a modern C compile

I moved to ANSI C (for the most part), which means you need a better C compiler than HP-UX ships with.

Shared home directory across platforms

NFS mounted home directories limit the usefulness of this tactic, as you may have to support different complier output formats from a common home directory. That's your local site policy in action, not mine. The work-around is to build subdirectories name for functions of uname output, then set variable in your profile to match those. E.g. bob builds a copy with "HOME=/home/bob/minix7-i786" on his new Minix workstation.

Tracked alias bugs

Some versions of ksh and bash will not let go of a tracked alias. So if you tried to run xapply before you installed it, then the shell will continue to fail to see the new binary, unless you do something screwy:

predator /home/662/ksb .. exec csh
% exec ksh -i -o vi
predator /home/662/ksb ..

Firewall issues with using rsync and tcpmux

If your instance can't get to the rsync port or the tcpmux port from your host, due to network or firewall restrictions, use ssh port forwarding to map those ports to local loop-back ports. See below.

To resume the build process, after loosing your shell

Assuming you got msrc_base installed you may restore MPS and CFG from the saved values in auto.cf:

$ eval `hxmd -Cauto.cf -G localhost "echo export CFG=MSRCMUX_CFG MPS=MSRCMUX_MPS`

Then just pickup where you left off. This is also a good spell to use for automated update scripts. Later you can use efmd and remove the echo shim.

If you must use an `ssh` port forward

I'm assuming your PATH on the remote host visits ~/bin. If it doesn't, then you'll need to build that directory and add it to your search path. And that you've edited your .profile as in step 1 above (on the target machine).

First build a hxmd configuration file which maps our local hostname to the target machine's HOSTTYPE, HOSTOS, and what ever other attributes required to meet local site policy. We do this because the proxy connection comes from this host (localhost or the reverse name for our IP address), since it makes the network connection to the tcpmux service.

local$ vi /tmp/proxy.cf

Next, scp the Makefile.host from the master source cache to Makefile on the remote machine. Also copy the muxcat program (a perl script) to bin/muxcat. Use ssh's -R option to forward the local tcpmux service to the remote machine on port 3001.

local$ scp `whence muxcat` remote:bin/muxcat
local$ scp /usr/msrc/opt/ksb/home/Makefile.host remote:Makefile
local$ ssh -R 3001:$MPS:1 remote

From the remote shell set MPS, with a prefix to set the port option to muxcat:

remote$ export MPS="-p 3001 localhost" CFG=/tmp/remote.cf
remote$ make -e hosttype
pick up at command 7

Note that this makes the MPS attribute in auto.cf is less-than-useful. This is because we may not have a clear network path back to the host, since we needed an ssh tunnel to get here the first time, and the value in auto.cf is not going to represent the client's network view. This also makes the restart spell above less-that-useful.

Uses of these tactics

I use these tactics to test production upgrades from my home directory. It only takes about 3 minutes to build the whole thing on a high-speed link. Then I do all my testing and remove the whole structure. This is much faster than setting up a whole virtual machine to test, and it test changes almost as well. Save the dedicated test machines for those that don't have your cosmic powers.