SHMUX(8)SHMUX(8)NAMEshmux - Shell Multiplexor
SYNOPSISshmux [ -h | -V ]
shmux [ -bBdFmpqQstv ] [ -C timeout ] [ -M max ] [ -r rcmd ] [ -S mode
] [ -e list ] [ -E list ] [ -a analyzer ] [ -A condition ] [ -o dir ] [
-P timeout ] [ -T timeout ] -c command [ - | targets... ]
DESCRIPTIONshmux is program for executing the same command on many hosts in paral‐
lel. For each target, a child process is spawned by shmux, and a shell
on the target obtained one of the supported methods: rsh, ssh, or sh.
The output produced by the children is received by shmux and either
(optionally) output in turn to the user, and/or written to files for
later processing. The list of targets may either be specified on the
command line or be read from the standard input.
The default method may be set by using the -r option, and may be over‐
ridden for any target by prefixing the target name by the method and a
colon. Two special methods, ssh1 and ssh2 may be used to select the
SSH protocol version 1 or 2, respectively. For the sh method, it is up
to the specified command to contact the target. In order for the exe‐
cuted command to know which target it is being invoked for, the
SHMUX_TARGET environment variable is set by shmux.
Before executing the specified command, shmux will optionally ping each
target to ensure that it can be reached, and/or run a dummy test com‐
mand to make sure that the target not only is alive, but that it is
possible to cleanly execute a command on it. Both these tests are typ‐
ically run with a fairly short timeout to quickly dismiss unavailable
targets rather than waiting for the standard (longer) network timeout.
After executing the specified command, shmux will optionally process
the output or run a user specified command to determine whether the
command was successful or resulted in an error. Optionally, the com‐
mand output is suppressed for successful targets.
Finally shmux will print out a summary of the results and list targets
with errors.
OPTIONS-h Display a brief help message.
-V Display the version information.
-c command
Specify the command to execute on targets.
-C timeout
Specify a timeout for the command being executed on targets.
This should be a number followed by a time unit. The following
are valid time units: s(econds), m(inutes), h(our), d(ays),
w(eeks). See the PROCESS MANAGEMENT section for details on how
this is handled.
-M max Defines the maximum number of spawned processes. While there is
no real (or hard coded) limitation for this, the system
resources are typically bounded and this affects shmux in turn.
The most critical resource is the maximum number of open files a
process may have. But one should also consider the load imposed
on the system by the creation of a (potentially) large number of
processes.
-r rcmd
Defines the default method used to run a shell on targets.
-S mode
This options defines the strategy used by shmux when spawning
children to run the command on targets. If the mode is set to
"all", then processes will be spawned as fast as possible. When
set to "check", processes will be spawned as long as commands
succeed. Finally, the default mode, "one", is to spawn only one
process at first, and wait for it to succeed, at which point
shmux will switch to the "check" mode. In the "one" or "check"
modes, when the command fails on a target, shmux will stop
spawning new processes.
-F When using either the "one" or "check" spawn modes (see -S),
shmux will stop spawning new processes if the command fails on a
target. By default, shmux then waits for user input if avail‐
able, but reverts to spawning processes if such input cannot be
obtained (because the interactive mode is disabled, either via
-B or lack of a terminal). Using the -F option changes this be‐
haviour to force shmux to quit once all existing processes have
terminated after such a failure.
-e list
Defines which command exit codes should be considered errors and
reported as such. The list should be a comma separated list of
ranges of the form "x", "x-y", "x-" or "-y" where "x" and "y"
are numbers between 0 and 255.
-E list
Defines which command exit codes should always be output to the
screen. Codes that are considered errors (see the -e option)
are always shown, but by default, other exit codes are only
shown if the -v option is specified. This allows more flexibil‐
ity.
-a analyzer
Defines how output should be analyzed by shmux after the command
completes on a target. By default, nothing is done. Valid
options are: lnregex, lnpcre, regex, pcre and run. This option
requires -o to be used as well.
-A condition
When the -a option is used, it is also necessary to configure
the chosen analyzer.
For line based regular expressions (lnregex and lnpcre), the -A
option should be specified twice: once for the standard output
and once for the standard error output. In both cases, the
argument should be the name of a file containing a list of regu‐
lar expressions, one per line. Each regular expression must be
prefixed by the character '=' or '!'. The output of the command
is analyzed line per line by checking whether it matches any of
the specified regular expressions. If the first matching regu‐
lar expression is prefixed by '=', then the command is consid‐
ered to be successful (so far). Any other result (e.g. no
match, or a match prefixed by '!') indicates an error condition.
The default is to consider standard output normal, and standard
error output indicative of an error.
For other regular expressions (regex and pcre), the -A option
should be specified twice: once for the standard output and once
for the standard error output. This option allows defining the
expression to be applied to the output. If the condition starts
with the character '=' then a match indicates that the command
completed without error. If the condition starts with the char‐
acter '!' then a failure to match indicates that the command
completed without error. The second character of the condition
should either be '=' if the regular expression follows or '<' if
the regular expression should be read from the file which name
follows. (The '=' characters may be omitted.) A condition must
be specified for the standard output, but is optional for stan‐
dard error output. The default for the latter is to consider
any output as indicative of an error.
For the run analyzer, the -A must be specified at least once
with the name of a program to run, and optionally a second time
to specify a timeout which defaults to 15 seconds if unset.
This timeout is implemented using alarm(3) and will not other‐
wise be enforced by shmux. The program will be called after a
command completes on a target unless the command exit code is
considered an error (see -e) with two arguments: the target name
and the directory specified with the -o option. The output of
the program will always be shown to the user. Exit codes other
than 0 indicate that the command failed for the particular tar‐
get.
-o dir If specified, shmux will place the output and (for normal termi‐
nations) the exit code of the executed commands in files under
this directory. For each target, the files will be named after
the target name so as to make it easy to identify. The files
must not already exist, so it is recommended that the directory
be empty. This also means that each target must be unique. The
directory will be created if it does not already exist.
-p Ping targets to verify they are alive before doing anything.
The target names must be unique or bad things will happen.
-P timeout
Defines the initial target ping timeout in milliseconds, see
fping(8). (Implies -p.)
-t Before executing the specified command for a target, send a sim‐
ple "echo" test and verify that the output is correct. This
step goes beyond the simple ping test as it verifies that it is
possible to get a shell on the target. If this option is speci‐
fied twice (or once when -T is used), output for failed tests
will be displayed to help the user understand what went wrong.
-T timeout
Defines the test timeout in seconds. (Implies -t.)
-m By default the output is displayed as soon as it is received.
For multi-line outputs, this will typically result in output
from several targets being mixed. This option may be used to
keep each target output separate.
-b Show the bare output from the executed commands instead of pre‐
fixing each line by the corresponding target name.
-B Enables batch mode which forcefully disables the interactive
mode, ignoring input from the terminal. See also -s and -F.
-s Suppress the progress status line. See also -B.
-q Used once, this will suppress the output from successful targets
(as defined by the use of options options -e, -a and -A. If
using the run analyzer, the output of all targets is suppressed.
Note that using this option effectively implies -m in most cases
as failure is ultimately determined upon completion of the com‐
mand. (Only the lnregex and lnpcre analyzers are able to deter‐
mine errors while the command is running.) When specified
twice, this option allows to completely suppress output from the
command run on targets. This also changes the default spawn
stategy to "all". Using this option (once or twice) requires
using -o.
-Q Suppress the final summary of results.
-v Display internal status messages.
-D Display internal debug messages.
EXIT CODESshmux will optionally report the exit code of the command it runs for
each target. For the sh command, this will be the same as the command
itself, but this is the exception. For other methods, the exit code
reported by shmux really is the code returned by the program used to
spawn command for that method.
In particular, rsh typically only returns either 0 or 1 while different
implementations of ssh have different behaviours which may include
returning the actual exit code of the remotely executed command or not.
PROCESS MANAGEMENT
Children processes spawned by shmux are made process group leaders
(using setpgid(2)) in order to group all of their descendants in a sin‐
gle process group which allows to easily send signals to all of these
processes. This is especially useful for the sh method which can gen‐
erate many processes locally, but has no effect on remote processes.
shmux uses pipe(2) to capture the output generated by its children
(even when this output is redirected to a file with -o) and will read
from these until they are closed (by the child as well as its descen‐
dants, if any). Once there is nothing left to read, shmux will wait
for the child and its descendants to terminate before considering that
particular target done with.
The timeout is implemented by using alarm(3) which (unless canceled or
intercepted) will terminate the child process upon receipt but is not
inherited by descendants of that process. For that reason, shmux will
send a SIGALRM signal to the child's process group when the child
itself terminates upon such signal.
After the scheduled SIGALRM time, shmux will wait an additional 5 sec‐
onds for all of the processes in the group to terminate to give the
child a chance to handle the timeout more gracefully if desired. It
will then send a SIGTERM signal to the child's process group. After
another 5 seconds, if any process is still alive in the group, a final
SIGKILL signal is sent to the process group.
Note that only commands which terminate because of an ALRM signal or
because of a TERM or KILL signal after such signal has been issued by
shmux will be reported as having timed out in the final summary report
produced by shmux upon exiting. This is not as accurate as it should
be.
INTERACTIVE MODE
By default, shmux offers a minimal "interactive mode" while running: it
reads commands from the terminal and acts upon them accordingly. This
can be disabled using the -B option. The following commands are cur‐
rently recognized:
q Prevent shmux from spawning any more children, wen for active
children to complete, and then quit.
Q Quit immediately (e.g. abort), not waiting for any active chil‐
dren to terminate. This isn't recommended as children may be
left running unattended. A better alternative is to press con‐
trol-C which will be received by shmux and all running children.
These will typically abort, and shmux terminate in a cleaner
fashion.
space Initiate a pause, this prevents shmux from spawning any more
children to do its work. The following three commands may be
used to resume work.
1 This is equivalent to specifying "-S more" on the command line,
but may be used at any time.
return This is equivalent to specifying "-S check" on the command line,
but may be used at any time.
+ This is equivalent to specifying "-S all" on the command line,
but may be used at any time.
F This toggles the failure mode for the "one" and "check" spawn
modes (see -S) between "pause" and "quit" (see -F).
p Display the list of pending targets.
r Display the list of targets for which a children process is cur‐
rently running.
f Display the list of targets for which shmux failed to run the
command.
e Display the list of targets for which the command completed with
error(s).
s Display the list of targets for which the command completed suc‐
cessfully.
a Display the list of all targets with their status.
k Allows sending a signal to a target.
v Toggles whether internal messages are displayed or not.
D Toggles whether debug messages are displayed or not.
ENVIRONMENT VARIABLESshmux will use the following environment variables if set:
SHMUX_MAX
Specifies the maximum number of processes to spawn simultane‐
ously. If the -M option is specified, it overrides this vari‐
able.
SHMUX_RCMD
Specifies the default command used to run a shell on targets.
If the -r option is specified, it overrides this variable.
SHMUX_SH
Specify the shell to use for targets using the sh method.
(Default is "/bin/sh".)
SHMUX_RSH
Specify an alternative command to "rsh" to use for targets using
the rsh method.
SHMUX_SSH, SHMUX_SSH1, SHMUX_SSH2
Specify an alternative command to "ssh" to use for targets using
the ssh, ssh1 and ssh2 method (respectively). The variable
SHMUX_SSH is used for all methods, but superseded by the other
specific variables.
SHMUX_SSH_OPTS, SHMUX_SSH1_OPTS, SHMUX_SSH2_OPTS
The variables may be used to specify replace the default "-xa"
options passed to "ssh", "ssh1" and "ssh2" (respectively). The
variable SHMUX_SSH_OPTS is used for all methods, but superseded
by the other specific variables.
Multiple arguments may be supplied, using white-space characters
to separate them from each other. You can use single- and/or
double-quotes to preserve white-spaces in an argument (but not
backslashes). Note that doubling a single-quote inside a sin‐
gle-quoted string gives you a single-quote; likewise for double-
quotes (though you need to pay attention to which quotes
your shell is parsing and which quotes shmux is parsing).
SHMUX_SPAWNMODE
Specifies the default spawn mode. If the -S option is speci‐
fied, it overrides this variable.
SHMUX_ERRORCODES
Specifies the default list of exit codes to be considered
errors. If the -e option is specified, it overrides this vari‐
able.
SHMUX_SHOWCODES
Specifies the default list of exit codes to be always display.
If the -E option is specified, it overrides this variable.
EXIT STATUS
The following exit values are returned:
0 indicates that shmux was able to complete its work. However,
this does not mean that all targets successfully ran the com‐
mand, nor that shmux ran without any error.
1 indicates shmux was cleanly interrupted before it could finish
its work.
2 indicates shmux aborted upon user request, possibly leaving some
children running.
3 indicates shmux encountered a fatal error which caused it to
abort, possibly leaving some children running.
4 indicates a syntax error or a fatal error which caused shmux to
abort before it could spawn any children.
SEE ALSOfping(8), sh(1), rsh(1), ssh(1), regex(3), pcre(3).
AVAILABILITY
The latest official release of shmux is available on the web. The home
page is http://web.taranis.org/shmux/
AUTHOR
Christophe Kalt <kalt@taranis.org>
BUGS
All rsh and some ssh implementations effectively achieve user authenti‐
cation by using privileged ports. Since there are only 1024 such
ports, there is an obvious limitation on how many connections may be
open using these programs. To make matters worse such ports may be
held for four minutes (by default) as per the TCP specification. When
using shmux, one can very quickly run out of privileged ports, at which
point failures will start happening.
All rsh and ssh implementations fail to accurately return the remote
command exit code to the caller. Typically, rsh will simply not bother
to return this exit code, and ssh will try its best but is limited by
the fact it also tries to return an error code for SSH related errors.
In either case, this may pose problems when using -e.
Under Cygwin, child processes do not inherit alarm(3) settings from the
parent disabling this simple timeout enforcement system used by shmux.
Some shells will ignore SIGALRM, others die upon its receipt (regard‐
less of any trap). Again, this will affect the simple timeout enforce‐
ment system used by shmux.
Send bug reports to `shmux-bugs@taranis.org'.
$Date: 2007-07-11 08:09:36 $ SHMUX(8)
