SYNTH(1) BSD General Commands Manual SYNTH(1)NAMEsynth — custom package repository builder for FreeBSD and DragonFly
SYNOPSISsynth [help | configure | version | status | purge-distfiles]
synth [rebuild-repository | prepare-system | upgrade-system]
synth [everything | status-everything]
synth [status | build | just-build | install | force | test]
[file | port-origin ...]
DESCRIPTION
The synth program is an advanced concurrent ports building tool aimed at
system administrators that prefer or require the building of packages
from source rather than installing official binary packages. synth will
build packages in a clean environment can exactly mirror the system that
they are built on, it will create local package respositories and install
pkg(8) repository configuration file that causes locally built packages
to be used with the highest priority, all while allowing the system to
fully upgraded with a single command.
synth version
When synth is executed with the version command, an unknown command, or
no command at all, it will display a short block of text containing an
identification, summary, version number, copyright, usage, and a summary
of valid commands.
synth help
The help command displays lists each valid command with a short synopsis
of the command's function. It also elaborates the several commands expect
one or more port origins to be provided, or a path to a file with the
same information.
synth configure
The configure command launches an interactive menu for providing the user
with a clean method to configure synth. It works on the concept of pro‐
files, and the default profile “LiveSystem” is automatically installed
with values that are determined by scanning the system. Most of the con‐
figurable items are directories that are used to construct the build
environment, but they aren't limited to that.
Ports directory If the PORTSDIR environment variable is set
when the initial profile is created, it estab‐
lishes the initial value. Otherwise, synth will
extract the value of PORTSDIR from
/usr/share/mk and use that. If neither source
points to a valid path, /usr/dports and
/usr/ports will be successfully tried. If
those don't exist either, synth will exit with
an error message.
Packages directory This is the location of the local repository
metadata and all the packages that are built by
synth. It clearly must have lots of available
disk space. The default value is
/var/synth/live_packages but it should be
altered if the system has a /var partition with
limited space.
Distfiles directory This indicates where all the ports source dis‐
tribution files are stored, and the initial
value is determined by scanning the existing
configuration. The defaults are defined by
ports and are /usr/ports/distfiles for
(FreeBSD) and /usr/distfiles for (DragonFly).
Most users will want to keep the initial value
as that directory should already contain dist‐
files.
Port options directory This is the directory where all the selected
options for ports are cached. The initial
value comes from a system scan, so chances are
it has the correct value. However, if the user
would like a separate configuration area for
port options, they would create the new direc‐
tory and set this value accordingly (but the
user would have to ensure the new location is
passed to the port when configuring port
options later in that case).
Build logs directory A log is produced for every port built (subse‐
quent builds of the same port will overwrite
previous logs). This item dictates where those
logs are stored, which is /var/log/synth by
default. If the /var partition is limited, the
user will definitely want to change this loca‐
tion as the uncompressed logs can potentially
consume gigabyte of space.
Build base directory This is the mount point for all the builders.
When tmpfs is used, this location requires very
little space as it contains empty directories
that serve as mount points. If tmpfs is not
used, then this location should have access to
a lot available diskspace as it will provide
the work area and virtual /usr/local for each
builder. The default location of
/usr/obj/synth-live should be fine for most
users (it generally has lots of space and tmpfs
is the default build mode).
System root directory Most users will use the base system as the
basis for the build environment, and thus the
default setting of / is correct. Advanced users
may use DESTDIR with installworld to create an
alternative build base (e.g. i386 cross-build
on amd64 architecture or a modified base that
serves as a fallout test for an upcoming fea‐
ture). Those would be the users that would mod‐
ify this value.
Compiler cache directory If ccache is properly installed on the system,
set this value to the location of the cache.
All the builders will leverage it. It is dis‐
abled by default. It can be disabled again by
entering any invalid path as a new value, e.g.
“none”.
Num. concurrent builders The represents the number of ports that can be
simultaneously built on the system. The
selected value is influenced by the number of
physical cores and hyperthreads the system has,
the amount of memory on the system, the amount
of available swap, and if that swap is a solid-
state drive or not. Generally memory is the
limiting resource when tmpfs is used, so the
default value for the number of builders is
generally 75% of the number of CPUs of the sys‐
tem. The user is free to balance jobs versus
builders as well.
Max. jobs per builder If memory is constrained, it's often a better
performance tradeoff to reduce the number of
builders and increase the number of jobs per
builder. The default value varies depending on
the system, but it will never exceed 5.
Use tmpfs for work area If the initial scan shows there is at least
1.25 Gb per builder, this item will default to
“true”. Building on tmpfs is a big speed spot
and it's recommended if the memory resources
allow it. This item is directed at the work
area for building the port
Use tmpfs for /usr/local Before a port can start building, all the build
and library dependencies have to be installed
into the localbase. Extracting, installing and
later deinstalling is a lot of work for a disk,
so using tmpfs here is another performance
boost. It is recommended that users with ade‐
quate memory leave this enabled.
Display using ncurses During the concurrent building, the default
display is constructed using ncurses. It shows
a lot of good information and it should be
used. The display will fall back to a text mode
if the terminal doesn't support it. This set‐
ting forces text mode unconditionally, but few
users will want or need to use this option.
Fetch prebuilt packages When active, this option will scan the external
repository for suitable prebuilt packages to
satisfy build requirements. To be considered
suitable, the ABI, options, and dependencies of
the remote package must all match requirements.
If a package is located, it will be fetched and
placed in the packages directory and later
incorporated into the local repository. This
option is not active by default, meaning that
synth will normally build everything from
source and ignore external repositories.
To create alternative profiles, press the “>” key at the prompt. A new
menu will appear that provides options to switch to another existing pro‐
file or create a new one that can be tailored and named. Any newly cre‐
ated profile becomes the new default profile, but this can be easily
changed by repeating the profile selection process.
Pressing the Alphabetic key associated with each configuration item
brings up a prompt to change the value. Directory entries must be valid
paths except in the case of ccache. The boolean values are modified with
a single keypress (“T” or “F”) and the numeric items require positive
integers. After changing the values, the old menu returns and altered
values are marked with an asterisk. Simply entering the “Enter” key
(carriage return) saves the changes while pressing the “Escape” key will
discard them.
synth upgrade-system
The upgrade-system command automates the full upgrade of the system upon
which synth is installed. It is expected that the ports tree has been
updated to the latest version before the command is executed. Using the
ports tree as a reference, synth will query pkg(8) to determine what is
currently installed and which software has newer versions available. It
will rebuild those packages and any package that depends libraries they
contain or that have a runtime dependency on the rebuild packages. In
addition, any port that has changed with respect to its options, depen‐
dencies, or ABI will be rebuilt as well. After analysis, the concurrent
builder will begin building the set of packages that have been identified
as requiring a rebuild. When that phase is complete, the local repository
will be rebuilt without waiting for confirmation. Finally, synth will
command pkg(8) to upgrade the system using the local repository, effec‐
tively bringing it completely up to date.
synth prepare-system
The prepare-system command has the same functionality of the
upgrade-system command with the exception that the packages will not be
installed. Once the local repository is fully rebuilt, the command exits.
synth rebuild-repository
The rebuild-repository command will perform a full sanity check on all
packages present in the packages directory and remove the ones which fail
the check. The local Synth repository will be created (or recreated)
using the remaining packages. This step is also done as part of several
other commands, so it mainly exists for scripting purposes.
synth status
The status command with no arguments performs a dry-run of the
upgrade-system command. It will not delete obsolete packages, nor rebuild
the local repository. It will list all the ports that will be rebuild
along with a total, and it also logs the same information to
/tmp/synth_status_results.txt since the full list is often longer than
the terminal height.
synth just-build [file | port-origin ...]
The just-build command takes a list (one or more) of port origins (an
origin is the combination of a category and the port directory, e.g.
lang/python27) and it will builds the packages of those ports if neces‐
sary. If analysis determines the packages are already up to date, nothing
is done.
This is the first of several commands that take arguments. The arguments
are either an unlimited number of port origins, e.g.
synth just-build editors/joe editors/nano editors/libreoffice
or they are limited to one, a path to a file, e.g.
synth just-build /tmp/build.list
A file that is equivalent to the first example would contain:
editors/joe
editors/nano
editors/libreoffice
When the building task is complete, synth exits.
synth build [file | port-origin ...]
The build command differs from the just-build command by asking the user
if they wish to rebuild the local repository when the building is com‐
plete. The answer is usually “N” (for “no”) because rebuilding the repos‐
itory can take a few minutes to complete, and it only makes sense to do
when all the building is complete. Should the user opt to rebuild the
repository, synth will then ask if the user wishes to install the pack‐
ages that come from the ports list on the system.
synth install [file | port-origin ...]
The install command is similar to the build command except that it will
not ask permission to rebuild the repository or install the resulting
packages. It will just do it all in sequence.
synth force [file | port-origin ...]
The force command is similar to the build command except that any pack‐
ages related to the ports list will be deleted first, regardless if they
are up to date or not. This results in that every port on the given list
will build. As with the build command, synth will request permission to
rebuild the repository and possibly install the packages when the build‐
ing phase is complete.
synth test [file | port-origin ...]
The test command will pre-delete any existing packages related to the
ports list before rebuilding them with extra DEVELOPER_MODE tests and
settings. When the building is complete, it will just exit without offer‐
ing to rebuild the repository.
synth status [file | port-origin ...]
This version of the status command provides a dry-run and reports on what
would happen if the just-build command was executed with the same argu‐
ments. The output and logging is similar to the solo status command, but
the build scope is limited to the given ports list.
synth everything
The everything command is not meant for most users. It will attempt to
build every port in the ports tree, which can take several days depending
on the power and resources of the building machine. Even downloading the
tens of gigabytes worth of distfiles consumes significant time!
synth status-everything
The status-everything command is just a dry-run for the everything com‐
mand. Since synth works through increment building (that is, it will not
build a port if an up to date package already exists for it and all the
packages dependencies are also in place) , the resultant build list can
be much shorter than the full list of ports in the tree. Only those rare
users that wish to build the entire set of ports would use this command.
synth purge-distfiles
This is a useful command that will first scan the entire ports tree to
gather a full list of distfiles used by ports, and then it will scan the
contents of the given distfiles directory. All source distribution files
that cannot be matched to at least one port will be removed. This can
result in gigabytes of recovered disk space if significant time has
passed since the distfiles directory was last cleaned.
ENVIRONMENT
PORTSDIR The PORTSDIR environment variable is only checked on the
very first signficant command of synth (meaning everything
except help and version) when the default profile is cre‐
ated. It is also checked when adding new profiles. The
variable dictates the default location of the ports tree
with a higher priority than the standard locations.
SYNTHPROFILE When this environment variable is set to name of an exist‐
ing synth profile, that profile will be loaded when synth
is launch rather than the active profile. It is a tempo‐
rary override; removing the variable will allow the default
profile to load again.
WHYFAIL When this environment variable is defined to any value, the
sanity check routines will provide more detail on depen‐
dency and option check failures on a separate output line.
ENTERAFTER This environment variable only applies with the test com‐
mand, and then only when a single port is specified. If
both of those conditions are satisfied, and the variable
has the value of:
- extract
- patch
- configure
- build
- stage
- install
- deinstall
then an interactive build will be attempted. First, all the
port's prerequisites are built normally and synth will
return to text mode. If everything built successfully, the
specified port will build up to and including the point
specified by ENTERAFTER. At that point, control will be
given to the user by providing them with a tcsh shell with
the current directory set at the root of the builder. When
the user is done, they should terminate the shell with the
exit command to signal synth to clean up and terminate.
This is a developer tool to troubleshoot port build prob‐
lems and it is not intended for the average user.
FILES
The following files may be used or created:
synth.ini This is the configuration file. It is automatically
generated and manipulated using the interactive config‐
uration menu. There is no need to touch this file other
than to remove unwanted profiles as currently that is
not possible from the program itself. It is normally
located at /usr/local/etc/synth/synth.ini
<profile>-make.conf
This is an optional, user-provided file. If it exists,
the builder's /etc/make.conf will be appended with the
contents of this file. For the default profile, the
file would normally be located at
/usr/local/etc/synth/LiveSystem-make.conf
<profile>-environment
This is an optional, user-provided file. It contains a
list of name-value pairs joins with an equals sign
(e.g. HTTP_PROXY=http://proxyserver:8888), one pair per
line. If the file exists, these variables will be
defined in the builder environment. For the default
profile, the file would normally be located at
/usr/local/etc/synth/LiveSystem-environment
synth_status_results.txt
Whenever a status command is run, a list of ports that
would build with the actual command is produced at
/tmp/synth_status_results.txt
synth_prefetch_list.txt
Whenever a status command is run, and the option to use
suitable prebuilt packages has been selected, a list of
packages that will be fetched as a result will be writ‐
ten to /tmp/synth_prefetch_list.txt
port logs Every single attempt at building a port results in a
build log. The name of the log is always in the form of
[category]___[port].log so that subsequent builds will
overwrite previous attempts. They are created in the
directory specified by the configuration.
hook_run_start If this file exists, and it is executable (or it is a
symbolic link that leads to an executable file) then it
will be executed by synth at the start of the build.
The same requirement exists for all hooks. This hook,
and the next five hooks all define the following common
environment variables: PROFILE, DIR_PACKAGES,
DIR_REPOSITORY, DIR_PORTS, DIR_OPTIONS, DIR_DISTFILES,
DIR_LOGS, DIR_BUILDBASE. Uniquely, this hook also
defines PORTS_QUEUED which is the number of ports that
synth will attempt to build. This file must be colated
with synth.ini, so that would be
/usr/local/etc/synth/hook_run_start for most installa‐
tions.
hook_run_end This hook is fired when the build completes. In addi‐
tion to the common definitions, it defines PORTS_BUILT,
PORTS_FAILED, PORTS_IGNORED, and PORTS_SKIPPED in the
environment, which represents the results of the build.
hook_pkg_success This hook is fired whenever a builder successfully
packages a port. In addition to the common definitions,
it defines RESULT=success, ORIGIN, and PKGNAME in the
environment.
hook_pkg_failure This hook is fired whenever a builder fails to package
a port. In addition to the common definitions, it
defines RESULT=failure, ORIGIN, and PKGNAME in the
environment.
hook_pkg_ignored This hook is fired prior to the start of a build when
synth prescans the queue for ports that are to be
ignored according to the ports tree. Once the build has
started, this hook is never fired again. In addition to
the common definitions, it defines RESULT=ignored,
ORIGIN, and PKGNAME in the environment.
hook_pkg_skipped This hook is fired whenever a package gets ignored or
fails which in turns cascades into a number of ports
getting removed from the build queue due to the depen‐
dency's failure. This may fire before the build starts
due to ignored prescanning, or any time during the
build. In addition to the common definitions, it
defines RESULT=skipped, ORIGIN, and PKGNAME in the
environment.
SEE ALSOpkg(8)NOTES
Graceful exit
During development, the interrupt signal (SIGINT) was captured and han‐
dled. It turns out this handling extended to the builders, so any event
resulting in a SIGINT in the build would lock up the builder permanently.
There's no apparent workaround for this given the design, so pressing
Control-C is no longer handled. It will stop the build, but it will leave
everything in bad state. The next significant call to synth will attempt
to clean that up though. If a user wants to stop the build, the best
approach is to hit the Control-Q key combination. The program will shut
down as soon as it can. During the building phase, that mean no new
builds will start, but the current builds will continue until completion.
Unprivileged users
At this time, synth can only be run by the root user.
Impulse indicator
A common question is "What does the Impulse indicator on the display
mean?" It is the package build rate (packages/hour) over the last 500
seconds. For the first roughly ten minutes of a bulk run, the “Impulse”
rate and the “Pkg/Hour” rate will be identical. After that, the “Impulse”
build rate will differ from the rate averaged over the entire elapsed
time, often by a great amount. It's just a way to gauge the package pro‐
duction rate over the previous few minutes.
Supported architectures
Synth is written in Ada, and thus an Ada compiler or cross-compiler is
required to build it. For FreeBSD, that means synth is currently limited
to amd64 and i386. It is feasible to port GNAT to the ARMv7 architecture
as this has already been done (See lang/gnatdroid-armv7 port) but it's
not a trivial task to do it. Other architectures would be challenging,
although not impossible, to support.
HISTORYsynth was conceived long before coding began in December of 2015. It was
designed to be system-agnostic from the beginning, and in particular sup‐
port for FreeBSD and DragonFly was added simulaneously. It was hoped that
it would attract users of the older PortMaster and PortUpgrade ports man‐
agement tools by providing them with a superior and well-maintained
alternative.
synth was written entirely by John Marino ⟨marino@FreeBSD.org⟩ and
released under the Internet Software Consortium license.
BSD February 10, 2016 BSD
