VM-BHYVE(8) BSD System Manager's Manual VM-BHYVE(8)NAMEvm — utility to manage bhyve virtual machines
SYNOPSISvm init
vm switch list
vm switch info [name]
vm switch create name
vm switch import name bridge
vm switch vlan name vlan-id
vm switch nat name on|off
vm switch add name interface
vm switch remove name interface
vm switch destroy name
vm create [-t template] [-s size] name
vm destroy name
vm list
vm info [name]
vm install name iso
vm start name
vm stop name
vm console name [com1|com2]
vm rename name new-name
vm add [-d device] [-t type] [-s size|switch] name
vm reset name
vm poweroff name
vm startall
vm stopall
vm configure name
vm clone name new-name
vm snapshot [-f] name|name@snapshot
vm rollback [-r] name@snapshot
vm iso [url]
vm image list
vm image create [-d description] name
vm image provision uuid new-name
vm image destroy uuid
DESCRIPTION
The vm utility is used to provide simplified management of bhyve(8) vir‐
tual machines, including networking and console access.
Networking is handled by creating one or more virtual switches. Each
switch has a simple name which is referenced in the virtual machine con‐
figuration file. The vm utility automatically creates a bridge(4) device
for each virtual switch and assigns virtual machine tap(4) interfaces
dynamically.
All configuration for virtual machines is stored in a simple rc style
configuration file. When virtual machines are first created, the configu‐
ration file is copied from a template which can be specified by the user.
Multiple templates can be created providing an easy way to provision
guests with specific configurations.
vm gracefully handles reboot and shutdown commands from inside the
guests, whilst providing full management of the virtual machine from the
host system.
BASIC SETUP
Once vm is installed, create the directory which will store your virtual
machine configuration and data. This directory will be referred to as
$vm_dir throughput this man page.
Add the following into /etc/rc.conf
vm_enable="YES"
vm_dir="/your/vm/path"
vm_list=""
vm_delay="5"
The first and second lines are required to enable the vm utiltity. Please
see the startall command description for more information on the third
and fourth settings.
Now run the vm init command to finish initialisation. This will create
subdirectories inside $vm_dir to hold configuration and templates. It
will also load any required kernel modules. This command needs to be run
on each boot, which is normally handled by the rc.d script.
Copy the sample templates into the directory $vm_dir/.templates/. You
can create and edit the templates as required. It is recommended to keep
a template called default.conf, as this will be used when no template is
manually specified.
ZFS
If you are using a ZFS dataset to store your virtual machines, and want a
new child dataset created for each one, specify the dataset to use in
/etc/rc.conf as follows:
$vm_dir="zfs:pool/dataset"
In contrast to earlier versions, if $vm_dir is a normal path, a standard
subdirectory will be created for each virtual machine, regardless of the
file system type. However, vm is now able to handle situations where the
dataset mountpoint does not match the dataset name.
QUICKSTART
Create a virtual switch called public (which is the switch name specified
in the default templates) and attach it to a real interface. Use your
own interface in place of em0 as required.
# vm switch create public
# vm switch add public em0
Download an ISO file to use for installation:
# vm iso ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-disc1.iso
Create a new guest using the default template and disk size, then start
the installation. The install subcommand will pause while the guest boot
loader completes. Once successfully loaded, use the console command to
complete the installation.
# vm create my-guest
# vm install my-guest FreeBSD-10.1-RELEASE-amd64-disc1.iso
# vm console my-guest
Please note that Linux guests currently require the sysutils/grub2-bhyve
package to be installed. This is use in place of bhyveload(8) to load the
guest kernel into memory.
WINDOWS SUPPORT
Windows guests are supported on versions of FreeBSD that have UEFI sup‐
port in bhyve(8). As of October 2015, this is only FreeBSD-CURRENT, so
you will first need to build and install the head FreeBSD branch manu‐
ally.
You will also need a copy of the UEFI firmware (see URL below), which
needs to be placed in $vm_dir/.config/BHYVE_UEFI.fd.
As there is no VGA console in bhyve(8), an unattended installation ISO is
required which allows Windows to install and boot without any user inter‐
action. Instructions for creating a suitable ISO can be found at the URL
below.
Once the installation ISO is ready, has been placed in the $vm_dir/.iso
directory, and you have the UEFI firmware, installation can be performed
as normal.
# vm create -t windows -s 30G winguest
# vm install winguest win_repack.iso
Windows installation has been tested with 2012r2 and takes around 20-25
minutes. During install, the guest will reboot twice (three runs in
total). You can see the guest reboot by watching the log file
$vm_dir/guestname/vm-bhyve.log. The third run should boot fully into
Windows. The virtio network adapter will request an IP address using
DHCP. Connect to the guest console and press i to see the IP address that
has been assigned. The default unattended installation files should make
RDP available, using Administrator and Test123 as the default login
details.
The UEFI firmware (BHYVE_UEFI_20151002.fd), as well as instructions for
creating an unattended installation ISO can currently be obtained from
https://people.freebsd.org/~grehan/bhyve_uefi/
SUBCOMMANDS
init
This should be run once after each host reboot before running any
other vm commands. The main function of the init command is as
follows:
o Load all necessary kernel modules if not already loaded
o Set tap devices to come up automatically when opened
o Create any configured virtual switches
switch list
List virtual switches. This reads all configured virtual switches
from the $vm_dir/.config/switch file and displays them. If the
virtual switches are loaded, it also tries to display the
bridge(4) interface that has been assigned to each one.
switch info [name]
This command shows detailed information about the specified vir‐
tual switch. If no switch name is provided, information is out‐
put for all configured switches. Information displayed includes
the following:
o Basic switch settings
o Overall bytes sent and received via this switch
o Physical ports connected
o Virtual ports, including the associated virtual machine
switch create name
Create a new virtual switch. The name must be supplied and may
only contain letters, numbers and dashes. However, it may not
contain a dash at the beginning or end.
When a new virtual switch is created, the persistent configura‐
tion file is updated and a new bridge(4) interface is provi‐
sioned.
switch import name bridge
This command allows you to import an existing bridge interface
that has been created manually and use it for virtual machines.
Once a bridge is imported, you can use the switch name in guest
configuration. Ideally the manual bridge should be configured in
/etc/rc.conf, so that it is available on each host boot.
Please note that this creates a 'manual' switch and is designed
to allow you to configure your own bridge. None of the add,
remove, vlan or nat commands are supported on manual switches.
If a manual switch is destroyed using the destroy command, we
remove all vm-bhvye configuration, but leave the bridge(4) inter‐
face intact.
switch vlan name vlan-id
Assign a VLAN number to a virtual switch. The VLAN number must be
between 0-4094.
When adding an interface to a VLAN enabled virtual switch, a new
vlan(4) interface is created. This interface has the relevent
parent interface and VLAN tag configured. This vlan interface is
then added to the virtual switch. As such, all traffic between
guests on the same switch is untagged and travels freely. How‐
ever, all traffic exiting via physical interfaces is tagged.
If the virtual switch already has physical interfaces assigned,
they are all removed from the bridge, reconfigured, then re-
added.
To remove the VLAN configuration from a virtual switch, specify a
vlan-id of 0.
switch nat name on|off
Enable or disable NAT functionality on the specified switch.
Please note that pf is required for this functionality and must
be enabled in /etc/rc.conf. If DHCP is desired, please install
the dnsmasq pacakge. vm-bhyve will generate a sample dnsmasq con‐
figuration in /usr/local/etc/dnsmasq.conf.bhyve, but it is up to
the user to either use this configuration directly, or merge with
any existing dnsmasq settings you have configured.
The switch should have no host ports assigned, as these will end
up on the private side of the NAT network. vm automatically
detects the hosts default gateway, which is used as the forward‐
ing interface for NAT connections.
Once enabled, a 172.16.X.0/24 network is assigned to the switch
(bridge) interface. X is chosen based on the ID of the bridge
interface. For example, if the switch is using bridge10, the net‐
work will be 172.16.10.0/24. dnsmasq can be used to provide DHCP
to the guests, and pf rules are inserted to provide the NAT
translation.
/etc/pf.conf is created if it doesn't exist, and a single include
statement is added. This include statement can be moved within
the file if required.
switch add name interface
Add the specified interface to the named virtual switch.
The interface will immediately be added to the relevant bridge if
possible, and stored in the persistent switch configuration file.
If a vlan-id is specified on the virtual switch, this will cause
a new vlan(4) interface to be created.
switch remove name interface
Removes the specified interface from the named virtual switch and
updates the persistent configuration file.
switch destroy name
Completely remove the named virtual switch and all configuration.
The associated bridge(4) interface will be removed, as well as
any vlan(4) interfaces if they are not in use by other virtual
switches.
create [-t template] [-s size] name
Create a new virtual machine.
Unless specified, the default.conf template will be used and a
20GB virtual disk image is created. This command will created the
virtual machine directory $vm_dir/$name, and create the configu‐
ration file and empty disk image within.
-t template Specifies the template to use from within the
$vm_dir/.templates directory. The .conf suffix is
not required.
-s size The size of disk image to create in GB. Unless
specified, the guest image will be a sparse file
20GB in size.
destroy name
Removes the specified virtual machine from the system, deleting
all associated disk images & configuration.
list
List all the virtual machines in the $vm_dir directory. This will
show the basic configuration for each virtual machine, and
whether they are currently running.
info [name]
Shows detailed information about the specified virtual machine.
If no name is given, information for all virtual machines is dis‐
played.
This output includes detailed information about network and disk
devices, including the space usage for all virtual disks (exlud‐
ing custom disk devices). If the guest is running, the output
also shows the amount of host memory curently in use, and addi‐
tional network details including bytes sent/received for each
virtual interface.
install name iso
Start a guest installation for the named virtual machine, using
the specified ISO file. The iso argument should be the filename
of an ISO file already downloaded into the $vm_dir/.iso direc‐
tory. ISO files in this directory can be managed using the iso
subcommand described below.
Once started, the guest loader will be booted in the foreground.
This allows you to choose the Install boot option for guests that
require it. Once the loader has completed, you will be returned
to the shell and bhyve will continue running in the background.
Use the console subcommand to connect to the guest and complete
installation.
After installation, the guest can be rebooted and will restart
using its own disk image to boot. At this point the installation
ISO file is still attached, allowing you to use the CD/DVD image
for any post installation tasks. The ISO file will remain
attached after each reboot until the guest is fully stopped.
start name
Start the named virtual machine. The guest will boot and run com‐
pletely in the background. Use the console subcommand to connect
to it if required.
For each network adapter specified in the guest configuration, a
tap(4) interface will be created. If possible, the tap interface
will be attached the relevent bridge(4) interface, based on the
virtual switch specified in the guest configuration.
stop name
Stop a named virtual machine. All tap(4) and nmdm(4) devices will
be automatically cleaned up once the guest has exited.
console name [com1|com2]
Connect to the console of the named virtual machine. Without net‐
work access, this is the primary way of connecting to the guest
once it is running.
By default this will connect to the first com port specified in
the client configuration, which is usually com1. Alternatively
you can specify the com port to connect to.
This looks for the nmdm(4) device associated with the virtual
machine, and connects to it with cu(1). Use ~+Ctrl-D to exit the
console and return to the host.
rename name new-name
Renames the specified virtual machine. The guest must be stopped
to use this function.
add [-d device] [-t type] [-s size|switch] name
Add a new network or disk device to the named virtual machine.
The options depend on the type of device that is being added:
-d device The type of device to add. Currently this can
either be disk or network
-t type For disk devices, this specifies the type of
disk device to create. Valid options for this
are zvol, sparse-zvol and file. If not speci‐
fied, this defaults to file.
-s size|switch For disk devices, this is used to specify the
size of the disk image to create. For network
devices, use this option to specify the virtual
switch to connect the network interface to.
For both types of device, the emulation type will be chosen auto‐
matically based on the emulation used for the existing guest
devices.
reset name
Forcefully reset the named virtual machine. This can cause cor‐
ruption to the guest file system just as with real hardware and
should only be used if necessary.
poweroff name
Forcefully power off the named virtual machine. As with reset
above, this does not inform the guest to shutdown gracefully and
should only be used if the guest can not be shut down using nor‐
mal methods.
startall
Start all virtual machines configured for auto-start. This is the
command used by the rc.d scripts to start all machines on boot.
The list of virtual machines should be specified using the
$vm_list variable in /etc/rc.conf. This allows you to use shared
storage for virtual machine data, whilst making sure that the
correct guests are started automatically on each host. (Or to
just make sure your required guests start on boot whilst leaving
test/un-needed guests alone)
The delay between starting guests can be set using the $vm_delay
variable, which defaults to 5 seconds. Too small a delay can
cause problems, as each guest doesn't have enough time to claim a
null modem device before the next guest starts. Increasing this
value can be useful if you have disk-intensive guests and want to
give each guest a chance to fully boot before the next starts.
stopall
Stop all running virtual machines. This sends a stop command to
all bhyve(8) instances, regardless of whether they were starting
using vm or not.
configure name
The configure command simply opens the virtual machine configura‐
tion file in your default editor, allowing you to easily make
changes. Please note, changes do not take effect until the vir‐
tual machine is fully shutdown and restarted.
clone name new-name
Create a clone of the virtual machine name, as long as it is cur‐
rently powered off. The new machine will be called new-name, and
will be ready to boot with a newly assigned UUID and empty log
file.
Please note that this function requires ZFS, and a snapshot will
be taken of the original guest, along with any descendant
datasets.
snapshot [-f] name|name@snapshot
Create a snapshot of the names virtual machine. This command is
only supported with ZFS and will take a snapshot of the guest
dataset and any descendent ZVOL devices.
The guest and snapshot name can be specified in the normal
name@snapshot way familiar to ZFS users. If no snapshot name is
given, the snapshot is based on the current timestamp in
Y-m-d-H:M:S format.
By default the guest must be stopped to use this command,
although you can force a snapshot of a running guest by using the
-f option.
rollback [-r] name@snapshot
Rollback the guest to the specified snapshot. This will roll back
the guest dataset and all descendent ZVOL devices.
Normally, ZFS will only allow you to roll back to the most recent
snapshot. If the snapshot given is not the most recent, ZFS will
produce a warning detailing that you need to use the -r option to
remove the more recent snapshots. It will also produce a list of
the snapshots that will be destroyed if you use this option. The
-r option can be passed directly into vm rollback
The guest must always be stopped to use this command.
iso [url]
List all the ISO files currently stored in the $vm_dir/.iso
directory. This is often useful during guest installation, allow‐
ing you to copy and paste the ISO filename.
If a url is specified, instead of listing ISO files, it attempts
to download the given file using fetch(1).
image list
List available images. Any virtual machine can be packaged into
an image, which can then be used to create additional machines.
All images have a globally unique ID (UUID) which is used to
identify them. The list command shows the UUID, the original
machine name, the date it was created and a short description of
the image.
Please note that these commands rely on using ZFS featured to
package/unpackage the images, and as such are only available when
using a ZFS dataset as the storage location.
image create [-d description] name
Create a new image from the named virtual machine. This will cre‐
ate a compressed copy of the original guest dataset, which is
stored in the $vm_dir/images directory. It also creates a
UUID.manifest file which contains details about the image.
Once complete, it will display the UUID which has been assigned
to this image.
image provision uuid new-name
Create a new virtual machine, named new-name, from the specified
image UUID.
image destroy uuid
Destroy the specified image.
CONFIGURATION FORMAT
Each virtual machine has a configuration file that specifies the hardware
configuration. This uses a similar format to the rc files, making them
easy to edit by hand. The settings for each guest are stored in
$vm_dir/$vm_name/$vm_name.conf. An overview of the available configura‐
tion options is listed below.
guest Specify the type of guest to be installed in this vir‐
tual machine. Current valid options for this are
generic, freebsd, netbsd, openbsd, windows, centos,
ubuntu, debian & alpine linux.
uefi Set this (any non-empty value) for guests that need
UEFI firmware.
cpu A numeric value specifying the number of virtual CPU
cores to assign to the guest.
memory The amount of memory to assign to the guest. This can
be specified in megabytes or gigabytes using the M and
G suffixes.
hostbridge This option allows you to specify the type of host‐
bridge used for the guest hardware. Normally you can
leave this as default, which is to use a standard
bhyve hostbridge.
There are two other options. amd, which is almost
identical to the standard hostbridge, but advertises
itself with a vendor ID of AMD. There are also some
special cases where you may require no hostbridge at
all, which can be achieved using the none value.
comports This option allows you to specify which com ports to
create for the guest. The default is to create a sin‐
gle com1 port. Valid values for this are com1 and
com2. You can also connect two com ports by specify‐
ing both, separated by a space.
utctime Set this option to yes if the guest RTC should keep
UTC time.
debug If this is set to yes, all output from the bhyve(8)
process will be written to ${vm_dir}/guest/bhyve.log.
This is useful for debugging purposes as it allows you
to see any error messages that are being produced by
bhyve(8) itself.
network0_type The emulation to use for the first network adapter.
This option can be unspecified if no guest networking
is required. The recommended value for this is
virtio-net. Additional network interfaces can be con‐
figured by adding additional networkX_type and
networkX_switch values, replacing X with the next
available integer.
network0_switch The virtual switch to connect interface 0 to. This
should correspond to a virtual switch created using
the vm switch create subcommand. If the virtual switch
is not found, an interface will still be assigned, but
not connected to any bridge.
Note that this field is no longer strictly required.
If you are using a custom device for the networking
that is already configured, you may not need the
interface connected to a virtual switch. See the
network0_device configuration option.
network0_device Normally vm-bhyve will create a tap(4) device at run-
time for each virtual network interface. This may be
an issue in more advanced configurations where you
want to pre-configure the networking manually in a way
unsupported by vm-bhyve. This option allows you to
instruct vm-bhyve to use an existing network device
for this virtual interface, rather than creating one
dynamically.
network0_mac This option allows you to specify a mac address to use
for this interface. If not provided, bhyve(8) will
generate a mac address.
disk0_type The emulation type for the first virtual disk. At
least one virtual disk is required. Valid options for
this are currently virtio-blk and ahci-hd. Additional
disks can be added by adding additional diskX_type and
diskX_name values, replacing X with the next available
integer.
disk0_name The filename for the first virtual disk. The first
disk is created automatically when provisioning a new
virtual machine. If additional disks are added manu‐
ally, the image will need to be created, usually done
using the truncate(1) or zfs(8) commands. Alterna‐
tively, you can use the vm add command, which will
create the disk image for you.
Normally disk images or zvols are stored directly
inside the guest. To use a disk image that is stored
anywhere else, you can specify the full path in this
option, and configure the device as custom
disk0_dev The type of device to use for the disk. If not speci‐
fied, this will default to file, and a sparse file,
located in the guest directory, will be used as the
disk image. Other options include zvol & sparse-zvol,
which will used a ZVOL as the disk image, created
directly under the guest dataset. Alternatively you
can specify custom, in which case diskX_name should be
the full path to the image file or device.
disk0_opts Any additional options to use for this disk device.
Multiple options can be specified, separated by a
comma. Please see the bhyve(8) man page for more
details on supported options.
uuid This option allows you to specify a fixed UUID for the
guests SMBIOS. Normally, the UUID is generated by
bhyve(8) based on the hostname and guest name. Because
this may change if guests are moved between systems,
the vm create command automatically assigns a UUID to
all newly created guests.
passthruX Specify a device to pass through to the guest. You
will need to reserve the device first so that is it
claimed by the ppt driver on boot.
Once the device is successfully reserved, you can add
it to the guest by adding passthruX="1/2/3" to the
guest configuration file, where X is an integer start‐
ing at 0, and 1/2/3 is the Base/Slot/Function of the
device. If you are passing through multiple functions
on the same device, make sure they are specified
together in the configuration file in the same
sequence as the original device.
Please see https://wiki.freebsd.org/bhyve/pci_passthru
for more details on how this works.
virt_random Set this option to yes if you want to create a
virtio-rnd device for this guest.
grub_commands Sometimes you may need to provide grub commands that
differ from those generated by vm-bhyve. In this
case, create a text file inside the guest's directory
containing the commands you need to run. Usually the
last entry in the file should be boot, followed by a
newline. The newline is important, otherwise the last
command may not actually be executed.
Once you have created the file, set this option to the
name of the file. From that point on, during a normal
boot vm-bhyve will use the commands from this file
rather than attempting to generate them automatically.
zfs_dataset_opts This allows you to specify one or more ZFS properties
to set on the dataset when a guest is created. Because
properties are assigned as the dataset is created,
this option is most useful when specified inside a
template. As a guest is created, all properties listed
in this option will be applied to the guest dataset.
Multiple properties can be specified, separated by a
space. Please note that spaces are not currently sup‐
ported in the property values.
zfs_zvol_opts Allows you to specify ZFS properties that should be
assigned to any ZVOLs that are created for a guest. As
with zfs_dataset_opts, this makes most sense when
entered into a template, as the properties can be
assigned while a guest is being created. Some ZVOL
options, such as volblocksize can only be set at cre‐
ation time.
Multiple properties can be specified, separated by a
space. For example, the following will configure the
ZVOL block size to 128k, and turn compression off.
zfs_zvol_opts="volblocksize=128k compress=off"
linux_kernel CentOS guests require the kernel to be loaded inside
the boot loader. This option is required for these
guests and specifies the kernel version number to
load. Hopefully this requirement will be relaxed as
bhyve(8) development continues.
SEE ALSObhyve(8), bhyveload(8), tap(4), bridge(4), vlan(4), nmdm(4), cu(1),
fetch(1), truncate(1), zfs(8)BUGS
Please report all bugs/issues/feature requests to the github project at
https://github.com/churchers/vm-bhyve
AUTHOR
Matt Churchyard ⟨churchers@gmail.com⟩
BSD Oct 27, 2015 BSD
