MOGTOOL(1) User Contributed Perl Documentation MOGTOOL(1)NAMEmogtool-- Inject/extract data to/from a MogileFS installation
WARNING: this utility is deprecated! See MogileFS::Utils
SYNOPSIS
$ mogtool [general-opts] <command> [command-opts] <command-args>
$ mogtool --trackers=127.0.0.1:6001 --domain=foo --class=bar ...
$ mogtool --conf=foo.conf ...
$ mogtool inject thefile.tgz thefilekey
$ mogtool inject --bigfile thebigfile.tgz thefilekey
$ mogtool inject --bigfile --gzip thebigfile.tar thefilekey
$ mogtool inject --bigfile --gzip mydirectory thedirkey
$ mogtool inject --bigfile --gzip /dev/hda4 thedevkey
$ mogtool inject --nobigfile bigcontiguousfile bigcfilekey
$ mogtool inject --bigfile --gzip --verify \
--description="Description" \
--receipt="foo@bar.com, baz@bar.com" \
--concurrent=5 --chunksize=32M \
somehugefile thefilekey
$ mogtool extract thefilekey thenewfile.tgz
$ mogtool extract thefilekey -
$ mogtool extract --bigfile thedirkey .
$ mogtool extract --bigfile --asfile thedirkey thefile.tgz
$ mogtool extract --bigfile thedevkey /dev/hda4
$ mogtool delete thekey
$ mogtool locate --noverify thekey
$ mogtool locate --bigfile thekey
$ mogtool list
$ mogtool listkey key
GENERAL OPTIONS--debug
Turn on MogileFS debug output.
--trackers=<[preferred_ip/]ip:port>[,<[preferred_ip/]ip:port>]*
Specify one or more trackers for your MogileFS installation. Note
that you can specify preferred IPs to override the default IPs
with. So it would look something like 10.10.0.1/10.0.0.1:8081.
--domain=<domain>
Set the MogileFS domain to use.
--class=<class>
Set the class within the domain to use. Defaults to _default.
--conf=<file>
Specify a configuration file to load from.
--lib=<directory>
Specify a directory to use as a library path. Right now, this
should be the directory where you expect to find the MogileFS.pm
file, if it's not actually installed.
COMMANDS
inject|i
Insert a resource into MogileFS. See "INJECT OPTIONS" and "INJECT
ARGUMENTS" for the rest of how to use the inject mode.
extract|x
Extract a resource from MogileFS. See "EXTRACT OPTIONS" and
"EXTRACT ARGUMENTS" for how to use extract.
delete|rm
Delete a resource. See "DELETE OPTIONS" and "DELETE ARGUMENTS".
locate|lo key
List the paths to the file identified by the given key.
list|ls
List all big files contained in MogileFS. No options, no
arguments.
listkey|lsk key
List all files which match the key. Key is just a prefix, and this
will list all keys which match the prefix. So if you specify key as
"ABC1" then you'll get all keys which start with the characters
"ABC1"
INJECT OPTIONS
The following options are used to control the behavior of the injector.
--bigfile|-b
If specified, use chunking to break the resource into manageable
pieces.
--chunksize=<size>[B|K|M|G]
When instructed to break files into chunks, the injector will use
the specified chunk size as the maximum chunk size. Defaults to
64M. You can specify the chunk size manually and specify the
units--defaults to bytes.
--gzip|-z
If specified, mogtool will gzip the data as it's going into
MogileFS. The resource will be marked as compressed.
Note that you do not need to specify this if the resource is
already gzipped, but it doesn't hurt. (We automatically detect
that and mark it as compressed.)
--overwrite
If you previously were working on injecting a big file as chunks
and the process died, normally mogtool refuses to do it again.
Specify this option to force the overwrite of that file.
NOTE: Other than in the above case (partial failure), mogtool will
not prompt before overwriting an existing file.
--verify
If on, we do a full MD5 verification of every chunk after it is
replicated. This can take a while on large files!
--description=<text>
Specifies a description for this file. Optional, but assists in
reporting and listing the large files in MogileFS. (This is also
displayed in any receipts that are created.)
--receipt=<email address>[, <email address>]*
If specified, emails a copy of the receipt file to the specified
comma-separated email addresses. Also creates a local filesystem
copy of the receipt file.
--concurrent=<number>
Specifies the number of concurrent processes to run for MogileFS
insertion. If you are noticing mogtool spend most of it's time
waiting for children and not actually buffering data, you may wish
to raise this number. The default is 1 but we've found 3 or 4 work
well.
INJECT ARGUMENTS
resource
What you actually want to inject. This can be a file, directory,
or a raw partition in the format /dev/X.
Please see "USAGE EXAMPLES" for more information on how to inject
these different types of resources and the differences thereof.
key Specifies the key to save this file to. For big files, the key is
actually "_big_N:key" and "key,#" where N is one of a bunch of
things we use and # is the chunk number.
Generally, you want this to be descriptive so you remember what it
is later and can identify the file just by looking at the key.
EXTRACT OPTIONS
--bigfile|-b
If specified, indicates that this resource was chunked on injection
and should be reassembled for extraction.
--gzip|-z
Specifies to mogtool that it should ungzip the output if and only
if it was compressed when inserted into the MogileFS system. So,
if you're extracting a file that wasn't gzipped to begin with, this
doesn't do anything.
--asfile
Useful when extracting something previously inserted as a
directory--this option instructs mogtool to treat the resource as a
file and not actually run it through tar for decompression.
EXTRACT ARGUMENTS
key Specifies the key to get the file from.
destination
What destination means varies depending on what type of resource
you're extracting. However, no matter what, you can specify a
single dash (-) to mean STDOUT.
Please see the usage examples for more information on how extract
works.
DELETE OPTIONS
--bigfile|-b
The resource is a "big file" and all chunks should be deleted.
DELETE ARGUMENTS
key Specifies the key of the file to delete.
LOCATE OPTIONS--verify
Verify that the returned paths actually contain the file. The
locate commands defaults to verify, you can disable it with
--noverify.
--bigfile|-b
The resource is a "big file" and the locations of the information
key should be printed.
LOCATE ARGUMENTS
key Specifies the key of the file to locate
RETURN VALUES
0 Success during operation.
1 During the locate, list, or listkey operation, the key was not
found.
2 Some fatal error occurred.
USAGE EXAMPLES
Please note that all examples assume you have a default config file
that contains the tracker and domain to use. Saves us from having to
clutter up the command line.
Small Files (<64MB)
When it comes to using small files, mogtool is very, very easy.
Injection
$ mogtool inject foo.dbm foo.dbm.2004.12
Injects the file foo.dbm into MogileFS under the key of
foo.dbm.2004.12.
$ mogtool inject --gzip foo.dbm foo.dbm.2004.12
Injects the same file to the same key, but compresses it on the fly for
you.
Extraction
$ mogtool extract foo.dbm.2004.12 newfoo.dbm
Retrieves the key foo.dbm.2004.12 and saves it as newfoo.dbm.
$ mogtool extract --gzip foo.dbm.2004.12 newfoo.dbm
Gets the file and automatically decompresses it, if and only if it was
compressed. So basically, you can turn on gzip in your config file and
mogtool will do the smart thing each time.
$ mogtool extract foo.dbm.2004.12 -
Print the resource to standard out. If you want, you can pipe it
somewhere or redirect to a file (but why not just specify the
filename?).
Large Files (>64MB)
Given mogtool's ability to break files into chunks and later reassemble
them, inserting large files (even files over the 4GB barrier) is
relatively easy.
Injection
$ mogtool inject --bigfile largefile.dat largefile.dat
As expected, inserts the file largefile.dat into the MogileFS system
under the name largefile.dat. Not very creative. Uses the default
64MB chunks.
$ mogtool inject --bigfile --chunksize=16M largefile.dat largefile.dat
Specify to use 16MB chunks instead of the default. Otherwise, the
same.
$ mogtool inject --bigfile --chunksize=1000K --gzip largefile.dat somekey
Do it again, but specify 1000KB chunks, gzip automatically, and upload
it under a different key somekey.
$ mogtool inject --bigfile --concurrent=5 --gzip largefile.dat somekey
Same as above, but use 5 children processes for uploading chunks to
MogileFS. This can take up to 300MB of memory in this example! (It
tends to use about (concurrency + 1) * chunksize bytes.)
$ mogtool inject --bigfile --chunksize=32M --concurrent=3 --gzip \
--receipt="foo@bar.com" --verify --description="A large file" \
largefile.dat somekey
Break this file into 128MB chunks, set a description, use 3 children to
upload them, gzip the file as you go, do a full MD5 verification of
every chunk, then email a receipt with all of the MogileFS paths to me.
Lots of flexibility with mogtool.
Extraction
$ mogtool extract --bigfile somekey newfile.dat
In its basic form, extracts the previously inserted large file and
saves it as newfile.dat.
$ mogtool extract --bigfile --gzip somekey newfile.dat
If the file was gzipped on entry, ungzip it and save the result. If it
wasn't gzipped, then we just save it.
Directories
Directories are easily injected and extracted with mogtool. To create
the data stream that is inserted into MogileFS, we use tar.
Injection
$ mogtool inject --bigfile mydir mykey
Run mydir through tar and then save it as mykey.
$ mogtool inject --bigfile --gzip --concurrent=5 mydir mykey
Inject, but also gzip and use multiple injectors.
Note how this is just like injecting a large file. See injection
examples for large files for more examples.
Extraction
$ mogtool extract --bigfile mykey .
Extract the previously injected directory mykey to your local
directory.
$ mogtool extract --bigfile --asfile mykey foo.tar
Take the previously generated tarball and save it as foo.tar. Simply
creates the file instead of extracting everything inside.
Partitions/Devices
mogtool has the ability to inject raw partitions into MogileFS and to
retrieve them later and write them back to a partition. They're
treated just like directories for the most part, we just don't pipe
things through tar.
Injection
$ mogtool inject --bigfile /dev/hda3 hda3.backup
Save a raw copy of your partition /dev/hda3 to the key hda3.backup.
$ mogtool inject --bigfile --gzip /dev/hda3 hda3.backup
Same, but compress on the fly during injection.
Extraction
$ mogtool extract --bigfile hda3.backup /dev/hda4
Extract the partition at hda3.backup to the partition /dev/hda4.
WARNING: mogtool won't ask for confirmation, make sure you don't
mistype partition numbers!
Deleting a Resource
WARNING: Please make sure you're specifying the right parameter, as
delete does not prompt for confirmation of the request!
$ mogtool delete thekey
Delete a normal file.
$ mogtool delete --bigfile thekey
Delete a chunked file--this deletes all chunks and the receipt, so the
file is gone.
Listing Big Files
$ mogtool list backup
Lists all large files stored in MogileFS. It is not possible to list
all normal files at this time.
Listing Files Matching a Key
$ mogtool listkey abc1
Lists all files in MogileFS whose keys start with the characters
"abc1".
CONFIGURATION FILE
Instead of adding a ton of options to the command line every time,
mogtool enables you to create a default configuration file that it will
read all of the options from. It searches two locations for a default
configuration file: ~/.mogtool and /etc/mogilefs/mogtool.conf.
(Alternately, you can specify --conf=whatever as an option on the
command line.)
The file can consist of any number of the following items:
trackers = 10.0.0.3:7001, 10.10.0.5/10.0.0.5:7001
domain = mogiledomain
class = fileclass
lib = /home/foo/lib
gzip = 1
big = 1
overwrite = 1
chunksize = 32M
receipt = foo@bar.com, baz@bar.com
verify = 1
concurrent = 3
KNOWN BUGS
None? Send me any you find! :)
PLANNED FEATURES--concurrent for extract
It would be nice to have concurrent extraction going on.
recover mode
If the receipt file is ever corrupt in MogileFS it would be useful
to recover a file given just a receipt. It would have the same
arguments as the extract mode, except use a receipt file as the
data source.
partition size verification
We can easily get the partition size when we save one to MogileFS,
so we should use that information to determine during extraction if
a target partition is going to be big enough.
on the fly gzip extraction
Right now we can gzip on an injection, but we should support doing
decompression on the fly coming out of MogileFS.
make list take a prefix
If you can specify a prefix, that makes things easier for finding
small files that are stored in MogileFS.
more information on list
Have list load up the info file and parse it for information about
each of the big files being stored. Maybe have this as an option
(-l). (This means the reading and parsing of info files should be
abstracted into a function.)
AUTHOR
Mark Smith <junior@danga.com> - most of the implementation and
maintenance.
Brad Fitzpatrick <brad@danga.com> - concepts and rough draft.
Robin H. Johnson <robbat2@orbis-terrarum.net> - locate function.
Copyright (c) 2002-2004 Danga Interactive. All rights reserved.
perl v5.14.1 2011-01-05 MOGTOOL(1)
