Test::Aggregate(3) User Contributed Perl Documentation Test::Aggregate(3)NAMETest::Aggregate - Aggregate "*.t" tests to make them run faster.
VERSION
Version 0.364
SYNOPSIS
use Test::Aggregate;
my $tests = Test::Aggregate->new( {
dirs => $aggregate_test_dir,
} );
$tests->run;
ok $some_data, 'Test::Aggregate also re-exports Test::More functions';
DESCRIPTION
WARNING: this is ALPHA code. The interface is not guaranteed to be
stable. Further, check out Test::Aggregate::Nested (included with this
distribution). It's a more robust implemenation which does not have
the same limitations as "Test::Aggregate".
A common problem with many test suites is that they can take a long
time to run. The longer they run, the less likely you are to run the
tests. This module borrows a trick from "Apache::Registry" to load up
your tests at once, create a separate package for each test and wraps
each package in a method named "run_the_tests". This allows us to load
perl only once and related modules only once. If you have modules
which are expensive to load, this can dramatically speed up a test
suite.
DEPRECATION
For a whole variety of reasons, tests run in BEGIN/CHECK/INIT/INIT
blocks are now deprecated. They cause all sorts of test sequence
headaches. Plus, they break the up-coming nested TAP work. You will
have a problem if you use this common idiom:
BEGIN {
use_ok 'My::Module' or die;
}
Instead, just "use" the module and put the "use_ok" tests in a t/load.t
file or something similar and don't aggregate it. See the following
for more information: <http://use.perl.org/~Ovid/journal/38974>.
USAGE
Create a separate directory for your tests. This should not be a
subdirectory of your regular test directory. Write a small driver
program and put it in your regular test directory ("t/" is the
standard):
use Test::Aggregate;
my $other_test_dir = 'aggregate_tests';
my $tests = Test::Aggregate->new( {
dirs => $other_test_dir
});
$tests->run;
ok $some_data, 'Test::Aggregate also re-exports Test::More functions';
Take your simplest tests and move them, one by one, into the new test
directory and keep running the "Test::Aggregate" program. You'll find
some tests will not run in a shared environment like this. You can
either fix the tests or simply leave them in your regular test
directory. See how this distribution's tests are organized for an
example.
Note that "Test::Aggregate" also exports all exported functions from
"Test::More", allowing you to run other tests after the aggregated
tests have run.
use Test::Aggregate;
my $other_test_dir = 'aggregate_tests';
my $tests = Test::Aggregate->new( {
dirs => $other_test_dir
});
$tests->run;
ok !(-f 't/data/tmp.txt'), '... and our temp file should be deleted';
Some tests cannot run in an aggregate environment. These may include
test for this with the $ENV{TEST_AGGREGATE} variable:
package Some::Package;
BEGIN {
die __PACKAGE__ ." cannot run in aggregated tests"
if $ENV{TEST_AGGREGATE};
}
METHODS
"new"
my $tests = Test::Aggregate->new(
{
dirs => 'aggtests',
verbose => 1, # optional, but recommended
dump => 'dump.t', # optional
shuffle => 1, # optional
matching => qr/customer/, # optional
set_filenames => 0, # optional and not recommended
tidy => 1, # optional and experimental
test_nowarnings => 0, # optional and experimental
}
);
Creates a new "Test::Aggregate" instance. Accepts a hashref with the
following keys:
· "dirs" (either this or "tests" is mandatory)
The directories to look in for the aggregated tests. This may be a
scalar value of a single directory or an array refernce of multiple
directories.
· "tests" (either this or "dirs" is mandatory)
Instead of providing directories for the aggregated tests, you may
supply an array reference with a list of tests to aggregate. If
both are supplied, these tests will be appended to the list of
tests found in "dirs".
The "matching" parameter does not apply to test files identified
with this key.
· "verbose" (optional, but strongly recommended)
If set with a true value, each test programs success or failure
will be indicated with a diagnostic output. The output below means
that "aggtests/slow_load.t" was an aggregated test which failed.
This means it's much easier to determine which aggregated tests are
causing problems.
t/aggregate.........2/?
# ok - aggtests/boilerplate.t
# ok - aggtests/00-load.t
# not ok - aggtests/subs.t
# ok - aggtests/slow_load.t
t/aggregate.........ok
t/pod-coverage......ok
t/pod...............ok
Note that three possible values are allowed for "verbose":
· 0 (default)
No individual test program success or failure will be
displayed.
· 1
Only failing test programs will have their failure status
shown.
· 2
All test programs will have their success/failure shown.
· "dump" (optional)
You may list the name of a file to dump the aggregated tests to.
This is useful if you have test failures and need to debug why the
tests failed.
· "shuffle" (optional)
Ordinarily, the tests are sorted by name and run in that order.
This allows you to run them in any order.
· "matching" (optional)
If supplied with a regular expression (requires the "qr" operator),
will only run tests whose filename matches the regular expression.
· "set_filenames" (optional)
If supplied with a true value, this will cause the following to be
added for each test:
local $0 = $test_filename;
This is the default behavior.
· "findbin" (optional)
If supplied with a true value, this will cause FindBin::again() to
be called before each test file.
This is turned off by default.
Note that older versions of FindBin (pre 1.47) sometimes get
confused about where the bin directory is when I set $0. I don't
know why, but this is a rarely used option and only happens pre 5.8
perl, so I'm not too worried about it. Just keep it in mind.
· "dry" (optional)
Just print the tests which will be run and the order they will be
run in (obviously the order will be random if "shuffle" is true).
· "tidy"
If supplied a true value, attempts to run "Perl::Tidy" on the
source code. This is a no-op if "Perl::Tidy" cannot be loaded.
This option is "experimental". Plus, if your tests are terribly
convoluted, this could be slow and possibly buggy.
If the value of this argument is the name of a file, assumes that
this file is a ".perltidyrc" file.
· "test_nowarnings"
Disables "Test::NoWarnings" (fails if the module cannot be loaded).
This is experimental and somewhat problematic. Let me know if
there are any problems.
"run"
$tests->run;
Attempts to aggregate and run all tests listed in the directories
specified in the constructor.
SETUP/TEARDOWN
Since "BEGIN" and "END" blocks are for the entire aggregated tests and
not for each test program (see "CAVEATS"), you might find that you need
to have setup/teardown functions for tests. These are useful if you
need to setup connections to test databases, clear out temp files, or
any of a variety of tasks that your test suite might require. Here's a
somewhat useless example, pulled from our tests:
#!/usr/bin/perl
use strict;
use warnings;
use lib 'lib', 't/lib';
use Test::Aggregate;
use Test::More;
my $dump = 'dump.t';
my ( $startup, $shutdown ) = ( 0, 0 );
my ( $setup, $teardown ) = ( 0, 0 );
my $tests = Test::Aggregate->new(
{
dirs => 'aggtests',
dump => $dump,
startup => sub { $startup++ },
shutdown => sub { $shutdown++ },
setup => sub { $setup++ },
teardown => sub { $teardown++ },
}
);
$tests->run;
is $startup, 1, 'Startup should be called once';
is $shutdown, 1, '... as should shutdown';
is $setup, 4, 'Setup should be called once for each test program';
is $teardown, 4, '... as should teardown';
Note that you can still dump these to a dump file. This will only work
if "Data::Dump::Streamer" 1.11 or later is installed.
There are four attributes which can be passed to the constructor, each
of which expects a code reference:
· "startup"
startup => \&connect_to_database,
This function will be called before any of the tests are run. It
is not run in a BEGIN block.
· "shutdown"
shutdown => \&clean_up_temp_files,
This function will be called after all of the tests are run. It
will not be called in an END block.
· "setup"
setup => sub {
my $filename = shift;
# this gets run before each test program.
},
The setup function will be run before every test program. The name
of the test file will be passed as the first argument.
· "teardown"
teardown => sub {
my $filename = shift;
# this gets run after every test program.
}
The teardown function gets run after every test program. The name
of the test file will be passed as the first argument.
GLOBAL VARIABLES
You shouldn't be using global variables and a dependence on them can
break your code. However, Perl provides quite a few handy global
variables which, unfortunately, can easily break your tests if you
change them in one test and another assumes an unchanged value. As a
result, we localize many of Perl's most common global variables for
you, using the following syntax:
local %ENV = %ENV;
The following global variables are localized for you. Any others must
be localized manually per test.
· @INC
· %ENV
· %SIG
· $/
· $_
· $|
CAVEATS
Not all tests can be included with this technique. If you have
"Test::Class" tests, there is no need to run them with this.
Otherwise:
· "exit"
Don't call exit() in your aggregated tests. We now warn very
verbosely if this is done, but we still exit on the assumption that
further tests cannot run.
· "__END__" and "__DATA__" tokens.
These won't work and the tests will call BAIL_OUT() if these tokens
are seen. However, this limitation does not apply to
Test::Aggregate::Nested.
· "BEGIN" and "END" blocks.
Since all of the tests are aggregated together, "BEGIN" and "END"
blocks will be for the scope of the entire set of aggregated tests.
If you need setup/teardown facilities, see "TEARDOWN" in SETUP.
· Syntax errors
Any syntax errors encountered will cause this program to
BAIL_OUT(). This is why it's recommended that you move your tests
into your new directory one at a time: it makes it easier to
figure out which one has caused the problem.
· "no_plan"
Unfortunately, due to how this works, the plan is always "no_plan".
<http://groups.google.com/group/perl.qa/browse_thread/thread/d58c49db734844f4/cd18996391acc601?#cd18996391acc601>
for more information.
· "Test::NoWarnings"
Great module. It loves to break aggregated tests since some might
have warnings when others will not. You can disable it like this:
my $tests = Test::Aggregate->new(
dirs => 'aggtests/',
startup => sub { $INC{'Test/NoWarnings.pm'} = 1 },
);
As an alternative, you can also disable it with:
my $tests = Test::Aggregate->new({
dirs => 'aggtests',
test_nowarnings => 0,
});
We do work internally to subtract the extra test added by
"Test::NoWarnings". It's painful and experimental. Good luck.
· "Variable "$x" will not stay shared at (eval ..."
Because each test is wrapped in a method call, any of your subs
which access a variable in an outer scope will likely throw the
above warning. Pass in arguments explicitly to suppress this.
Instead of:
my $x = 17;
sub foo {
my $y = shift;
return $y + $x;
}
Write this:
my $x = 17;
sub foo {
my ( $y, $x ) = @_;
return $y + $x;
}
However, consider Test::Aggregate::Nested. This warning does not
apply with that module.
· Singletons
Be very careful of code which loads singletons. Oftimes those
singletons in test suites may be altered for testing purposes, but
later attempts to use those singletons can fail dramatically as
they're not expecting the alterations. (Your author has painfully
learned this lesson with database connections).
DEBUGGING AGGREGATE TESTS
Before aggregating tests, make sure that you add tests one at a time to
the aggregated test directory. Attempting to add many tests to the
directory at once and then experiencing a failure means it will be much
harder to track down which tests caused the failure.
Debugging aggregated tests which fail is a multi-step process. Let's
say the following fails:
my $tests = Test::Aggregate->new(
{
dump => 'dump.t',
shuffle => 1,
dirs => 'aggtests',
}
);
$tests->run;
Manually run the tests
The first step is to manually run all of the tests in the "aggtests"
dir.
prove -r aggtests/
If the failures appear the same, fix them just like you would fix any
other test failure and then rerun the "Test::Aggregate" code.
Sometimes this means that a different number of tests run from what the
aggregted tests run. Look for code which ends the program prematurely,
such as an exception or an "exit" statement.
Run a dump file
If this does not fix your problem, create a dump file by passing "dump
=> $dumpfile" to the constructor (as in the above example). Then try
running this dumpfile directly to attempt to replicate the error:
prove -r $dumpfile
Tweaking the dump file
Assuming the error has been replicated, open up the dump file. The
beginning of the dump file will have some code which overrides some
"Test::Builder" internals. After that, you'll see the code which runs
the tests. It will look similar to this:
if ( __FILE__ eq 'dump.t' ) {
Test::More::diag("******** running tests for aggtests/boilerplate.t ********")
if $ENV{TEST_VERBOSE};
aggtestsboilerplatet->run_the_tests;
Test::More::diag("******** running tests for aggtests/subs.t ********")
if $ENV{TEST_VERBOSE};
aggtestssubst->run_the_tests;
Test::More::diag("******** running tests for aggtests/00-load.t ********")
if $ENV{TEST_VERBOSE};
aggtests00loadt->run_the_tests;
Test::More::diag("******** running tests for aggtests/slow_load.t ********")
if $ENV{TEST_VERBOSE};
aggtestsslow_loadt->run_the_tests;
}
You can try to narrow down the problem by commenting out all of the
"run_the_tests" lines and gradually reintroducing them until you can
figure out which one is actually causing the failure.
COMMON PITFALLS
My Tests Threw an Exception But Passed Anyway!
This really isn't a "Test::Aggregate" problem so much as a general Perl
problem. For each test file, "Test::Aggregate" wraps the tests in an
eval and checks "my $error = $@". Unfortunately, we sometimes get code
like this:
$server->ip_address('apple');
And internally, the 'Server' class throws an exception but uses its own
evals in a "DESTROY" block (or something similar) to trap it. If the
code you call uses an eval but fails to localize it, it wipes out your
eval. Neat, eh? Thus, you never get a chance to see the error. For
various reasons, this tends to impact "Test::Aggregate" when a
"DESTROY" block is triggered and calls code which internally uses eval
(e.g., "DBIx::Class"). You can often fix this with:
DESTROY {
local $@ = $@; # localize but preserve the value
my $self = shift;
# do whatever you want
}
"BEGIN" and "END" blocks
Remember that since the tests are now being run at once, these blocks
will no longer run on a per-test basis, but will run for the entire
aggregated set of tests. You may need to examine these individually to
determine the problem.
"CHECK" and "INIT" blocks.
Sorry, but you can't use these (just as in modperl). See perlmod for
more information about them and why they won't work.
"Test::NoWarnings"
This is a great test module. When aggregating tests together, however,
it can cause pain as you'll often discover warnings that you never new
existed. For a quick fix, add this before you attempt to run your
tests:
$INC{'Test/NoWarnings.pm'} = 1;
That will disable "Test::NoWarnings", but you'll want to go in later to
fix them.
Paths
Many tests make assumptions about the paths to files and moving them
into a new test directory can break this.
$0
Tests which use $0 can be problematic as the code is run in an "eval"
through "Test::Aggregate" and $0 may not match expectations. This also
means that it can behave differently if run directly from a dump file.
As it turns out, you can assign to $0! We do this by default and set
the $0 to the correct filename. If you don't want this behavior, pass
"set_filenames => 0" to the constructor.
Minimal test case
If you cannot solve the problem, feel free to try and create a minimal
test case and send it to me (assuming it's something I can run).
AUTHOR
Curtis Poe, "<ovid at cpan.org>"
BUGS
Please report any bugs or feature requests to "bug-test-aggregate at
rt.cpan.org", or through the web interface at
http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Test-Aggregate
<http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Test-Aggregate>. I
will be notified, and then you'll automatically be notified of progress
on your bug as I make changes.
SUPPORT
You can find documentation for this module with the perldoc command.
perldoc Test::Aggregate
You can also look for information at:
· AnnoCPAN: Annotated CPAN documentation
http://annocpan.org/dist/Test-Aggregate
<http://annocpan.org/dist/Test-Aggregate>
· CPAN Ratings
http://cpanratings.perl.org/d/Test-Aggregate
<http://cpanratings.perl.org/d/Test-Aggregate>
· RT: CPAN's request tracker
http://rt.cpan.org/NoAuth/Bugs.html?Dist=Test-Aggregate
<http://rt.cpan.org/NoAuth/Bugs.html?Dist=Test-Aggregate>
· Search CPAN
http://search.cpan.org/dist/Test-Aggregate
<http://search.cpan.org/dist/Test-Aggregate>
ACKNOWLEDGEMENTS
Many thanks to mauzo (<http://use.perl.org/~mauzo/> for helping me find
the 'skip_all' bug.
Thanks to Johan LindstrA~Xm for pointing me to Apache::Registry.
COPYRIGHT & LICENSE
Copyright 2007 Curtis "Ovid" Poe, all rights reserved.
This program is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.
perl v5.14.1 2010-12-04 Test::Aggregate(3)
