autopkgtest implementation and team practices


autopkgtest - Debian Perl Group autopkgtest implementation and team practices


The autopkgtest package, which provides as-installed testing of Debian packages, is integrated with pkg-perl packages as of version 3.5.1 (September 2014). The integration is intended to allow for continuous integration testing at

This document describes the implementation and best practices for the team.



Testsuite: autopkgtest-pkg-perl

in the source package section of debian/control will activate an implicit test control file that’s currently embedded in the autopkgtestpackage. This test control file has an indirection layer that pushes all the test logic into the pkg-perl-autopkgtest package, maintained inside the team. The entry point for the logic is the runner ( script, which runs tests in /usr/share/pkg-perl-autopkgtest/\*.ddirectories.

Thanks to the uniformity of CPAN distributions, this implicit control file works with most (currently about 75%) of the pkg-perl packages. The rest will need package specific tweaks, as detailed below.

An explicit test control file that corresponds to the implicit one described above is included in the pkg-perl-autopkgtest package in /usr/share/doc/pkg-perl-autopkgtest/examples and can be viewed at

The current tests activated by Testsuite: autopkgtest-pkg-perl are listed below. If some of them have expected failures, they can be skipped by listing the tests in debian/tests/pkg-perl/SKIP.



This test re-uses the build time test suite present in most CPAN distributions to check that changes in the package’s dependencies haven’t introduced regressions in the package functionality.

The package and its build dependencies are installed, the t/directory and files are copied into a temporary directory, and the build time test suite is run against the installed modules. The temporary directory is used to make sure that the installed modules get used instead of the source tree.

If the test suite needs files outside the t/ directory, those can be listed in the debian/tests/pkg-perl/smoke-files(since pkg-perl-autopkgtest version 0.26; previously called debian/tests/pkg-perl/test-files) file. Note that this overrides the default list of t/ and, so you normally need to include those too.

Some common test files are removed from the temporary directory; they currently include pod.t, pod-coverage.t, boilerplate.t, 04critic.t, and 97_meta.t. If the file debian/tests/pkg-perl/smoke-skip (since pkg-perl-autopkgtestversion 0.26; previously called debian/tests/pkg-perl/skip-smoke) exists, it is used as a list of files to be removed.

The environment variables AUTOMATED_TESTING and NONINTERACTIVE_TESTING are set. Since version 0.25, these can be overridden and others can be added in the configuration file debian/tests/pkg-perl/smoke-env (since pkg-perl-autopkgtest version 0.26; previously called debian/tests/pkg-perl/env-smoke). The file syntax is one line per variable setting, in the form KEY=VAL. Empty lines and comments are allowed.

The tests are run with the prove --merge option, which merges and synchronizes stderr and stdout from the test scripts. In some rare cases this option can break the test suite. It can be disabled by unsetting PKG_PERL_PROVE_ARGS in debian/tests/pkg-perl/smoke-env.

For the benefit of tests looking for something in lib/ or blib/, dummy module files are installed there and included in a dummy MANIFEST file. An empty MANIFEST.SKIP is also provided.

If the file debian/tests/pkg-perl/smoke-setup exists and is executable, it will be executed right before running the tests, with the working directory set to the package source and $TDIR set to the test running directory. The testing gets aborted if this program returns a nonzero value. Please use this facility responsibly and only as a last resort if the other hooks don’t work for you.

From version 0.28 onwards, if the file debian/tests/pkg-perl/smoke-cleanupexists and is executable, it will be executed when the smoke tests exit, with the working directory set to the package source and $TDIR set to the test running directory.

The smoke test by default runs prove on t/\.t*. In case a package should run other test scripts, debian/tests/pkg-perl/smoke-tests can be used since 0.29 to declare which files should be used instead. Example: t/\/\*.t*.



This test ensures that the runtime dependencies are sufficient for the most basic usage of the module. Warnings from such basic usage are considered failures because they can be quite annoying on consumers of the module.

The package and its runtime dependencies are installed, and the command perl -w -MFoo::Bar -e1 is executed, where Foo::Bar is the name of the main module in the package. If this exits with a nonzero status, or if there is output on stderr, the test is considered failed. Since pkg-perl-autopkgtest version 0.34, the test is repeated with PERL_DL_NONLAZY=1 set in the environment; this makes sure there are no unresolved symbols in binary modules.

The name of the module is read from META.json or META.yml, but it can be overridden in the debian/tests/pkg-perl/use-name(since pkg-perl-autopkgtest version 0.26; previously called debian/tests/pkg-perl/module-name. The test is skipped if the name isn’t present in one of the above sources.

There are some rare cases where warnings are to be expected even from the most basic usage. Since pkg-perl-autopkgtest version 0.34 such expected warnings can be listed as regular expressions in debian/tests/pkg-perl/use-whitelist.



This test checks that the runtime dependencies and recommendations cover all the requirements of the modules shipped in the package, and that the modules are syntactically valid Perl.

The test first installs the package and its runtime dependencies, including recommendations. All the .pm files in the package are then run through a syntax check with perl -wc If the command exits with a nonzero status, the test is considered failed. Standard error output is not treated as a failure. If there are other files that you’d like to check, you can specify those as regular expressions in debian/tests/pkg-perl/syntax-extra (since pkg-perl-autopkgtestversion 0.34).

If some of the module files have expected problems passing this test, they can be skipped by listing them in debian/tests/pkg-perl/syntax-skip(since pkg-perl-autopkgtest version 0.26; previously called debian/tests/pkg-perl/skip-syntax).

Packages that Suggest other packages are exempt from this test, unless they include a skip-syntax file. This is because some of the module files are expected to need the suggested packages to work, but there is no mechanism to install them in the environment provided by autopkgtest.

Even with the implicit test control file, all the packages would need source uploads to include the Testsuite header, which would take quite a while to happen for all the team packages (currently numbering almost 3000). To speed up adoption, the current plan is to make use a whitelist of known good pkg-perl packages that it will treat as having the Testsuite: autopkgtest-pkg-perl header. This whitelist is maintained at and is based on test runs of all our packages such as

The whitelist is considered a temporary measure that can be dropped once we have Testsuite: autopkgtest-pkg-perl headers in all our packages.

As of May 2015, there are 2029 whitelisted packages, 378 packages that declare Testsuite: autopkgtest-pkg-perl, and 604 non-whitelisted pkg-perl packages that currently don’t pass the default checks and therefore need manual work.

The list of packages needing manual work is available at and current as of 20150524, but is not actively updated at the moment (volunteers welcome.)

Team practices

Once the set of known good packages has stabilized and is live on, we should do a mass commit in our git repository adding Testsuite: autopkgtest-pkg-perl headers to the known good packages.

As of pkg-perl-tools version 0.18, there is a team specific lintian check (activated by lintian --profile pkg-perl) triggering on the lack of the Testsuite header and pointing to this document. Coupled with the mass commit above, affected packages are expected to be those that need tweaking to get the tests working. Maintainers encountering the lintian warning are encouraged to fix those (other team members are happy to help with that!), but they are also naturally free to ignore the warning and postpone the addition of the header.

When uploading a package with a Testsuite header, uploaders should adopt a habit of verifying that the tests pass so as not to cause unnecessary work to the volunteers.

Bugs related to autopkgtest failures should be filed with

Usertags: autopkgtest

so that they show up on the tracking list at;

See for more information about usertags.


Setting up a sbuild chroot for autopkgtest (formerly known as adt-run) is easy. Just follow the following steps.

chroot type

You need to setup a chroot type that supports ephemeral sessions for autopkgtest to run properly. It will fail otherwise.

One option is to use lvm-snapshot, so you need to create a logical volume and mount it first. For instance, assuming that /dev/sdb1 is empty and available to be used as a physical volume:

sudo apt-get install lvm2
sudo pvcreate /dev/sdb1
sudo vgcreate sbuild /dev/sdb1
sudo lvcreate -L2G -n sid sbuild
sudo mkfs.ext4 /dev/mapper/sbuild-sid
sudo mkdir -p /srv/sbuild/sid
sudo mount /dev/mapper/sbuild-sid /srv/sbuild/sid

A simpler alternative would be to use directory type, which doesn’t need any previous setup before it is created, just a regular directory (ephemeral sessions will be setup through union filesystem).


Note that sbuild uses schroot underneath to manage the chroots.

* Create a new chroot

Note that installing recommends is needed so all the autopkgtest tools are available later.

sudo apt-get install --install-recommends sbuild autopkgtest
sudo sbuild-createchroot sid /srv/sbuild/sid

At this point you can run sudo umount /srv/sbuild/sid if you used LVM.

* Grant permission to use the chroot

sudo adduser {username} sbuild

* Setup directory chroot

First, you should rename the file created automatically on the previous step (it’s not mandatory but useful):

sudo mv /etc/schroot/chroot.d/sid-amd64-sbuild-* /etc/schroot/chroot.d/sid-amd64-sbuild.conf

Then, you have to setup the chroot properties for the directory type to make sure ephemeral sessions can be created:

sudo sed -i -e 's,^union-type=none,union-type=aufs,' /etc/schroot/chroot.d/sid-amd64-sbuild.conf

Or lvm-snapshot type replacing the contents with these:

lvm-snapshot-options=--size 2G
description=Debian sid/amd64 autobuilder

* Update chroot source

Use this command to update the package list, perform a dist upgrade, and autoremove obsolete packages in the chroot.

sudo sbuild-update -udr sid-amd64-sbuild

* Modify chroot source

In case you need to make other kind of changes to the chroot.

sudo schroot -c source:sid-amd64-sbuild

* Instantiate chroot

Play with an ephemeral chroot based on the source chroot setup before (you do not need to be root at this point). Please, be warned that this chroot instance will be automatically removed when you exit the shell.

schroot -c sid-amd64-sbuild

* Additional mounts

Sometimes you need to mount other filesystem or directories in the chroot (e.g. /home). If you miss anything, you can add that directory in the file /etc/schroot/sbuild/fstab, where sbuild is the profile for the chroots created by sbuild.

You can find several examples in the desktop and default profiles.

See Enrico Zini’s blog entry at and/or sbuild-createchroot(8) for details.


You can use eatmydata do improve the speed of your builds inside the chroot pretty easily.

* Make sure eatmydata is installed in the chroot

You can either install it manually in the source chroot or add --include=eatmydata option before running sbuild-createchrootabove.

* Enable eatmydata in the chroot configuration

Add command-prefix=eatmydata to /etc/schroot/chroot.d/sid-amd64-sbuild.conf.

See for more details.


An example package using debian/tests/pkg-perl/SKIP: libtest-file-sharedir-perl

An example package using debian/tests/pkg-perl/skip-smoke: libtext-csv-xs-perl

(Since pkg-perl-autopkgtest version 0.26, the name debian/tests/pkg-perl/smoke-skip is preferred.)

An example package using debian/tests/pkg-perl/test-files: libnet-ldap-perl

(Since pkg-perl-autopkgtest version 0.26, the name debian/tests/pkg-perl/smoke-files is preferred.)

An example package that uses debian/tests/pkg-perl/module-name: libnet-radius-perl

(Since pkg-perl-autopkgtest version 0.26, the name debian/tests/pkg-perl/use-name is preferred.)

An example package using debian/tests/pkg-perl/syntax-skip: libplack-perl

An example package using debian/tests/pkg-perl/env-smoke: libsvn-notify-mirror-perl

(Since pkg-perl-autopkgtest version 0.26, the name debian/tests/pkg-perl/smoke-env is preferred.)

An example package disabling prove --merge in debian/tests/pkg-perl/smoke-env: libterm-progressbar-perl

An example package using debian/tests/pkg-perl/smoke-setup: libnet-sftp-sftpserver-perl

An example package using debian/tests/pkg-perl/smoke-cleanup: libmemcached-libmemcached-perl


Running autopkgtest against a package in the archive, assuming your schroot name is sid-amd64-sbuild:

autopkgtest libfoo-bar-perl -- schroot sid-amd64-sbuild # since autopkgtest 4.0
adt-run libfoo-bar-perl --- schroot sid-amd64-sbuild

(Make sure that schroot source:sid-amd64-sbuild has a deb-src entry in sources.list and that you have run apt-get updatethere at least once before trying this example.)

Running autopkgtest on a .changes file:

autopkgtest libfoo-bar-perl_0.01_amd64.changes -- schroot sid-amd64-sbuild # since autopkgtest 4.0
adt-run libfoo-bar-perl_0.01_amd64.changes --- schroot sid-amd64-sbuild

Running autopkgtest with the control information in the current directory but the actual package from the archive:

autopkgtest -B . -- schroot sid-amd64-sbuild # since autopkgtest 4.0
adt-run -B .// --- schroot sid-amd64-sbuild

(The source tree doesn’t need to be built, the binary package is installed from the archive with apt.)

See /usr/share/doc/autopkgtest/README.running-tests.rst.gz for more information.


* Niko Tyni

* Alex Muntada

* gregor herrmann


Copyright (c) 2014-2017 by the individual authors and contributors noted above. All rights reserved. This document is free software; you may redistribute it and/or modify it under the same terms as Perl itself

Perl is distributed under your choice of the GNU General Public License or the Artistic License. On Debian GNU/Linux systems, the complete text of the GNU General Public License can be found in `/usr/share/common-licenses/GPL’ and the Artistic License in `/usr/share/common-licenses/Artistic’.