autopkgtest implementation and team practices

NAME

autopkgtest - Debian Perl Group autopkgtest implementation and team practices

INTRODUCTION

The autopkgtest package, which provides as-installed testing of Debian packages, is integrated with pkg-perl packages as of version 3.5.1 (September 2014). The integration is intended to allow for continuous integration testing at https://ci.debian.net/.

This document describes the implementation and best practices for the team.

IMPLEMENTATION

Declaring

Testsuite: autopkgtest-pkg-perl

in the source package section of debian/control will activate an implicit test control file that’s currently embedded in the autopkgtestpackage. This test control file has an indirection layer that pushes all the test logic into the pkg-perl-autopkgtest package, maintained inside the team. The entry point for the logic is the runner (https://anonscm.debian.org/cgit/pkg-perl/packages/pkg-perl-tools.git/tree/autopkgtest/scripts/runner) script, which runs tests in /usr/share/pkg-perl-autopkgtest/\*.ddirectories.

Thanks to the uniformity of CPAN distributions, this implicit control file works with most (currently about 75%) of the pkg-perl packages. The rest will need package specific tweaks, as detailed below.

An explicit test control file that corresponds to the implicit one described above is included in the pkg-perl-autopkgtest package in /usr/share/doc/pkg-perl-autopkgtest/examples and can be viewed at https://anonscm.debian.org/cgit/pkg-perl/packages/pkg-perl-tools.git/tree/autopkgtest/examples/default-tests-control.

The current tests activated by Testsuite: autopkgtest-pkg-perl are listed below. If some of them have expected failures, they can be skipped by listing the tests in debian/tests/pkg-perl/SKIP.

smoke

Source: https://anonscm.debian.org/cgit/pkg-perl/packages/pkg-perl-tools.git/tree/autopkgtest/scripts/build-deps.d/smoke

This test re-uses the build time test suite present in most CPAN distributions to check that changes in the package’s dependencies haven’t introduced regressions in the package functionality.

The package and its build dependencies are installed, the t/directory and test.pl files are copied into a temporary directory, and the build time test suite is run against the installed modules. The temporary directory is used to make sure that the installed modules get used instead of the source tree.

If the test suite needs files outside the t/ directory, those can be listed in the debian/tests/pkg-perl/smoke-files(since pkg-perl-autopkgtest version 0.26; previously called debian/tests/pkg-perl/test-files) file. Note that this overrides the default list of t/ and test.pl, so you normally need to include those too.

Some common test files are removed from the temporary directory; they currently include pod.t, pod-coverage.t, boilerplate.t, 04critic.t, and 97_meta.t. If the file debian/tests/pkg-perl/smoke-skip (since pkg-perl-autopkgtestversion 0.26; previously called debian/tests/pkg-perl/skip-smoke) exists, it is used as a list of files to be removed.

The environment variables AUTOMATED_TESTING and NONINTERACTIVE_TESTING are set. Since version 0.25, these can be overridden and others can be added in the configuration file debian/tests/pkg-perl/smoke-env (since pkg-perl-autopkgtest version 0.26; previously called debian/tests/pkg-perl/env-smoke). The file syntax is one line per variable setting, in the form KEY=VAL. Empty lines and comments are allowed.

The tests are run with the prove --merge option, which merges and synchronizes stderr and stdout from the test scripts. In some rare cases this option can break the test suite. It can be disabled by unsetting PKG_PERL_PROVE_ARGS in debian/tests/pkg-perl/smoke-env.

For the benefit of tests looking for something in lib/ or blib/, dummy module files are installed there and included in a dummy MANIFEST file. An empty MANIFEST.SKIP is also provided.

If the file debian/tests/pkg-perl/smoke-setup exists and is executable, it will be executed right before running the tests, with the working directory set to the package source and $TDIR set to the test running directory. The testing gets aborted if this program returns a nonzero value. Please use this facility responsibly and only as a last resort if the other hooks don’t work for you.

From version 0.28 onwards, if the file debian/tests/pkg-perl/smoke-cleanupexists and is executable, it will be executed when the smoke tests exit, with the working directory set to the package source and $TDIR set to the test running directory.

The smoke test by default runs prove on t/\.t*. In case a package should run other test scripts, debian/tests/pkg-perl/smoke-tests can be used since 0.29 to declare which files should be used instead. Example: t/\/\*.t*.

use.t

Source: https://anonscm.debian.org/cgit/pkg-perl/packages/pkg-perl-tools.git/tree/autopkgtest/scripts/runtime-deps.d/use.t

This test ensures that the runtime dependencies are sufficient for the most basic usage of the module. Warnings from such basic usage are considered failures because they can be quite annoying on consumers of the module.

The package and its runtime dependencies are installed, and the command perl -w -MFoo::Bar -e1 is executed, where Foo::Bar is the name of the main module in the package. If this exits with a nonzero status, or if there is output on stderr, the test is considered failed. Since pkg-perl-autopkgtest version 0.34, the test is repeated with PERL_DL_NONLAZY=1 set in the environment; this makes sure there are no unresolved symbols in binary modules.

The name of the module is read from META.json or META.yml, but it can be overridden in the debian/tests/pkg-perl/use-name(since pkg-perl-autopkgtest version 0.26; previously called debian/tests/pkg-perl/module-name. The test is skipped if the name isn’t present in one of the above sources.

There are some rare cases where warnings are to be expected even from the most basic usage. Since pkg-perl-autopkgtest version 0.34 such expected warnings can be listed as regular expressions in debian/tests/pkg-perl/use-whitelist.

syntax.t

Source: https://anonscm.debian.org/cgit/pkg-perl/packages/pkg-perl-tools.git/tree/autopkgtest/scripts/runtime-deps-and-recommends.d/syntax.t

This test checks that the runtime dependencies and recommendations cover all the requirements of the modules shipped in the package, and that the modules are syntactically valid Perl.

The test first installs the package and its runtime dependencies, including recommendations. All the .pm files in the package are then run through a syntax check with perl -wc file.pm. If the command exits with a nonzero status, the test is considered failed. Standard error output is not treated as a failure. If there are other files that you’d like to check, you can specify those as regular expressions in debian/tests/pkg-perl/syntax-extra (since pkg-perl-autopkgtestversion 0.34).

If some of the module files have expected problems passing this test, they can be skipped by listing them in debian/tests/pkg-perl/syntax-skip(since pkg-perl-autopkgtest version 0.26; previously called debian/tests/pkg-perl/skip-syntax).

Packages that Suggest other packages are exempt from this test, unless they include a skip-syntax file. This is because some of the module files are expected to need the suggested packages to work, but there is no mechanism to install them in the environment provided by autopkgtest.

ci.debian.net

Even with the implicit test control file, all the packages would need source uploads to include the Testsuite header, which would take quite a while to happen for all the team packages (currently numbering almost 3000). To speed up adoption, the current plan is to make https://ci.debian.net/ use a whitelist of known good pkg-perl packages that it will treat as having the Testsuite: autopkgtest-pkg-perl header. This whitelist is maintained at https://anonscm.debian.org/cgit/collab-maint/debian-ci-config.git and is based on test runs of all our packages such as https://lists.debian.org/debian-perl/2014/09/msg00100.html.

The whitelist is considered a temporary measure that can be dropped once we have Testsuite: autopkgtest-pkg-perl headers in all our packages.

As of May 2015, there are 2029 whitelisted packages, 378 packages that declare Testsuite: autopkgtest-pkg-perl, and 604 non-whitelisted pkg-perl packages that currently don’t pass the default checks and therefore need manual work.

The list of packages needing manual work is available at https://people.debian.org/~ntyni/pkg-perl/autopkgtest-fail.txt and current as of 20150524, but is not actively updated at the moment (volunteers welcome.)

Team practices

Once the set of known good packages has stabilized and is live on https://ci.debian.net/, we should do a mass commit in our git repository adding Testsuite: autopkgtest-pkg-perl headers to the known good packages.

As of pkg-perl-tools version 0.18, there is a team specific lintian check (activated by lintian --profile pkg-perl) triggering on the lack of the Testsuite header and pointing to this document. Coupled with the mass commit above, affected packages are expected to be those that need tweaking to get the tests working. Maintainers encountering the lintian warning are encouraged to fix those (other team members are happy to help with that!), but they are also naturally free to ignore the warning and postpone the addition of the header.

When uploading a package with a Testsuite header, uploaders should adopt a habit of verifying that the tests pass so as not to cause unnecessary work to the https://ci.debian.net/ volunteers.

Bugs related to autopkgtest failures should be filed with

User: debian-perl@lists.debian.org
Usertags: autopkgtest

so that they show up on the tracking list at https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=autopkgtest;users=debian-perl@lists.debian.org

See https://wiki.debian.org/bugs.debian.org/usertags for more information about usertags.

SETUP

Setting up a sbuild chroot for autopkgtest (formerly known as adt-run) is easy. Just follow the following steps.

chroot type

The simplest type of chroot is directory, which doesn’t need any configuration before it is been created.

In case you prefer to use lvm-snapshot instead of regular directories, you need to create a logical volume first. For instance, assuming that /dev/sdb1 is empty and available to be used as a physical volume:

sudo apt-get install lvm2
sudo pvcreate /dev/sdb1
sudo vgcreate sbuild /dev/sdb1
sudo lvcreate -L2G -n sid sbuild
sudo mkfs.ext4 /dev/mapper/sbuild-sid
sudo mkdir -p /srv/sbuild/sid
sudo mount /dev/mapper/sbuild-sid /srv/sbuild/sid

sbuild

Note that sbuild uses schroot underneath to manage the chroots.

* Create a new chroot

Note that installing recommends is needed so all the autopkgtest tools are available later.

sudo apt-get install --install-recommends sbuild autopkgtest
sudo sbuild-createchroot sid /srv/sbuild/sid http://httpredir.debian.org/debian

At this point you can run sudo umount /srv/sbuild/sid if you used LVM.

* Grant permission to use the chroot

sudo adduser {username} sbuild

* Setup directory chroot

First, you should rename the file created automatically on the previous step:

sudo mv sid-amd64-sbuild-* sid-amd64-sbuild.conf

Then, you have to setup the chroot properties for the directory type:

echo "union-type=aufs" | sudo tee -a /etc/schroot/chroot.d/sid-amd64-sbuild.conf

Or lvm-snapshot type replacing the contents with these:

[sid-amd64-sbuild]
type=lvm-snapshot
lvm-snapshot-options=--size 2G
device=/dev/mapper/sbuild-sid
groups=root,sbuild
root-groups=root,sbuild
profile=sbuild
description=Debian sid/amd64 autobuilder

* Update chroot source

Use this command to update the package list, perform a dist upgrade, and autoremove obsolete packages in the chroot.

sudo sbuild-update -udr sid

* Modify chroot source

In case you need to make other kind of changes to the chroot.

sudo schroot -c source:sid

* Instantiate chroot

Play with an ephemeral chroot based on the source chroot setup before (you do not need to be root at this point). Please, be warned that this chroot instance will be automatically removed when you exit the shell.

schroot -c sid

* Additional mounts

Sometimes you need to mount other filesystem or directories in the chroot (e.g. /home). If you miss anything, you can add that directory in the file /etc/schroot/sbuild/fstab, where sbuild is the profile for the chroots created by sbuild.

You can find several examples in the desktop and default profiles.

See Enrico Zini’s blog entry at https://www.enricozini.org/blog/2008/tips/joys-of-schroot/ and/or sbuild-createchroot(8) for details.

Examples

An example package using debian/tests/pkg-perl/SKIP: libtest-file-sharedir-perl

An example package using debian/tests/pkg-perl/skip-smoke: libtext-csv-xs-perl

https://anonscm.debian.org/cgit/pkg-perl/packages/libtext-csv-xs-perl.git/tree/debian/tests/pkg-perl/skip-smoke

(Since pkg-perl-autopkgtest version 0.26, the name debian/tests/pkg-perl/smoke-skip is preferred.)

An example package using debian/tests/pkg-perl/test-files: libnet-ldap-perl

https://anonscm.debian.org/cgit/pkg-perl/packages/libnet-ldap-perl.git/tree/debian/tests/pkg-perl/test-files

(Since pkg-perl-autopkgtest version 0.26, the name debian/tests/pkg-perl/smoke-files is preferred.)

An example package that uses debian/tests/pkg-perl/module-name: libnet-radius-perl

https://anonscm.debian.org/cgit/pkg-perl/packages/libnet-radius-perl.git/tree/debian/tests/pkg-perl/module-name

(Since pkg-perl-autopkgtest version 0.26, the name debian/tests/pkg-perl/use-name is preferred.)

An example package using debian/tests/pkg-perl/syntax-skip: libplack-perl

An example package using debian/tests/pkg-perl/env-smoke: libsvn-notify-mirror-perl

https://anonscm.debian.org/cgit/pkg-perl/packages/libsvn-notify-mirror-perl.git/tree/debian/tests/pkg-perl/env-smoke

(Since pkg-perl-autopkgtest version 0.26, the name debian/tests/pkg-perl/smoke-env is preferred.)

An example package disabling prove --merge in debian/tests/pkg-perl/smoke-env: libterm-progressbar-perl

An example package using debian/tests/pkg-perl/smoke-setup: libnet-sftp-sftpserver-perl

An example package using debian/tests/pkg-perl/smoke-cleanup: libmemcached-libmemcached-perl

Usage

Running autopkgtest against a package in the archive, assuming your schroot name is sid:

autopkgtest libfoo-bar-perl -- schroot sid # since autopkgtest 4.0
adt-run libfoo-bar-perl --- schroot sid

(Make sure that schroot source:sid has a deb-src entry in sources.list and that you have run apt-get updatethere at least once before trying this example.)

Running autopkgtest on a .changes file:

autopkgtest libfoo-bar-perl_0.01_amd64.changes -- schroot sid # since autopkgtest 4.0
adt-run libfoo-bar-perl_0.01_amd64.changes --- schroot sid

Running autopkgtest with the control information in the current directory but the actual package from the archive:

autopkgtest -B . -- schroot sid # since autopkgtest 4.0
adt-run -B .// --- schroot sid

(The source tree doesn’t need to be built, the binary package is installed from the archive with apt.)

See /usr/share/doc/autopkgtest/README.running-tests.rst.gz for more information.

AUTHORS

* Niko Tyni

* Alex Muntada

* gregor herrmann

LICENSE

Copyright (c) 2014-2016 by the individual authors and contributors noted above. All rights reserved. This document is free software; you may redistribute it and/or modify it under the same terms as Perl itself

Perl is distributed under your choice of the GNU General Public License or the Artistic License. On Debian GNU/Linux systems, the complete text of the GNU General Public License can be found in `/usr/share/common-licenses/GPL’ and the Artistic License in `/usr/share/common-licenses/Artistic’.