Tests Before a Yum Release

Each of these should be run on any release before it happens:

yum install somepkg
yum install something_that_doesn't_exist
yum -y install something | tee >/tmp/foobar
yum remove
yum localinstall
yum list updates
yum update
yum groupinstall
yum groupremove
yum grouplist
yum info
yum groupinfo
yum makecache
yum clean all
yum -d0 -e0 install
yum --version
yum search
yum provides
yum shell
   repo
   config
   etc
   
yum deplist
yum list obsoletes
yum list available
yum list installed
yum check-update

If anything fails the release shouldn't happen.

Automated testing

(Based on discussion a Boston FUDcon 2007) The above testing ideas are done from a clean cache i.e. "yum clean all" DavidMalcolm has been writing some test code ("rpmfluff") that could be leveraged to building an automated test suite for yum. Code can currently be found here: https://testing.108.redhat.com/source/browse/testing/trunk/incubator/rpmfluff/

What things will we need to construct test cases?

• yum cache: Each repo has a yum cache (unless it's been enabled since the last time yum was run). Existing manual testing is done from a clean cache. We could automate the following cases:

  • non-existant cache: a freshly configured repo, from dropping in a .repo file

  • clean cache

  • warm cache, up-to-date relative to the server

  • warm cache, out-of-date relative to the server

  • warm cache, up-to-date, but the server has broken metadata (sync not yet complete?)

  • warm cache, up-to-date, but the server has out-of-date metadata (slow mirror, a few days behind)

  • warm cache, but of an older format (from an older version of yum)

  • evil combinations of the above: Extras from today, with Base + Update from yesterday

• state of the repos: typical changes:

  • bumping package version, release, epoch

  • new packages appearing in the repo

  • packages disappearing

• state of the RPM database: what's the software in the db? We will need to have a separate RPM db from the system db, during the test run (rpmfluff handles this)

• fake root in which to install stuff. Parallel to the separate RPM db: we'll want to modify a file tree without hosing the system DB

• yum config: need to easily wire this all up so that the test doesn't need to overwrite the system yum install

• version of yum under test

• version of rpm under test

• what plugins are we using? tests for plugins?

Depsolving: need to be able to check policy, are the results from a run "reasonable"?

Other stuff:

• correctness. Will want assertions/checking/logging of various kinds

• performance. How long does stuff take? Does it hog CPU/memory? (could we use http://www.bootchart.org/ for this?)