Add the concept of atexit hooks to Zaza

[ChrisMacNaughton]

These hooks will be run after a test run completes via the
test portion of Zaza's lifecycle. An example situation where
atexit hooks can be beneficial to Zaza would be the case where
resources are created and verified in one test, and then the
thing under test is modified, and then the resources should
be re-verified. A specific example of the above would be for
a charm or payload upgrade.

An minimal example test using atexit:

import zaza.atexit

def cleanup_vm():
    for vm in cls.nova_client.servers.list():
        if vm.name == 'ins-1':
            vm.delete()
            openstack_utils.resource_removed(
                cls.nova_client.servers,
                vm.id,
                msg="Waiting for the Nova VM {} to be deleted".format(vm.name))

class VMTests(test_utils.OpenStackBaseTest):
    """Encapsulate VM tests."""

    @classmethod
    def setUpClass(cls):
        """Run class setup for running tests."""
        super(VMTests, cls).setUpClass()
        zaza.atexit(cleanup_vm)

    def test_share(self):
        instance = None
        for vm in cls.nova_client.servers.list():
            if vm.name == 'ins-1':
                instance = vm
       id instance is None:
            instance = self.launch_guest(
                guest_name='ins-1')
       fip = neutron_tests.floating_ips_from_instance(instance)[0]
       openstack_utils.ping_response(fip)

In the above example, a VM is spawned on the first pass through
the test case but not cleaned up. The second time through the
test case, the previous VM is used, and the verification is that
the instance is pinged.

When Zaza finishes the entire test run, it will call the atexit
function that was registered and cleanup the server.

[coreycb]

I think this was inspired due to the need to run tempest twice (pre and post upgrade). For that solution I wonder if we should have 2 different tests_options sections in the tests.yaml, where the first could use keep-workspace=True and the 2nd keep-workspace=False. This would also allow running different tests the in pre vs the post upgrade. As seen in [1] the same tests_options section is used for both the pre and post upgrade runs.

[1] https://github.com/openstack-charmers/charmed-openstack-tester/blob/8d8fb4a27e3d85bed0eb2b03877ce6ad16bb8d30/tests/charm-upgrade/tests/tests.yaml

[ChrisMacNaughton]

The initial inspiration was that discussion but I actually think that the tempest test case should setup and cleanup the config in the test's setup/teardown rather than via another step.

At this point, the goal for something like this is, as noted in the commit, the ability to setup resources in a test, then do more tests / change things, and then revalidate the pre-existing resources, while still cleaning up at the very end of a test run.

[ajkavanagh]

This might be confusingly named (or at least it might be confusing me). Python has an atexit handler: https://docs.python.org/3/library/atexit.html - for doing exactly this job when a "script/program" exits. I'm also a bit hesitant about it being a global? It seems like this might be better as a contextmanager that can be used in a with statement that always runs clean-up functions in the __exit__() function regardless of what happened?

[ChrisMacNaughton]

The idea is that there may be test resources that we want to out-live an invocation of the test case, so would want a "do this thing when the whole run is complete, regardless of pass/fail". We do something similar, but per test case, by allowing test classes to define cleanup methods.

An example use case for this comes in the form of the upgrade testing: A developer wants to spawn an instance before upgrading OpenStack, and confirm that the instance survives the upgrade, and is still functional.

The idea of a contextmanager would still end up cleaning up its "contained" resources at the end of the block, wouldn't it?

[ajkavanagh]

It's definitely an intriguing issue. I ran into some interesting, similar, issues with the event framework and in the end I've built a notification system into zaza (not merged yet). Outline of how it works here: https://github.com/ajkavanagh/zaza/blob/events-logging/zaza/notifications/__init__.py

Essentially, you can add a handler that listens for a notification and then do something at that point, and there are notifications at bundle, test case and test function level.

So, drawing parallels, it sounds like this needs clean-up at the end of a bundle? Can the bundle code just not be 'wrapped' with cleanup?