# Working with non-python tests

## A basic example for specifying tests in Yaml files

Here is an example `conftest.py` (extracted from Ali Afshnars special purpose pytest-yamlwsgi plugin). This `conftest.py` will collect `test*.yml` files and will execute the yaml-formatted content as custom tests:

    # content of conftest.py

    import pytest

    def pytest_collect_file(parent, path):
        if path.ext == ".yml" and path.basename.startswith("test"):
            return YamlFile(path, parent)

    class YamlFile(pytest.File):
        def collect(self):
            import yaml # we need a yaml parser, e.g. PyYAML
            raw = yaml.safe_load(self.fspath.open())
            for name, spec in raw.items():
                yield YamlItem(name, self, spec)

    class YamlItem(pytest.Item):
        def __init__(self, name, parent, spec):
            super(YamlItem, self).__init__(name, parent)
            self.spec = spec

        def runtest(self):
            for name, value in self.spec.items():
                # some custom test execution (dumb example follows)
                if name != value:
                    raise YamlException(self, name, value)

        def repr_failure(self, excinfo):
            """ called when self.runtest() raises an exception. """
            if isinstance(excinfo.value, YamlException):
                return "\n".join([
                    "usecase execution failed",
                    "   spec failed: %r: %r" % excinfo.value.args[1:3],
                    "   no further details known at this point."
                ])

        def reportinfo(self):
            return self.fspath, 0, "usecase: %s" % self.name

    class YamlException(Exception):
        """ custom exception for error reporting. """

You can create a simple example file:

    # test_simple.yml
    ok:
        sub1: sub1

    hello:
        world: world
        some: other

and if you installed PyYAML or a compatible YAML-parser you can now execute the test specification:

    nonpython $ py.test test_simple.yml
    ======= test session starts ========
    platform linux2 -- Python 2.7.9, pytest-2.8.0.dev4, py-1.4.28, pluggy-0.3.0
    rootdir: $PWD/doc/en, inifile: pytest.ini
    collected 2 items

    test_simple.yml .F

    ======= FAILURES ========
    _______ usecase: hello ________
    usecase execution failed
       spec failed: 'some': 'other'
       no further details known at this point.
    ======= 1 failed, 1 passed in 0.12 seconds ========

You get one dot for the passing `sub1: sub1` check and one failure. Obviously in the above `conftest.py` you’ll want to implement a more interesting interpretation of the yaml-values. You can easily write your own domain specific testing language this way.

Note

`repr_failure(excinfo)` is called for representing test failures. If you create custom collection nodes you can return an error representation string of your choice. It will be reported as a (red) string.

`reportinfo()` is used for representing the test location and is also consulted when reporting in `verbose` mode:

    nonpython $ py.test -v
    ======= test session starts ========
    platform linux2 -- Python 2.7.9, pytest-2.8.0.dev4, py-1.4.28, pluggy-0.3.0 -- $PWD/.env/bin/python2.7
    rootdir: $PWD/doc/en, inifile: pytest.ini
    collecting ... collected 2 items

    test_simple.yml::ok PASSED
    test_simple.yml::hello FAILED

    ======= FAILURES ========
    _______ usecase: hello ________
    usecase execution failed
       spec failed: 'some': 'other'
       no further details known at this point.
    ======= 1 failed, 1 passed in 0.12 seconds ========

While developing your custom test collection and execution it’s also interesting to just look at the collection tree:

    nonpython $ py.test --collect-only
    ======= test session starts ========
    platform linux2 -- Python 2.7.9, pytest-2.8.0.dev4, py-1.4.28, pluggy-0.3.0
    rootdir: $PWD/doc/en, inifile: pytest.ini
    collected 2 items
    <YamlFile 'example/nonpython/test_simple.yml'>
      <YamlItem 'ok'>
      <YamlItem 'hello'>

    =======  in 0.12 seconds ========