# Writing plugins

It is easy to implement local conftest plugins for your own project or pip-installable plugins that can be used throughout many projects, including third party projects. Please refer to Installing and Using plugins if you only want to use but not write plugins.

A plugin contains one or multiple hook functions. Writing hooks explains the basics and details of how you can write a hook function yourself. `pytest` implements all aspects of configuration, collection, running and reporting by calling well specified hooks of the following plugins:

- pytest default plugin reference: loaded from pytest’s internal `_pytest` directory.
- external plugins: modules discovered through setuptools entry points
- conftest.py plugins: modules auto-discovered in test directories

In principle, each hook call is a `1:N` Python function call where `N` is the number of registered implementation functions for a given specification. All specifications and implementations following the `pytest_` prefix naming convention, making them easy to distinguish and find.

## Plugin discovery order at tool startup

`pytest` loads plugin modules at tool startup in the following way:

- by loading all builtin plugins

- by loading all plugins registered through setuptools entry points.

- by pre-scanning the command line for the `-p name` option and loading the specified plugin before actual command line parsing.

- by loading all `conftest.py` files as inferred by the command line invocation:

  - if no test paths are specified use current dir as a test path
  - if exists, load `conftest.py` and `test*/conftest.py` relative to the directory part of the first test path.

  Note that pytest does not find `conftest.py` files in deeper nested sub directories at tool startup. It is usually a good idea to keep your conftest.py file in the top level test or project root directory.

- by recursively loading all plugins specified by the `pytest_plugins` variable in `conftest.py` files

## conftest.py: local per-directory plugins

Local `conftest.py` plugins contain directory-specific hook implementations. Hook Session and test running activities will invoke all hooks defined in `conftest.py` files closer to the root of the filesystem. Example of implementing the `pytest_runtest_setup` hook so that is called for tests in the `a` sub directory but not for other directories:

    a/conftest.py:
        def pytest_runtest_setup(item):
            # called for running each test in 'a' directory
            print ("setting up", item)

    a/test_sub.py:
        def test_sub():
            pass

    test_flat.py:
        def test_flat():
            pass

Here is how you might run it:

    py.test test_flat.py   # will not show "setting up"
    py.test a/test_sub.py  # will show "setting up"

Note

If you have `conftest.py` files which do not reside in a python package directory (i.e. one containing an `__init__.py`) then “import conftest” can be ambiguous because there might be other `conftest.py` files as well on your PYTHONPATH or `sys.path`. It is thus good practise for projects to either put `conftest.py` under a package scope or to never import anything from a conftest.py file.

## Writing your own plugin

If you want to write a plugin, there are many real-life examples you can copy from:

- a custom collection example plugin: A basic example for specifying tests in Yaml files
- around 20 doc:builtin plugins which provide pytest’s own functionality
- many external plugins providing additional features

All of these plugins implement the documented well specified hooks to extend and add functionality.

Note

Make sure to check out the excellent cookiecutter-pytest-plugin project, which is a cookiecutter template for authoring plugins.

The template provides an excellent starting point with a working plugin, tests running with tox, comprehensive README and entry-pointy already pre-configured.

Also consider contributing your plugin to pytest-dev once it has some happy users other than yourself.

## Making your plugin installable by others

If you want to make your plugin externally available, you may define a so-called entry point for your distribution so that `pytest` finds your plugin module. Entry points are a feature that is provided by setuptools. pytest looks up the `pytest11` entrypoint to discover its plugins and you can thus make your plugin available by defining it in your setuptools-invocation:

    # sample ./setup.py file
    from setuptools import setup

    setup(
        name="myproject",
        packages = ['myproject']

        # the following makes a plugin available to pytest
        entry_points = {
            'pytest11': [
                'name_of_plugin = myproject.pluginmodule',
            ]
        },
    )

If a package is installed this way, `pytest` will load `myproject.pluginmodule` as a plugin which can define well specified hooks.

## Requiring/Loading plugins in a test module or conftest file

You can require plugins in a test module or a conftest file like this:

    pytest_plugins = "name1", "name2",

When the test module or conftest plugin is loaded the specified plugins will be loaded as well. You can also use dotted path like this:

    pytest_plugins = "myapp.testsupport.myplugin"

which will import the specified module as a `pytest` plugin.

## Accessing another plugin by name

If a plugin wants to collaborate with code from another plugin it can obtain a reference through the plugin manager like this:

    plugin = config.pluginmanager.getplugin("name_of_plugin")

If you want to look at the names of existing plugins, use the `--traceconfig` option.

## Testing plugins

pytest comes with some facilities that you can enable for testing your plugin. Given that you have an installed plugin you can enable the `testdir` fixture via specifying a command line option to include the pytester plugin (`-p pytester`) or by putting `pytest_plugins = pytester` into your test or `conftest.py` file. You then will have a `testdir` fixure which you can use like this:

    # content of test_myplugin.py

    pytest_plugins = pytester  # to get testdir fixture

    def test_myplugin(testdir):
        testdir.makepyfile("""
            def test_example():
                pass
        """)
        result = testdir.runpytest("--verbose")
        result.fnmatch_lines("""
            test_example*
        """)

Note that by default `testdir.runpytest()` will perform a pytest in-process. You can pass the command line option `--runpytest=subprocess` to have it happen in a subprocess.

Also see the `RunResult` for more methods of the result object that you get from a call to `runpytest`.

# Writing hook functions

## hook function validation and execution

pytest calls hook functions from registered plugins for any given hook specification. Let’s look at a typical hook function for the `pytest_collection_modifyitems(session, config, items)` hook which pytest calls after collection of all test items is completed.

When we implement a `pytest_collection_modifyitems` function in our plugin pytest will during registration verify that you use argument names which match the specification and bail out if not.

Let’s look at a possible implementation:

    def pytest_collection_modifyitems(config, items):
        # called after collection is completed
        # you can modify the ``items`` list

Here, `pytest` will pass in `config` (the pytest config object) and `items` (the list of collected test items) but will not pass in the `session` argument because we didn’t list it in the function signature. This dynamic “pruning” of arguments allows `pytest` to be “future-compatible”: we can introduce new hook named parameters without breaking the signatures of existing hook implementations. It is one of the reasons for the general long-lived compatibility of pytest plugins.

Note that hook functions other than `pytest_runtest_*` are not allowed to raise exceptions. Doing so will break the pytest run.

## firstresult: stop at first non-None result

Most calls to `pytest` hooks result in a list of results which contains all non-None results of the called hook functions.

Some hook specifications use the `firstresult=True` option so that the hook call only executes until the first of N registered functions returns a non-None result which is then taken as result of the overall hook call. The remaining hook functions will not be called in this case.

## hookwrapper: executing around other hooks

New in version 2.7: (experimental)

pytest plugins can implement hook wrappers which wrap the execution of other hook implementations. A hook wrapper is a generator function which yields exactly once. When pytest invokes hooks it first executes hook wrappers and passes the same arguments as to the regular hooks.

At the yield point of the hook wrapper pytest will execute the next hook implementations and return their result to the yield point in the form of a `CallOutcome` instance which encapsulates a result or exception info. The yield point itself will thus typically not raise exceptions (unless there are bugs).

Here is an example definition of a hook wrapper:

    import pytest

    @pytest.hookimpl(hookwrapper=True)
    def pytest_pyfunc_call(pyfuncitem):
        # do whatever you want before the next hook executes

        outcome = yield
        # outcome.excinfo may be None or a (cls, val, tb) tuple

        res = outcome.get_result()  # will raise if outcome was exception
        # postprocess result

Note that hook wrappers don’t return results themselves, they merely perform tracing or other side effects around the actual hook implementations. If the result of the underlying hook is a mutable object, they may modify that result but it’s probably better to avoid it.

## Hook function ordering / call example

For any given hook specification there may be more than one implementation and we thus generally view `hook` execution as a `1:N` function call where `N` is the number of registered functions. There are ways to influence if a hook implementation comes before or after others, i.e. the position in the `N`-sized list of functions:

    # Plugin 1
    @pytest.hookimpl_spec(tryfirst=True)
    def pytest_collection_modifyitems(items):
        # will execute as early as possible

    # Plugin 2
    @pytest.hookimpl_spec(trylast=True)
    def pytest_collection_modifyitems(items):
        # will execute as late as possible

    # Plugin 3
    @pytest.hookimpl_spec(hookwrapper=True)
    def pytest_collection_modifyitems(items):
        # will execute even before the tryfirst one above!
        outcome = yield
        # will execute after all non-hookwrappers executed

Here is the order of execution:

1. Plugin3’s pytest_collection_modifyitems called until the yield point because it is a hook wrapper.
2. Plugin1’s pytest_collection_modifyitems is called because it is marked with `tryfirst=True`.
3. Plugin2’s pytest_collection_modifyitems is called because it is marked with `trylast=True` (but even without this mark it would come after Plugin1).
4. Plugin3’s pytest_collection_modifyitems then executing the code after the yield point. The yield receives a `CallOutcome` instance which encapsulates the result from calling the non-wrappers. Wrappers shall not modify the result.

It’s possible to use `tryfirst` and `trylast` also in conjunction with `hookwrapper=True` in which case it will influence the ordering of hookwrappers among each other.

## Declaring new hooks

Plugins and `conftest.py` files may declare new hooks that can then be implemented by other plugins in order to alter behaviour or interact with the new plugin:

`pytest_addhooks`(pluginmanager)

  called at plugin registration time to allow adding new hooks via a call to pluginmanager.add_hookspecs(module_or_class, prefix).

Hooks are usually declared as do-nothing functions that contain only documentation describing when the hook will be called and what return values are expected.

For an example, see newhooks.py from xdist: pytest distributed testing plugin.

## Optionally using hooks from 3rd party plugins

Using new hooks from plugins as explained above might be a little tricky because of the standard validation mechanism: if you depend on a plugin that is not installed, validation will fail and the error message will not make much sense to your users.

One approach is to defer the hook implementation to a new plugin instead of declaring the hook functions directly in your plugin module, for example:

    # contents of myplugin.py

    class DeferPlugin(object):
        """Simple plugin to defer pytest-xdist hook functions."""

        def pytest_testnodedown(self, node, error):
            """standard xdist hook function.
            """

    def pytest_configure(config):
        if config.pluginmanager.hasplugin('xdist'):
            config.pluginmanager.register(DeferPlugin())

This has the added benefit of allowing you to conditionally install hooks depending on which plugins are installed.

# pytest hook reference

## Initialization, command line and configuration hooks

`pytest_load_initial_conftests`(args, early_config, parser)

  implements the loading of initial conftest files ahead of command line option parsing.

`pytest_cmdline_preparse`(config, args)

  (deprecated) modify command line arguments before option parsing.

`pytest_cmdline_parse`(pluginmanager, args)

  return initialized config object, parsing the specified args.

`pytest_namespace`()

  return dict of name->object to be made globally available in the pytest namespace. This hook is called at plugin registration time.

`pytest_addoption`(parser)

  register argparse-style options and ini-style config values.

  This function must be implemented in a plugin and is called once at the beginning of a test run.

  Parameters: | parser – To add command line options, call `parser.addoption(...)`. To add ini-file values call `parser.addini(...)`.

  Options can later be accessed through the `config` object, respectively:

  - `config.getoption(name)` to retrieve the value of a command line option.
  - `config.getini(name)` to retrieve a value read from an ini-style file.

  The config object is passed around on many internal objects via the `.config` attribute or can be retrieved as the `pytestconfig` fixture or accessed via (deprecated) `pytest.config`.

`pytest_cmdline_main`(config)

  called for performing the main command line action. The default implementation will invoke the configure hooks and runtest_mainloop.

`pytest_configure`(config)

  called after command line options have been parsed and all plugins and initial conftest files been loaded. This hook is called for every plugin.

`pytest_unconfigure`(config)

  called before test process is exited.

## Generic “runtest” hooks

All runtest related hooks receive a `pytest.Item` object.

`pytest_runtest_protocol`(item, nextitem)

  implements the runtest_setup/call/teardown protocol for the given test item, including capturing exceptions and calling reporting hooks.

  Parameters: | item – test item for which the runtest protocol is performed. nextitem – the scheduled-to-be-next test item (or None if this is the end my friend). This argument is passed on to `pytest_runtest_teardown()`.
  Return boolean: | True if no further hook implementations should be invoked.

`pytest_runtest_setup`(item)

  called before `pytest_runtest_call(item)`.

`pytest_runtest_call`(item)

  called to execute the test `item`.

`pytest_runtest_teardown`(item, nextitem)

  called after `pytest_runtest_call`.

  Parameters: | nextitem – the scheduled-to-be-next test item (None if no further test item is scheduled). This argument can be used to perform exact teardowns, i.e. calling just enough finalizers so that nextitem only needs to call setup-functions.

`pytest_runtest_makereport`(item, call)

  return a `_pytest.runner.TestReport` object for the given `pytest.Item` and `_pytest.runner.CallInfo`.

For deeper understanding you may look at the default implementation of these hooks in `_pytest.runner` and maybe also in `_pytest.pdb` which interacts with `_pytest.capture` and its input/output capturing in order to immediately drop into interactive debugging when a test failure occurs.

The `_pytest.terminal` reported specifically uses the reporting hook to print information about a test run.

## Collection hooks

`pytest` calls the following hooks for collecting files and directories:

`pytest_ignore_collect`(path, config)

  return True to prevent considering this path for collection. This hook is consulted for all files and directories prior to calling more specific hooks.

`pytest_collect_directory`(path, parent)

  called before traversing a directory for collection files.

`pytest_collect_file`(path, parent)

  return collection Node or None for the given path. Any new node needs to have the specified `parent` as a parent.

For influencing the collection of objects in Python modules you can use the following hook:

`pytest_pycollect_makeitem`(collector, name, obj)

  return custom item/collector for a python object in a module, or None.

`pytest_generate_tests`(metafunc)

  generate (multiple) parametrized calls to a test function.

After collection is complete, you can modify the order of items, delete or otherwise amend the test items:

`pytest_collection_modifyitems`(session, config, items)

  called after collection has been performed, may filter or re-order the items in-place.

## Reporting hooks

Session related reporting hooks:

`pytest_collectstart`(collector)

  collector starts collecting.

`pytest_itemcollected`(item)

  we just collected a test item.

`pytest_collectreport`(report)

  collector finished collecting.

`pytest_deselected`(items)

  called for test items deselected by keyword.

And here is the central hook for reporting about test execution:

`pytest_runtest_logreport`(report)

  process a test setup/call/teardown report relating to the respective phase of executing a test.

## Debugging/Interaction hooks

There are few hooks which can be used for special reporting or interaction with exceptions:

`pytest_internalerror`(excrepr, excinfo)

  called for internal errors.

`pytest_keyboard_interrupt`(excinfo)

  called for keyboard interrupt.

`pytest_exception_interact`(node, call, report)

  (experimental, new in 2.4) called when an exception was raised which can potentially be interactively handled.

  This hook is only called if an exception was raised that is not an internal exception like “skip.Exception”.

# Reference of objects involved in hooks

class `Config`

  access to configuration values, pluginmanager and plugin hooks.

  `option` = None

    access to command line option as attributes. (deprecated), use `getoption()` instead

  `pluginmanager` = None

    a pluginmanager instance

  `add_cleanup`(func)

    Add a function to be called when the config object gets out of use (usually coninciding with pytest_unconfigure).

  `warn`(code, message, fslocation=None)

    generate a warning for this test session.

  classmethod `fromdictargs`(option_dict, args)

    constructor useable for subprocesses.

  `addinivalue_line`(name, line)

    add a line to an ini-file option. The option must have been declared but might not yet be set in which case the line becomes the the first line in its value.

  `getini`(name)

    return configuration value from an ini file. If the specified name hasn’t been registered through a prior `parser.addini` call (usually from a plugin), a ValueError is raised.

  `getoption`(name, default=<NOTSET>, skip=False)

    return command line option value.

    Parameters: | name – name of the option. You may also specify the literal `--OPT` option instead of the “dest” option name. default – default value if no option of that name exists. skip – if True raise pytest.skip if option does not exists or has a None value.

  `getvalue`(name, path=None)

    (deprecated, use getoption())

  `getvalueorskip`(name, path=None)

    (deprecated, use getoption(skip=True))

class `Parser`

  Parser for command line arguments and ini-file values.

  Variables: | extra_info – dict of generic param -> value to display in case there’s an error processing the command line arguments.

  `getgroup`(name, description='', after=None)

    get (or create) a named option Group.

    Name: | name of the option group.
    Description: | long description for –help output.
    After: | name of other group, used for ordering –help output.

    The returned group object has an `addoption` method with the same signature as `parser.addoption` but will be shown in the respective group in the output of `pytest. --help`.

  `addoption`(*opts, **attrs)

    register a command line option.

    Opts: | option names, can be short or long options.
    Attrs: | same attributes which the `add_option()` function of the argparse library accepts.

    After command line parsing options are available on the pytest config object via `config.option.NAME` where `NAME` is usually set by passing a `dest` attribute, for example `addoption("--long", dest="NAME", ...)`.

  `addini`(name, help, type=None, default=None)

    register an ini-file option.

    Name: | name of the ini-variable
    Type: | type of the variable, can be `pathlist`, `args` or `linelist`.
    Default: | default value if no ini-file option exists but is queried.

    The value of ini-variables can be retrieved via a call to `config.getini(name)`.

class `CallInfo`

  Result/Exception info a function invocation.

  `when` = None

    context of invocation: one of “setup”, “call”, “teardown”, “memocollect”

  `excinfo` = None

    None or ExceptionInfo object.

class `TestReport`

  Basic test report object (also used for setup and teardown calls if they fail).

  `nodeid` = None

    normalized collection node id

  `location` = None

    a (filesystempath, lineno, domaininfo) tuple indicating the actual location of a test item - it might be different from the collected one e.g. if a method is inherited from a different module.

  `keywords` = None

    a name -> value dictionary containing all keywords and markers associated with a test invocation.

  `outcome` = None

    test outcome, always one of “passed”, “failed”, “skipped”.

  `longrepr` = None

    None or a failure representation.

  `when` = None

    one of ‘setup’, ‘call’, ‘teardown’ to indicate runtest phase.

  `sections` = None

    list of (secname, data) extra information which needs to marshallable

  `duration` = None

    time it took to run just the test

class `CallOutcome`

  Outcome of a function call, either an exception or a proper result. Calling the `get_result` method will return the result or reraise the exception raised when the function was called.

`get_plugin_manager`()

  Obtain a new instance of the `_pytest.config.PytestPluginManager`, with default plugins already loaded.

  This function can be used by integration with other tools, like hooking into pytest to run tests into an IDE.

class `PytestPluginManager`

  Bases: `pluggy.PluginManager`

  Overwrites `pluggy.PluginManager` to add pytest-specific functionality:

  - loading plugins from the command line, `PYTEST_PLUGIN` env variable and `pytest_plugins` global variables found in plugins being loaded;
  - `conftest.py` loading during start-up;
  `addhooks`(module_or_class)

    Deprecated since version 2.8.

    Use `pluggy.PluginManager.add_hookspecs()` instead.

  `parse_hookimpl_opts`(plugin, name)
  `parse_hookspec_opts`(module_or_class, name)
  `register`(plugin, name=None)
  `getplugin`(name)
  `hasplugin`(name)

    Return True if the plugin with the given name is registered.

  `pytest_configure`(config)
  `consider_preparse`(args)
  `consider_pluginarg`(arg)
  `consider_conftest`(conftestmodule)
  `consider_env`()
  `consider_module`(mod)
  `import_plugin`(modname)

class `PluginManager`

  Core Pluginmanager class which manages registration of plugin objects and 1:N hook calling.

  You can register new hooks by calling `addhooks(module_or_class)`. You can register plugin objects (which contain hooks) by calling `register(plugin)`. The Pluginmanager is initialized with a prefix that is searched for in the names of the dict of registered plugin objects. An optional excludefunc allows to blacklist names which are not considered as hooks despite a matching prefix.

  For debugging purposes you can call `enable_tracing()` which will subsequently send debug information to the trace helper.

  `register`(plugin, name=None)

    Register a plugin and return its canonical name or None if the name is blocked from registering. Raise a ValueError if the plugin is already registered.

  `unregister`(plugin=None, name=None)

    unregister a plugin object and all its contained hook implementations from internal data structures.

  `set_blocked`(name)

    block registrations of the given name, unregister if already registered.

  `is_blocked`(name)

    return True if the name blogs registering plugins of that name.

  `add_hookspecs`(module_or_class)

    add new hook specifications defined in the given module_or_class. Functions are recognized if they have been decorated accordingly.

  `get_plugins`()

    return the set of registered plugins.

  `is_registered`(plugin)

    Return True if the plugin is already registered.

  `get_canonical_name`(plugin)

    Return canonical name for a plugin object. Note that a plugin may be registered under a different name which was specified by the caller of register(plugin, name). To obtain the name of an registered plugin use `get_name(plugin)` instead.

  `get_plugin`(name)

    Return a plugin or None for the given name.

  `get_name`(plugin)

    Return name for registered plugin or None if not registered.

  `check_pending`()

    Verify that all hooks which have not been verified against a hook specification are optional, otherwise raise PluginValidationError

  `load_setuptools_entrypoints`(entrypoint_name)

    Load modules from querying the specified setuptools entrypoint name. Return the number of loaded plugins.

  `list_plugin_distinfo`()

    return list of distinfo/plugin tuples for all setuptools registered plugins.

  `list_name_plugin`()

    return list of name/plugin pairs.

  `get_hookcallers`(plugin)

    get all hook callers for the specified plugin.

  `add_hookcall_monitoring`(before, after)

    add before/after tracing functions for all hooks and return an undo function which, when called, will remove the added tracers.

    `before(hook_name, hook_impls, kwargs)` will be called ahead of all hook calls and receive a hookcaller instance, a list of HookImpl instances and the keyword arguments for the hook call.

    `after(outcome, hook_name, hook_impls, kwargs)` receives the same arguments as `before` but also a `_CallOutcome`` object which represents the result of the overall hook call.

  `enable_tracing`()

    enable tracing of hook calls and return an undo function.

  `subset_hook_caller`(name, remove_plugins)

    Return a new _HookCaller instance for the named method which manages calls to all registered plugins except the ones from remove_plugins.

class `Testdir`

  Temporary test directory with tools to test/run py.test itself.

  This is based on the `tmpdir` fixture but provides a number of methods which aid with testing py.test itself. Unless `chdir()` is used all methods will use `tmpdir` as current working directory.

  Attributes:

  Tmpdir: | The `py.path.local` instance of the temporary directory.
  Plugins: | A list of plugins to use with `parseconfig()` and `runpytest()`. Initially this is an empty list but plugins can be added to the list. The type of items to add to the list depend on the method which uses them so refer to them for details.

  `makeconftest`(source)

    Write a contest.py file with ‘source’ as contents.

  `makepyfile`(*args, **kwargs)

    Shortcut for .makefile() with a .py extension.

  `runpytest_inprocess`(*args, **kwargs)

    Return result of running pytest in-process, providing a similar interface to what self.runpytest() provides.

  `runpytest`(*args, **kwargs)

    Run pytest inline or in a subprocess, depending on the command line option “–runpytest” and return a `RunResult`.

  `runpytest_subprocess`(*args, **kwargs)

    Run py.test as a subprocess with given arguments.

    Any plugins added to the `plugins` list will added using the `-p` command line option. Addtionally `--basetemp` is used put any temporary files and directories in a numbered directory prefixed with “runpytest-” so they do not conflict with the normal numberd pytest location for temporary files and directories.

    Returns a `RunResult`.

class `RunResult`

  The result of running a command.

  Attributes:

  Ret: | The return value.
  Outlines: | List of lines captured from stdout.
  Errlines: | List of lines captures from stderr.
  Stdout: | `LineMatcher` of stdout, use `stdout.str()` to reconstruct stdout or the commonly used `stdout.fnmatch_lines()` method.
  Stderrr: | `LineMatcher` of stderr.
  Duration: | Duration in seconds.

  `parseoutcomes`()

    Return a dictionary of outcomestring->num from parsing the terminal output that the test process produced.

  `assert_outcomes`(passed=0, skipped=0, failed=0)

    assert that the specified outcomes appear with the respective numbers (0 means it didn’t occur) in the text output from a test run.

class `LineMatcher`

  Flexible matching of text.

  This is a convenience class to test large texts like the output of commands.

  The constructor takes a list of lines without their trailing newlines, i.e. `text.splitlines()`.

  `str`()

    Return the entire original text.

  `fnmatch_lines_random`(lines2)

    Check lines exist in the output.

    The argument is a list of lines which have to occur in the output, in any order. Each line can contain glob whildcards.

  `get_lines_after`(fnline)

    Return all lines following the given line in the text.

    The given line can contain glob wildcards.

  `fnmatch_lines`(lines2)

    Search the text for matching lines.

    The argument is a list of lines which have to match and can use glob wildcards. If they do not match an pytest.fail() is called. The matches and non-matches are also printed on stdout.