e2e_tests/docs/documentation.md

RUN

Abratest has 3 required inputs, but most importantly the test configuration is done through the .env files given with the --env_paths argument. So let's say we want to run abratest with these 3 .env files:

• config1.env [of TYPE authentik]

• config2.env [of TYPE wordpress]

• config3.env [of TYPE wordpress]

Now we run

abratest --env_paths path/config1.env;path/config2.env;path/config3.env [...other args]

abratest -> create Coordinator() instance
└── Coordinator() -> create Runner() subclass instances
    ├── RunnerAuthentik() [based on config1.env, loaded 
    │   │                  from abra/recipes/authentik]
    │   │     # RunnerAuthentik with 3 test files:
    │   ├── RUN pytest path/setup_authentik.py
    │   ├── RUN pytest path/test_authentik_1.py
    │   └── RUN pytest path/test_authentik_2.py
    ├── RunnerWordpress() [based on config2.env, loaded
    │   │                  from abra/recipes/wordpress]
    │   │     # RunnerWordpress with 1 test file
    │   ├── RUN pytest path/setup_authentik.py
    │   ├── RUN pytest path/test_authentik_1.py
    │   └── RUN pytest path/test_authentik_2.py
    └── RunnerWordpress() [based on config3.env, loaded
        │                  from abra/recipes/wordpress]
        │     # RunnerWordpress with 1 test file
        ├── RUN pytest path/setup_authentik.py
        ├── RUN pytest path/test_authentik_1.py
        └── RUN pytest path/test_authentik_2.py

Coordinator will take care of the correct order of the tests. In general, tests are placed in one of 3 categories: setups, tests and cleanups. To associate a test with one of these categories, place the Test in the corresponding list of the Runner class, i.e. Runner.setups = [test] or Runner.tests = [test]. The execution order will be.

[setups] ➔ [tests] ➔ [cleanups]

Furthermore, some Runner classes can depend on others. For example, RunnerWordpress depends on RunnerAuthentik. Therefore, Coordinator will make sure that RunnerAuthentik runs before RunnerWordpress. We will end up with with this order:

#	Runner	Type
1.	Authentik	setups
2.	Wordpress-1	setups
3.	Wordpress-2	setups
4.	Authentik	tests
5.	Wordpress-1	tests
6.	Wordpress-2	tests
7.	Authentik	cleanups
8.	Wordpress-1	cleanups
9.	Wordpress-2	cleanups

Create a custom Runner

To comprehend the process of creating a new subclass of Runner, let's examine a simplified rendition of the RunnerWordpress class. Within it, there exist two setup scripts and two test scripts, one of which operates conditionally.

from pytest_abra import Runner, Test

class RunnerWordpress(Runner):
    env_type = "wordpress"
    dependencies = ["authentik"]
    setups = [
        Test(test_file="setup_wordpress_1.py"),
        Test(test_file="setup_wordpress_2.py"),
    ]
    tests = [
        Test(test_file="test_wordpress.py"),
        Test(condition=condition_function, test_file="test_wordpress_conditional.py"),
    ]
    cleanups = []

The signature of condition functions can be seen below. The function takes one NamedTuple and returns of type bool. You can learn about the contents of the input by looking up the class ConditionArgs. Generally speaking, it provides access to all of the .env files, especially the one related to the current Runner.

def condition_function(args: ConditionArgs) -> bool:
    ...

Discovery of Runners and Tests

• Runners will be discovered, if they are defined in a moduled of name runner_*.py including a class of name Runner*.

• Tests will be discovered by filename as long as they are placed in the parent dir of runner_*.py or in any subdirectory.

DIR parent_dir
├── FILE runner_*.py
├── FILE test1.py
└── DIR subdir
    ├── DIR subsubdir
    │   └── test2.py
    └── test3.py

Create custom Tests

The test files are written in the same way as any other pytest test file. The only difference is that pytest-abra provides custom fixtures that make it easy to get the configuration by the provided .env files and to deal with URLS etc.

todo: add example