e2e_tests/prototyping/structure.md

Abratest has 3 required inputs, but most importantly the test configuration is done through the .env files given with the --env_paths argument. So let's say we want to run abratest with these 3 .env files:

• config1.env [of TYPE authentik]

• config2.env [of TYPE wordpress]

• config3.env [of TYPE wordpress]

Now we run

abratest --env_paths path/config1.env;path/config2.env;path/config3.env [...other args]

abratest -> create Coordinator() instance
└── Coordinator() -> create Runner() subclass instances
    ├── RunnerAuthentik() [based on config1.env, loaded 
    │   │                  from abra/recipes/authentik]
    │   │     # RunnerAuthentik with 3 test files:
    │   ├── RUN pytest path/setup_authentik.py
    │   ├── RUN pytest path/test_authentik_1.py
    │   └── RUN pytest path/test_authentik_2.py
    ├── RunnerWordpress() [based on config2.env, loaded
    │   │                  from abra/recipes/wordpress]
    │   │     # RunnerWordpress with 1 test file
    │   ├── RUN pytest path/setup_authentik.py
    │   ├── RUN pytest path/test_authentik_1.py
    │   └── RUN pytest path/test_authentik_2.py
    └── RunnerWordpress() [based on config3.env, loaded
        │                  from abra/recipes/wordpress]
        │     # RunnerWordpress with 1 test file
        ├── RUN pytest path/setup_authentik.py
        ├── RUN pytest path/test_authentik_1.py
        └── RUN pytest path/test_authentik_2.py

Coordinator will take care of the correct order of the tests. In general, tests are placed in one of 3 categories: setups, tests and cleanups. To associate a test with one of these categories, place the Test in the corresponding list of the Runner class, i.e. Runner.setups = [test] or Runner.tests = [test]. The execution order will be.

[setups] ➔ [tests] ➔ [cleanups]

Furthermore, some Runner classes can depend on others. For example, RunnerWordpress depends on RunnerAuthentik. Therefore, Coordinator will make sure that RunnerAuthentik runs before RunnerWordpress. We will end up with with this order:

#	Runner	Type
1.	Authentik	setups
2.	Wordpress-1	setups
3.	Wordpress-2	setups
4.	Authentik	tests
5.	Wordpress-1	tests
6.	Wordpress-2	tests
7.	Authentik	cleanups
8.	Wordpress-1	cleanups
9.	Wordpress-2	cleanups

To comprehend this process, let's examine a simplified rendition of the RunnerWordpress class. Within it, there exist two setup scripts and two test scripts, one of which operates conditionally.

class RunnerWordpress(Runner):
    env_type = "wordpress"
    dependencies = ["authentik"]
    setups = [
        Test(test_file="setup_wordpress_1.py"),
        Test(test_file="setup_wordpress_2.py"),
    ]
    tests = [
        Test(test_file="test_wordpress.py"),
        Test(condition=condition_function, test_file="test_wordpress_conditional.py"),
    ]
    cleanups = []