e2e_tests/docs/documentation.md

pytest-abra

Pytest-Abra is an installable python package baed on pytest, designed to test instances created with abra. After installation, you will have two things:

• abratest CLI command. Used to initialize the testing.

• pytest-abra Pytest plugin. Automatically loads custom fixtures in any pytest run (see pytest_abra/custom_fixtures.py)

Getting Started

Pytest-abra can easily be installed on any system but also offers a Docker image. To use pytest-abra, follow these steps:

Usage [without Docker]

Installation [without Docker]

To clone with submodules, use these git commands:

git clone --recurse-submodules <repository>
// optional:
git submodule update --init    // add submodule after normal cloning
git submodule update --remote  // update submodules

Create a python environment and install all dependencies via

pip install -e .
playwright install

Run [without Docker]

Run the helper script or directly use the cli command (see docs)

python main.py  # run pytest-abra
abratest [options]

Usage [with docker]

Installation [with docker]

To clone with submodules, use these git commands:

git clone --recurse-submodules <repository>
// optional:
git submodule update --init    // add submodule after normal cloning
git submodule update --remote  // update submodules

Build the image

docker compose build  # build the image
docker compose build --no-cache  # Force rebuild without cache

Run [with docker]

Run the script

docker compose run --rm app python main.py  # run pytest-abra
docker compose run --rm -it app /bin/bash  # use the container interactively

Documentation

After Installation, abratest can be called via terminal:

abratest [arguments]

To run successfully, very specific arguments are required. The easiest way to use abratest is with the helper script main.py. Of yourse you can implement a similar helper script in the language of your liking.

CLI Interface

The cli command abratest has 3 required arguments:

• --env_paths ENV_PATHS: list of the .env files used in the test
• --recipes_dir RECIPES_DIR: directory of all available abra recipes
• --output_dir OUTPUT_DIR: target directory for all test results

Furtheremore, there are these optional arguments:

• --resume: abratest will take the directory in output_dir with the most recent creation date and resume the tests there.
• --session_id SESSION_ID: Instead of generating a new session_id, the given session_id is used to run or resume the test. Overwrites --resume to False.
• --debug: enables playwright debug mode, see docs here
• --timeout: will overwrite the default playwright timeouts in [ms], see docs here and here. In our current setup, some tests can fail at 10s but will pass with 20s.

env_paths [required | string]

The .env files provied through the --env_paths argument are the most important input to abratest, as they serve as configuration for the tests. One or more paths pointing at .env files can be provided, multiple paths are separated with ";". These .env files are actually the same files that are used to configure the abra recipes for instance creation.

To run abratest with these .env configuration files

• /path/config_1.env [of TYPE authentik]
• /path/config_2.env [of TYPE wordpress]
• /path/config_3.env [of TYPE wordpress]

we simply call

abratest --env_paths /path/config_1.env;/path/config_2.env;/path/config_3.env [...other args]

Under the hood, each .env file in --env_paths will create one instance of a Runner subclass. Let's say we have config_2.env containing TYPE=wordpress. This will create an instance of RunnerWordpress. This class has to be imported from recipes_dir.

recipes_dir [required | string]

The required argument --recipes_dir has to point to the directory, where all the abra recipes are stored. We can call abratest with

abratest --recipes_dir /path/to/abra/recipes

The expected dir structure inside of recipes_dir is as follows:

DIR recipes_dir [contains abra recipes]
│
├── DIR authentik [authentik recipe]
│   ├── [files from authentik recipe]
│   └── DIR tests_authentik [pytest tests for authentik]
│       ├── FILE runner_authentik.py  # containing RunnerAuthentik class
│       └── [pytest_files]
│
└── DIR wordpress [wordpress recipe]
    ├── [files from wordpress recipe]
    └── DIR tests_wordpress [pytest tests for wordpress]
        ├── FILE runner_wordpress.py  # containing RunnerWordpress class
        └── [pytest_files]

The class RunnerWordpress will be automatically imported using importlib library, which is equivalent to the code below. Note that recipes_dir will be added to sys.path automatically for the import to work and that every Runner class matching recipes_dir.rglob("*/runner*.py") will be imported.

from wordpress.tests_wordpress.runner_wordpress import RunnerWordpress

output_dir [required | string]

Path to the directory where all test outputs are stored (test report, tracebacks, playwright traces etc.)

abratest --output_dir /path/to/output

Functionality

Abratest has 3 required inputs, but most importantly the test configuration is done through the .env files given with the --env_paths argument. So let's say we want to run abratest with these 3 .env files:

• config1.env [of TYPE authentik]

• config2.env [of TYPE wordpress]

• config3.env [of TYPE wordpress]

Now we run

abratest --env_paths path/config1.env;path/config2.env;path/config3.env [...other args]

abratest -> create Coordinator() instance
└── Coordinator() -> create Runner() subclass instances
    ├── RunnerAuthentik() [based on config1.env, loaded 
    │   │                  from abra/recipes/authentik]
    │   │     # RunnerAuthentik with 3 test files:
    │   ├── RUN pytest path/setup_authentik.py
    │   ├── RUN pytest path/test_authentik_1.py
    │   └── RUN pytest path/test_authentik_2.py
    ├── RunnerWordpress() [based on config2.env, loaded
    │   │                  from abra/recipes/wordpress]
    │   │     # RunnerWordpress with 1 test file
    │   ├── RUN pytest path/setup_authentik.py
    │   ├── RUN pytest path/test_authentik_1.py
    │   └── RUN pytest path/test_authentik_2.py
    └── RunnerWordpress() [based on config3.env, loaded
        │                  from abra/recipes/wordpress]
        │     # RunnerWordpress with 1 test file
        ├── RUN pytest path/setup_authentik.py
        ├── RUN pytest path/test_authentik_1.py
        └── RUN pytest path/test_authentik_2.py

Coordinator will take care of the correct order of the tests. In general, tests are placed in one of 3 categories: setups, tests and cleanups. To associate a test with one of these categories, place the Test in the corresponding list of the Runner class, i.e. Runner.setups = [test] or Runner.tests = [test]. The execution order will be.

[setups] ➔ [tests] ➔ [cleanups]

Furthermore, some Runner classes can depend on others. For example, RunnerWordpress depends on RunnerAuthentik. Therefore, Coordinator will make sure that RunnerAuthentik runs before RunnerWordpress. We will end up with with this order:

#	Runner	Type
1.	Authentik	setups
2.	Wordpress-1	setups
3.	Wordpress-2	setups
4.	Authentik	tests
5.	Wordpress-1	tests
6.	Wordpress-2	tests
7.	Authentik	cleanups
8.	Wordpress-1	cleanups
9.	Wordpress-2	cleanups

Create a test suite for a recipe

todo

To understand how a test suite is built, let's have a look at the files

runner_authentik.py -> required, defines the Runner subclass (see below) conftest.py -> not required. special file for pytest. is automatically discovered and loaded. convenient place to define fixtures and functions to be used in more than one test routine setup_authentik.py -> not required. can hold setup routine for authentik, has to be registered in runner_authentik.py

Create a custom Runner

To comprehend the process of creating a new subclass of Runner, let's examine a simplified rendition of the RunnerWordpress class. Within it, there exist two setup scripts and two test scripts, one of which operates conditionally.

from pytest_abra import Runner, Test

class RunnerWordpress(Runner):
    env_type = "wordpress"
    dependencies = ["authentik"]
    setups = [
        Test(test_file="setup_wordpress_1.py"),
        Test(test_file="setup_wordpress_2.py"),
    ]
    tests = [
        Test(test_file="test_wordpress.py"),
        Test(condition=condition_function, test_file="test_wordpress_conditional.py"),
    ]
    cleanups = []

The signature of condition functions can be seen below. The function takes one NamedTuple and returns of type bool. You can learn about the contents of the input by looking up the class ConditionArgs. Generally speaking, it provides access to all of the .env files, especially the one related to the current Runner.

def condition_function(args: ConditionArgs) -> bool:
    ...

Discovery of Runners and Tests

• Runners will be discovered, if they are defined in a moduled of name runner_*.py including a class of name Runner*.

• Tests will be discovered by filename as long as they are placed in the parent dir of runner_*.py or in any subdirectory.

DIR parent_dir
├── FILE runner_*.py
├── FILE test1.py
└── DIR subdir
    ├── DIR subsubdir
    │   └── test2.py
    └── test3.py

Create custom Tests

The test files are written in the same way as any other pytest test file. The only difference is that pytest-abra provides custom fixtures that make it easy to get the configuration by the provided .env files and to deal with URLS etc.

Step 1) Add new Test

Create a new testfile new_test.py in the same directory or a subdirectory of runner_wordpress.py. Register new_test.py as a test in the RunnerWordpress class. Set prevent_skip=True, so that you can run your new test over and over again for debugging, without it being skipped

# runner_wordpress.py
from pytest_abra import Runner, Test

class RunnerWordpress(Runner):
    env_type = "wordpress"
    tests = [
        Test(test_file="working_test.py"),
        Test(test_file="new_test.py", prevent_skip=True),
    ]

# new_test.py

def test_new():
    ...

Step 2) Call abratest

Call abratest with --debug to enable playwright debug mode and either --session_id or --resume.

abratest [required-options] --debug --session_id debug_session

This could be done by modifying main.py. The first time you run abratest, all tests will be executed as usual. The second time, all tests will be skipped as they have passed already. Only your new test will be run again and again, as the prevent_skip option is enabled. So you can run all tests once and then skip all tests besides your new test you want to debug.

todo: add example

Playwright Debug & Codegen

Use playwright debug mode or codegen to create testing code easily by recording browser actions https://playwright.dev/python/docs/codegen

abratest --debug  # launch your tests in debug mode
playwright codegen demo.playwright.dev/todomvc  # visit given url in codegen mode

Development

pytest  # test pytest-abra
pytest -m "not slow"  # test pytest-abra without slow tests
pytest  --collect-only  # debug test pytest-abra
docker compose run --rm app pytest  # run pytest-abra