e2e_tests/README.md

pytest-abra

Pytest-Abra is an installable python package design to test instances created with abra. After installation, you will have two things:

• abratest CLI command. Used to initialize the testing.

• pytest-abra Pytest plugin. Automatically loads custom fixtures in any pytest.

CLI (abratest)

Abratest can be called via terminal:

abratest [arguments]

To run successfully, very specific arguments are required. The easiest way to use abratest is with the helper script in main.py. Of yourse you can implement a similar helper script in the language of your liking. The cli command abratest has 3 required arguments:

• --env_paths: list of the .env files used in the test
• --recipes_dir: directory of all available abra recipes
• --output_dir: target directory for all test results

env_paths [string]

The variable env_paths consists of one or more paths pointing at .env files. The paths are separated with ";". These .env files are actually configuration files for abra recipes, but pytest-abra uses the same files for test configuration.

To run abratest with these .env configuration files

/path/to/config_1.env
/path/to/config_2.env
/path/to/config_3.env

we simply call

abratest --env_paths /path/to/config_1.env;/path/to/config_2.env;/path/to/config_3.env

Under the hood, each .env file in --env_paths will create one instance of a Runner subclass. Let's say we have wordpress_configuration.env containing TYPE=wordpress. This will create an instance of RunnerWordpress. This class has to be imported from recipes_dir.

recipes_dir [string]

The required argument --recipes_dir has to point to the directory, where all the abra recipes are stored. We can call abratest with

abratest --recipes_dir /path/to/abra/recipes

The expected dir structure inside of recipes_dir is as follows:

DIR recipes_dir [contains abra recipes]
│
├── DIR authentik [authentik recipe]
│   ├── [files from authentik recipe]
│   └── DIR tests_authentik [pytest tests for authentik]
│       ├── FILE runner_authentik.py  # containing RunnerAuthentik class
│       └── [pytest_files]
│
└── DIR wordpress [wordpress recipe]
    ├── [files from wordpress recipe]
    └── DIR tests_wordpress [pytest tests for wordpress]
        ├── FILE runner_wordpress.py  # containing RunnerWordpress class
        └── [pytest_files]

The class RunnerWordpress will be automatically imported by importlib, which is equivalent to

from wordpress.tests_wordpress.runner_wordpress import RunnerWordpress

output_dir [string]

Path to the directory where all test outputs are stored (test report, tracebacks, playwright traces etc.)

abratest --output_dir /path/to/output

Usage

To use pytest-abra, follow these steps:

1. GIT clone [with & without Docker]

To clone with submodules, use these git commands:

git clone --recurse-submodules <repository>
// optional:
git submodule update --init    // add submodule after normal cloning
git submodule update --remote  // update submodules

Run

You can run pytest-abra with and without Docker. Choose now and follow the steps accordingly:

2.1 Run without Docker

Installation

Create a python environment and install all dependencies via

pip install -e .
playwright install

Run the script with

python main.py

2.2 Run with Docker

docker compose build  # build the image
docker compose run --rm app python main.py  # run pytest-abra
docker compose run --rm -it app /bin/bash  # debug the container

Force rebuild with cache

docker-compose up --build

Force rebuild without cache

docker-compose build --no-cache

Codegen

Use playwright codegen to create code for new testes easily https://playwright.dev/python/docs/codegen

playwright codegen demo.playwright.dev/todomvc

Development

pytest  # test pytest-abra
pytest  --collect-only  # debug test pytest-abra
docker compose run --rm app pytest  # run pytest-abra