# pytest-abra

Pytest-Abra is an installable python package baed on pytest, designed to test instances created with [abra](https://docs.coopcloud.tech/abra/). After installation, you will have two things:

- `abratest` CLI command. *Used to initialize the testing.*

- `pytest-abra` Pytest plugin. *Automatically loads custom fixtures in any pytest run (see `pytest_abra/custom_fixtures.py`)*

# Getting Started

Pytest-abra can easily be installed on any system but also offers a Docker image. To use pytest-abra, follow these steps:

## Usage [without Docker]

### Installation [without Docker]

To clone with submodules, use these git commands:

```bash
git clone --recurse-submodules <repository>
// optional:
git submodule update --init    // add submodule after normal cloning
git submodule update --remote  // update submodules
```

Create a python environment and install all dependencies via

```bash
pip install -e .
playwright install
```

### Run [without Docker]

Run the helper script or directly use the cli command (see docs)

```bash
python main.py  # run pytest-abra
abratest [options]
```

## Usage [with docker]

### Installation [with docker]

To clone with submodules, use these git commands:

```bash
git clone --recurse-submodules <repository>
// optional:
git submodule update --init    // add submodule after normal cloning
git submodule update --remote  // update submodules
```

Build the image

```bash
docker compose build  # build the image
docker compose build --no-cache  # Force rebuild without cache
```

### Run [with docker]

Run the script

```bash
docker compose run --rm app python main.py  # run pytest-abra
docker compose run --rm -it app /bin/bash  # use the container interactively
```

# Documentation

After Installation, `abratest` can be called via terminal:

```bash
abratest [arguments]
```

To run successfully, very specific arguments are required. The easiest way to use `abratest` is with the helper script `main.py`. Of yourse you can implement a similar helper script in the language of your liking.

## CLI Interface

The cli command `abratest` has 3 **required arguments**:

- `--env_paths ENV_PATHS`: list of the .env files used in the test
- `--recipes_dir RECIPES_DIR`: directory of all available abra recipes
- `--output_dir OUTPUT_DIR`: target directory for all test results

Furtheremore, there are these optional arguments:

- `--resume`: `abratest` will take the directory in `output_dir` with the most recent creation date and resume the tests there.
- `--session_id SESSION_ID`: Instead of generating a new session_id, the given session_id is used to run or resume the test. Overwrites --resume to False.
- `--debug`: enables playwright debug mode, see docs [here](https://playwright.dev/python/docs/running-tests#debugging-tests)  
- `--timeout`: will overwrite the default playwright timeouts in [ms], see docs [here](https://playwright.dev/python/docs/api/class-browsercontext#browser-context-set-default-timeout) and [here](https://playwright.dev/python/docs/test-assertions#global-timeout). In our current setup, some tests can fail at 10s but will pass with 20s.

### env_paths [required | string]

The .env files provied through the `--env_paths` argument are the most important input to abratest, as they serve as configuration for the tests. One or more paths pointing at .env files can be provided, multiple paths are separated with ";". These .env files are actually the same files that are used to configure the `abra` recipes for instance creation.

To run `abratest` with these `.env` configuration files

- `/path/config_1.env` [of TYPE authentik]
- `/path/config_2.env` [of TYPE wordpress]
- `/path/config_3.env` [of TYPE wordpress]

we simply call

```
abratest --env_paths /path/config_1.env;/path/config_2.env;/path/config_3.env [...other args]
```

Under the hood, each `.env` file in `--env_paths` will create one instance of a `Runner` subclass. Let's say we have `config_2.env` containing `TYPE=wordpress`. This will create an instance of `RunnerWordpress`. This class has to be imported from `recipes_dir`.

### recipes_dir [required | string]

The required argument `--recipes_dir` has to point to the directory, where all the abra recipes are stored. We can call `abratest` with

```
abratest --recipes_dir /path/to/abra/recipes
```

The expected dir structure inside of `recipes_dir` is as follows:

```
DIR recipes_dir [contains abra recipes]
│
├── DIR authentik [authentik recipe]
│   ├── [files from authentik recipe]
│   └── DIR tests_authentik [pytest tests for authentik]
│       ├── FILE runner_authentik.py  # containing RunnerAuthentik class
│       └── [pytest_files]
│
└── DIR wordpress [wordpress recipe]
    ├── [files from wordpress recipe]
    └── DIR tests_wordpress [pytest tests for wordpress]
        ├── FILE runner_wordpress.py  # containing RunnerWordpress class
        └── [pytest_files]
```

The class `RunnerWordpress` will be automatically imported using `importlib` library, which is equivalent to the code below. Note that `recipes_dir` will be added to sys.path automatically for the import to work and that every `Runner` class matching `recipes_dir.rglob("*/runner*.py")` will be imported.

```python
from wordpress.tests_wordpress.runner_wordpress import RunnerWordpress
```

### output_dir [required | string]

Path to the directory where all test outputs are stored (test report, tracebacks, playwright traces etc.)

```
abratest --output_dir /path/to/output
```

# Functionality

Abratest has 3 required inputs, but most importantly the test configuration is done through the .env files given with the --env_paths argument. So let's say we want to run abratest with these 3 .env files:

- `config1.env` [of TYPE authentik]

- config2.env [of TYPE wordpress]

- config3.env [of TYPE wordpress]

Now we run 

```bash
abratest --env_paths path/config1.env;path/config2.env;path/config3.env [...other args]
```


```
abratest -> create Coordinator() instance
└── Coordinator() -> create Runner() subclass instances
    ├── RunnerAuthentik() [based on config1.env, loaded 
    │   │                  from abra/recipes/authentik]
    │   │     # RunnerAuthentik with 3 test files:
    │   ├── RUN pytest path/setup_authentik.py
    │   ├── RUN pytest path/test_authentik_1.py
    │   └── RUN pytest path/test_authentik_2.py
    ├── RunnerWordpress() [based on config2.env, loaded
    │   │                  from abra/recipes/wordpress]
    │   │     # RunnerWordpress with 1 test file
    │   ├── RUN pytest path/setup_authentik.py
    │   ├── RUN pytest path/test_authentik_1.py
    │   └── RUN pytest path/test_authentik_2.py
    └── RunnerWordpress() [based on config3.env, loaded
        │                  from abra/recipes/wordpress]
        │     # RunnerWordpress with 1 test file
        ├── RUN pytest path/setup_authentik.py
        ├── RUN pytest path/test_authentik_1.py
        └── RUN pytest path/test_authentik_2.py


```

Coordinator will take care of the correct order of the tests. In general, tests are placed in one of 3 categories: `setups`, `tests` and `cleanups`. To associate a test with one of these categories, place the Test in the corresponding list of the Runner class, i.e. Runner.setups = [test] or Runner.tests = [test]. The execution order will be.

> [setups] ➔ [tests] ➔ [cleanups]


Furthermore, some `Runner` classes can depend on others. For example, `RunnerWordpress` depends on `RunnerAuthentik`. Therefore, `Coordinator` will make sure that `RunnerAuthentik` runs before `RunnerWordpress`. We will end up with with this order:

| #   | Runner         | Type     |
| --- | -------------- | -------- |
| 1.  | Authentik      | setups   |
| 2.  | Wordpress-1    | setups   |
| 3.  | Wordpress-2    | setups   |
| 4.  | Authentik      | tests    |
| 5.  | Wordpress-1    | tests    |
| 6.  | Wordpress-2    | tests    |
| 7.  | Authentik      | cleanups |
| 8.  | Wordpress-1    | cleanups |
| 9.  | Wordpress-2    | cleanups |


# Create a test suite for a recipe

todo

To understand how a test suite is built, let's have a look at the files

runner_authentik.py  ->  required, defines the Runner subclass (see below)
conftest.py  ->  not required. special file for pytest. is automatically discovered and loaded. convenient place to define fixtures and functions to be used in more than one test routine
setup_authentik.py -> not required. can hold setup routine for authentik, has to be registered in runner_authentik.py

# Create a custom Runner

To comprehend the process of creating a new subclass of `Runner`, let's examine a simplified rendition of the `RunnerWordpress` class. Within it, there exist two setup scripts and two test scripts, one of which operates conditionally.


```python
from pytest_abra import Runner, Test

class RunnerWordpress(Runner):
    env_type = "wordpress"
    dependencies = ["authentik"]
    setups = [
        Test(test_file="setup_wordpress_1.py"),
        Test(test_file="setup_wordpress_2.py"),
    ]
    tests = [
        Test(test_file="test_wordpress.py"),
        Test(condition=condition_function, test_file="test_wordpress_conditional.py"),
    ]
    cleanups = []
```

The signature of condition functions can be seen below. The function takes one `NamedTuple` and returns of type `bool`. You can learn about the contents of the input by looking up the class `ConditionArgs`. Generally speaking, it provides access to all of the .env files, especially the one related to the current Runner.

```python
def condition_function(args: ConditionArgs) -> bool:
    ...
```

## Discovery of `Runners` and `Tests`

- Runners will be discovered, if they are defined in a moduled of name `runner_*.py` including a class of name `Runner*`.

- Tests will be discovered by filename as long as they are placed in the parent dir of `runner_*.py` or in any subdirectory.

```
DIR parent_dir
├── FILE runner_*.py
├── FILE test1.py
└── DIR subdir
    ├── DIR subsubdir
    │   └── test2.py
    └── test3.py
```

# Create custom Tests

The test files are written in the same way as any other pytest test file. The only difference is that pytest-abra provides custom fixtures that make it easy to get the configuration by the provided .env files and to deal with URLS etc.


### Step 1) Add new Test

Create a new testfile `new_test.py` in the same directory or a subdirectory of `runner_wordpress.py`.
Register `new_test.py` as a test in the `RunnerWordpress` class.
Set prevent_skip=True, so that you can run your new test over and over again for debugging, without it being skipped

```python
# runner_wordpress.py
from pytest_abra import Runner, Test

class RunnerWordpress(Runner):
    env_type = "wordpress"
    tests = [
        Test(test_file="working_test.py"),
        Test(test_file="new_test.py", prevent_skip=True),
    ]
```

```python
# new_test.py

def test_new():
    ...
```

### Step 2) Call abratest

Call abratest with `--debug` to enable playwright debug mode and either `--session_id` or `--resume`.

```bash
abratest [required-options] --debug --session_id debug_session
```

This could be done by modifying `main.py`. The first time you run abratest, all tests will be executed as usual. The second time, all tests will be skipped as they have passed already. Only your new test will be run again and again, as the prevent_skip option is enabled. So you can run all tests once and then skip all tests besides your new test you want to debug.

# todo: add example

# Playwright Debug & Codegen

Use playwright debug mode or codegen to create testing code easily by recording browser actions https://playwright.dev/python/docs/codegen

```bash
abratest --debug  # launch your tests in debug mode
playwright codegen demo.playwright.dev/todomvc  # visit given url in codegen mode
```

## Development

```bash
pytest  # test pytest-abra
pytest -m "not slow"  # test pytest-abra without slow tests
pytest  --collect-only  # debug test pytest-abra
docker compose run --rm app pytest  # run pytest-abra
```