local-it/e2e_tests
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

various (#16)

Browse source 
* add full integration test of cli / pytest_abra with all tests

* save path of runner_*.py in runner subclass to improve test discovery -> allows for same test name in two different runners

* reorganize output dir names

* use URL fixture everywhere

* rework coordinator interface

* add --session_id to cli args

* add log results table

* plenty of refactoring

* add assert messages

* add plenty of tests

* add /docs dir with plenty of documentation

* fix authentik setup

* add authentik cleanup, remove test user

* add random test user credential generation and integrate into test routine. random creds are saved to STATES

Reviewed-on: local-it-infrastructure/e2e_tests#16
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
This commit is contained in:
Daniel 2023-12-14 14:03:58 +01:00 • committed by dan
parent 016b88a68d
commit 2dd765a974
36 changed files with 1145 additions and 432 deletions
Download patch file Download diff file Expand all files Collapse all files

104
prototyping/documentation.md
View file

@ -1,104 +0,0 @@
# RUN
Abratest has 3 required inputs, but most importantly the test configuration is done through the .env files given with the --env_paths argument. So let's say we want to run abratest with these 3 .env files:
- config1.env [of TYPE authentik]
- config2.env [of TYPE wordpress]
- config3.env [of TYPE wordpress]
Now we run
```bash
abratest --env_paths path/config1.env;path/config2.env;path/config3.env [...other args]
```
```
abratest -> create Coordinator() instance
└── Coordinator() -> create Runner() subclass instances
├── RunnerAuthentik() [based on config1.env, loaded
│ │ from abra/recipes/authentik]
│ │ # RunnerAuthentik with 3 test files:
│ ├── RUN pytest path/setup_authentik.py
│ ├── RUN pytest path/test_authentik_1.py
│ └── RUN pytest path/test_authentik_2.py
├── RunnerWordpress() [based on config2.env, loaded
│ │ from abra/recipes/wordpress]
│ │ # RunnerWordpress with 1 test file
│ ├── RUN pytest path/setup_authentik.py
│ ├── RUN pytest path/test_authentik_1.py
│ └── RUN pytest path/test_authentik_2.py
└── RunnerWordpress() [based on config3.env, loaded
│ from abra/recipes/wordpress]
│ # RunnerWordpress with 1 test file
├── RUN pytest path/setup_authentik.py
├── RUN pytest path/test_authentik_1.py
└── RUN pytest path/test_authentik_2.py
```
Coordinator will take care of the correct order of the tests. In general, tests are placed in one of 3 categories: `setups`, `tests` and `cleanups`. To associate a test with one of these categories, place the Test in the corresponding list of the Runner class, i.e. Runner.setups = [test] or Runner.tests = [test]. The execution order will be.
> [setups] ➔ [tests] ➔ [cleanups]
Furthermore, some `Runner` classes can depend on others. For example, `RunnerWordpress` depends on `RunnerAuthentik`. Therefore, `Coordinator` will make sure that `RunnerAuthentik` runs before `RunnerWordpress`. We will end up with with this order:
| # | Runner | Type |
| --- | -------------- | -------- |
| 1. | Authentik | setups |
| 2. | Wordpress-1 | setups |
| 3. | Wordpress-2 | setups |
| 4. | Authentik | tests |
| 5. | Wordpress-1 | tests |
| 6. | Wordpress-2 | tests |
| 7. | Authentik | cleanups |
| 8. | Wordpress-1 | cleanups |
| 9. | Wordpress-2 | cleanups |
# Create a custom Runner
To comprehend the process of creating a new subclass of `Runner`, let's examine a simplified rendition of the `RunnerWordpress` class. Within it, there exist two setup scripts and two test scripts, one of which operates conditionally.
```python
from pytest_abra import Runner, Test
class RunnerWordpress(Runner):
env_type = "wordpress"
dependencies = ["authentik"]
setups = [
Test(test_file="setup_wordpress_1.py"),
Test(test_file="setup_wordpress_2.py"),
]
tests = [
Test(test_file="test_wordpress.py"),
Test(condition=condition_function, test_file="test_wordpress_conditional.py"),
]
cleanups = []
```
The signature of condition functions can be seen below. The function takes one `NamedTuple` and returns of type `bool`. You can learn about the contents of the input by looking up the class `ConditionArgs`. Generally speaking, it provides access to all of the .env files, especially the one related to the current Runner.
```python
def condition_function(args: ConditionArgs) -> bool:
...
```
# Create custom Tests
The test files are written in the same way as any other pytest test file. The only difference is that pytest-abra provides custom fixtures that make it easy to get the configuration by the provided .env files and to deal with URLS etc.
# todo: add example