* add full integration test of cli / pytest_abra with all tests
* save path of runner_*.py in runner subclass to improve test discovery -> allows for same test name in two different runners
* reorganize output dir names
* use URL fixture everywhere
* rework coordinator interface
* add --session_id to cli args
* add log results table
* plenty of refactoring
* add assert messages
* add plenty of tests
* add /docs dir with plenty of documentation
* fix authentik setup
* add authentik cleanup, remove test user
* add random test user credential generation and integrate into test routine. random creds are saved to STATES
Reviewed-on: local-it-infrastructure/e2e_tests#16
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* add tests for merge_html_reports function
Reviewed-on: local-it-infrastructure/e2e_tests#15
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* add fixture to make api calls with authentification
* add authentik test that checks the status of all blueprints
* add option to append any kind of data to html report
Reviewed-on: local-it-infrastructure/e2e_tests#14
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* general project refactoring
* various small improvements
* improve imap fixture with helper functions and typing
* add wordpress send email setup
* add wordpress receive email test
* add various documentation
Reviewed-on: local-it-infrastructure/e2e_tests#13
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* add functionality to --resume flag. latest test will resume by running failed tests again
* fix nextcloud setup -> all tests passing
* fix expect timeout by moving it to its own fixture
Reviewed-on: local-it-infrastructure/e2e_tests#12
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
Before, recipes_dir had to be present in the importable paths of the python interpreter. This was solved by adding it to the PYTHONPATH env var. Now, abratest handles this by itself.
Reviewed-on: local-it-infrastructure/e2e_tests#11
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* fix docker
-> use "pip install -e ." in installation
-> add symlinks in docker image
-> docker / non docker execution can run same main.py + cli
* remove sh scripts
* remove requirements.txt
Reviewed-on: local-it-infrastructure/e2e_tests#10
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* turn repo into installable package (pip install -e .)
* add hatchling build packend
* call it pytest-abra
* add pytest entrypoint, so that it gets loaded automatically if installed (and pytest is run)
* make fixtures optional, so that pytest can still be used in other context
* add cli script -> you can now directly run "pytest-abra" in console
Reviewed-on: local-it-infrastructure/e2e_tests#9
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
-> all program states available
Reviewed-on: local-it-infrastructure/e2e_tests#8
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* make it so that the actual tests can be moved anywhere, for example in abra recipe repos
-> major refactoring with pytest test discovery magic
* create RUNNER_DICT dynamically with importlib
-> none of the tests are hardcoded, more tests can be added by placing a folder
* autoload fixtures with pytest plugins
* add URL fixture to navigate on web pages. Includes url parser based on python urllib to generate correct links
* fix nextcloud setups and tests
* add email groundwork with imbox
Reviewed-on: local-it-infrastructure/e2e_tests#7
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* add EnvManager class
* holds all functions that are env file related
* integrates runner dependency resolution
* add integration and unit tests for EnvManager
Reviewed-on: local-it-infrastructure/e2e_tests#6
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* refactoring and rework: runner now has setups / tests / cleanups as lists
* add nextcloud runner
* add email testing prototype with imap fixture
* add dependency resolution (sort env files in input so that test order is correct)
Reviewed-on: local-it-infrastructure/e2e_tests#5
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
Before, a test had only access to it's own env file / configuration (wordpress could see wordpress env file). Now, all env files are available. Wordpress test can also read authentik env file, for example to get the authentik domain.
Reviewed-on: local-it-infrastructure/e2e_tests#4
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* fix flakey tests in authentik / wordpress
* make it possible to rerun tests partially -> passed will be skipped, failed will be repeated
* improve organization of all outputs (moving, renaming, keeping multiple versions etc.)
* add html reports, replace .txt tracebacks
* combine all html reports into one
* add demo runner with comments for documentation purposes
Reviewed-on: local-it-infrastructure/e2e_tests#3
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
* authentik sessions created successfully during setup without breaking tracing
* setup works on EN and DE localization by using regex patterns
* automated tracing with pytest --trace option, manual hook no longer needed
Reviewed-on: local-it-infrastructure/e2e_tests#2
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
