local-it/e2e_tests
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

rework-output-and-test-logic (#3)

Browse source 
* fix flakey tests in authentik / wordpress

* make it possible to rerun tests partially -> passed will be skipped, failed will be repeated

* improve organization of all outputs (moving, renaming, keeping multiple versions etc.)

* add html reports, replace .txt tracebacks

* combine all html reports into one

* add demo runner with comments for documentation purposes

Reviewed-on: local-it-infrastructure/e2e_tests#3
Co-authored-by: Daniel <d.brummerloh@gmail.com>
Co-committed-by: Daniel <d.brummerloh@gmail.com>
This commit is contained in:
Daniel 2023-11-29 14:14:46 +01:00 • committed by dan
parent d2cd6ba47f
commit 8172f685de
24 changed files with 588 additions and 418 deletions
Download patch file Download diff file Expand all files Collapse all files

57
main.py
View file

@ -2,27 +2,42 @@ import json
import os
from pathlib import Path
from src.utils import get_session_id
from src.wrapper import Wrapper
from loguru import logger
from src.coordinator import Coordinator
from src.dirmanager import DirManager
from src.utils import get_session_id
# ----------------------------- lookup env files ----------------------------- #
# This list of env files is the input to testing framework. each env file
# triggers the execution of one test Runner and provides configuration to the
# tests inside the runner. There can be dependencies, for example wordpress
# requires that authentik ran first to create the admin session and the user
# session. At the moment, wrong ordering results in unsuccessful test
# (wrong ordering would be wordpress env file is before authentik env file).
# The env file list is the input to testing framework. each env file triggers
# the execution of one test Runner and provides configuration to the tests
# inside the runner. There can be dependencies, for example wordpress requires
# that authentik ran first to create the admin session and the user session.
# At the moment, wrong ordering results in unsuccessful test (wrong ordering
# would be wordpress env file is before authentik env file).
ENV_FILES = [
Path("envfiles/login.test.dev.local-it.cloud.env"), # authentik
Path("envfiles/blog.test.dev.local-it.cloud.env"), # wordpress
]
# ----------------------------- define ouptut dir ---------------------------- #
OUTPUT_DIR = Path("./test-output").resolve()
# Set environment variables
# -------------------------- enable playwright debug ------------------------- #
# os.environ["PWDEBUG"] = "1"
# --------------------- load credentials to env variables -------------------- #
cred_file = Path("credentials.json")
with open(cred_file, "r") as f:
CREDENTIALS = json.load(f)
@ -31,7 +46,25 @@ os.environ["ADMIN_USER"] = CREDENTIALS["admin_user"]
os.environ["ADMIN_PASS"] = CREDENTIALS["admin_pass"]
# ----------------------------- define session_id ---------------------------- #
session_id = get_session_id()
wrapper = Wrapper(ENV_FILES, output_dir=OUTPUT_DIR, session_id=session_id)
wrapper.setup_test()
wrapper.run_test()
# session_id = "abc"
# ------------------------------- setup logging ------------------------------ #
DIR = DirManager(output_dir=OUTPUT_DIR, session_id=session_id)
log_file = DIR.RESULTS / "full.log"
logger.add(log_file)
# ---------------------------- initialize and run ---------------------------- #
coordinator = Coordinator(ENV_FILES, output_dir=OUTPUT_DIR, session_id=session_id)
coordinator.setup_test()
coordinator.run_test()
coordinator.combine_html()
coordinator.collect_traces()