local-it/dashboard
0
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

docs: update documentation on starting the dev environment

Browse source
This commit is contained in:
Maarten de Waard 2022-05-17 11:03:53 +02:00
parent 8aadfb0860
commit efbc1b21c9
No known key found for this signature in database
GPG key ID: 1D3E893A657CC8DA
4 changed files with 177 additions and 340 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

293
DEVELOPMENT.md
View file

@ -1,293 +0,0 @@
# Development
The main role for this repo is provide Single-Sign-On. The architecture to make
this happen has a lot of moving components. A quick overview:
- Hydra: Hydra is an Identity Provider, or IdP for short. It means connected
applications connect to Hydra to start a session with a user. Hydra provides
the application with the username and other roles/claims for the application.
This is done using the OIDC protocol. Hydra is developed by Ory and has
security as one of their top priorities. Also it is fully OpenSource.
- Login application: If hydra hits a new session/user, it has to know if this
user has access. To do so, the user has to login. Hydra does not support
this, so it will redirect to a login application. This is developed by the
Stackspin team (Greenhost) and part of this repository. It is a Python Flask
application.
Because the security decisions made by kratos (see below), a lot of the
interaction is done in the web-browser, rather then server-side.
This means the login application has an UI component which relies heavily on
JavaScript. As this is a relatively small application, it is based on
traditional Bootstrap + Jquery. This elements the requirement for yet an
other build environment.
- Kratos: This is Identity Manager and contains all the user profiles and
secrets (passwords). Kratos is designed to work mostly between UI (browser)
and kratos directly, over a public API endpoint without an extra server side
component/application. So authentication, form-validation etc, are all handled
by Kratos. Kratos only provides an API and not UI itself.
Kratos provides a Admin API, which is only used from the server-side flask
app to create/delete users.
- MariaDB: All three components need to store data. This is done in a MariaDB
database server. There is once instance, with three databases. As all
databases are very small this will not lead to resource limitation problems.
## Prerequisites
The current login panel is not yet installed available in released versions
of Stackspin. However, this does not prevent us from developing already on the
login panel. Experience with `helm` and `kubernetes` is expected when you follow
this manual.
On your provisioning machine, make sure to checkout:
`git@open.greenhost.net:stackspin/dashboard-backend.git`
Be sure to check out the latest main branch. Or select a more modern branch if you
want to test / install (optional) improvements of login panel.
Once this is all fetched, installation can be done with the following steps:
1. Create an overwrite ConfigMap file:
For local development, we have to configure the endpoint of the application to
be pointing to our development system. In this example, we use `localhost` on
http.
Because of CORS and strict configuration, all needs to end up on the same
system. With modern browser, it even have to run on the same port (at least with
firefox). As we want to mimic the real life setup as much as possible as,
we will do this by running a local proxy. In production this will be handled by
kubernetes ingress configuration.
First we will tell kratos and hydra where to find the right endpoints. An
overview of all relevant end-points:
The endpoints used by the browser are (public accessible)
- `localhost/kratos` -> kratos public API
- `localhost/web` -> login flask app
The endpoint used by the login app/API are:
- `localhost:8000` -> kratos Admin API (only local accessible)
- `localhost/kratos` -> kratos Public API
- `localhost:4445` -> hydra Admin API (only local accessible)
- `localhost:3306` -> MariaDB
To reflect those public endpoints in your cluster, we have to override the
default URLs in the cluster. We do this with a ConfigMap.
It is essential SMTP/e-mail is working during development, so an example
is included on how to override those if SMTP is not working on your
cluster. Otherwise those lines are irrelevant.
Create a file with the following content:
```
---
apiVersion: v1
kind: ConfigMap
metadata:
name: stackspin-dashboard-override
data:
values.yaml: |
kratos:
kratos:
config:
courier:
smtp:
# Kratos enforces the use of STARTTLS. Be sure your SMTP provider
# supports that (if not, it is time to switch providers)
#
# Uncomment and correct below lines if e-mail is not working in your
# cluster
# connection_uri: smtp://user@password@smtp.example.com:25/
# from_address: stackspin-admin@example.com
# For development, we forward all to our local server (or your dev server
# if that is remote)
serve:
public:
base_url: http://localhost/kratos/
selfservice:
default_browser_return_url: http://localhost/web/login
flows:
recovery:
ui_url: http://localhost/web/recovery
login:
ui_url: http://localhost/web/login
settings:
ui_url: http://localhost/web/settings
registration:
ui_url: http://localhost/web/registration
hydra:
hydra:
config:
urls:
# For development we redirect to localhost (or your dev server)
login: http://localhost/web/auth
consent: http://localhost/web/consent
logout: http://localhost/web/logout
```
2. Apply the ConfigMap to your cluster:
```
kubectl apply -n stackspin -f stackspin-dashboard-override.yaml
```
3. Tell flux to reconcile the configuration
Normally flux will do this on some interval. We will tell flux to apply
the override immediately.
```
flux reconcile kustomization core
flux reconcile helmrelease -n stackspin dashboard
```
## Setting up the development environment
1. Setup port redirects
To be able to work on the Login panel, we have to configure our development
system to access all the remote services and endpoints.
A helper script is available in this directory to setup and redirect the
relevant ports locally. It will open ports 8000, 8080, 4445, 5432 to get access
to all APIs:
```
cd project_root/login
./set-ssh-tunnel.sh "stackspin.example.com"
```
(the tunnel goes to the kubernetes node, so *not* to your provisioning machine,
it will uses SSH port forwarding to map ports, as a result you will also have
SSH session to your kubernetes node. Do not close this session, as closing the
session will close the forwarded ports as well)
2. Configure a local proxy
Because of strict CORS headers, we have to map the public kratos API and login
app which we will run locally, with a local proxy.
This can be done with any proxy server, for example with NGINX. Be sure you have
NGINX installed and listening on port 80 locally (`sudo apt-get install nginx`)
should be enough.
Now configure NGINX with this configuration in `/etc/nginx/sites-enabled/default`
```
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
index index.html;
server_name _;
# Flask app
location / {
proxy_pass http://127.0.0.1:5000/;
proxy_redirect default;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
# Kratos Public
location /kratos/ {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect default;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
```
Reload your NGINX:
```
sudo systemctl reload nginx.service
```
3. Run FLASK app
Now it is time to start the flask app. Please make sure you are using python 3 in your environment. And install the required dependencies:
```
cd projectroot/login
pip3 install -r requirements.txt
```
Then copy `source_env` to `source_env.local` and verify if you are happy with
the settings in the `source_env` file:
```
cat source_env.local
export HYDRA_ADMIN_URL=http://localhost:4445
export KRATOS_PUBLIC_URL=http://localhost/api
export KRATOS_ADMIN_URL=http://localhost:8000
export LOGIN_PANEL_URL=http://localhost/web
export DATABASE_URL="mysql+pymysql://stackspin:stackspin@localhost/stackspin"
```
Normally you only need to change the database password if you did not use the
insecure default.
Assuming you did not populate the database yet, run this to populate it:
```
. source_env.local
flask db upgrade
```
If that all looks fine, it is time to add you first user:
```
flask cli user create myemail@example.com
```
And now it is time to start the app:
```
./run.sh
```
If this starts smoothly, you should be ready to go.
## Test your setup
Hydra and kratos are now configured to redirect to localhost when they receive a
request. So to test the setup, you can go to one of your applications (for
example nextcloud), what we expect when you click the login button is the
following:
- Nextcloud redirect to Hydra (on sso.example.com)
- Hydra does not have a session, so ask to authorize on: http://localhost/login/auth
- Kratos does not have a session, so the login panel will ask to login on:
http://localhost/login/login
- You do not have a password setup yet, so you click "recover account", which
should bring you to: http://localhost/login/recovery
- You enter your email address and request a reset token. Check you e-mail. The
email should contain a link to http://localhost/api/self-service/recovery/..
- The link logs you in in kratos and ask you to setup a password. Complete this
step and you account is ready.
We started the flow with trying to reach nextcloud. Because we
did a password recovery in between, this information is lost. If you go again to
nextcloud manually, you should now be logged in automatically.
If you retry this, but now with a password (for example in a privacy window or
by removing you cookies), you should be redirected automatically after login.

176
README.md
View file

@ -1,3 +1,177 @@
# Stackspin dashboard backend # Stackspin dashboard backend
Backend for the [Stacksping dashboard](https://open.greenhost.net/stackspin/dashboard) Backend for the [Stackspin dashboard](https://open.greenhost.net/stackspin/dashboard)
## Login application
Apart from the dashboard backend this repository contains a flask application
that functions as the identity provider, login, consent and logout endpoints
for the OpenID Connect (OIDC) process.
The application relies on the following components:
- **Hydra**: Hydra is an open source OIDC server.
It means applications can connect to Hydra to start a session with a user.
Hydra provides the application with the username
and other roles/claims for the application.
Hydra is developed by Ory and has security as one of their top priorities.
- **Kratos**: This is Identity Manager
and contains all the user profiles and secrets (passwords).
Kratos is designed to work mostly between UI (browser) and kratos directly,
over a public API endpoint.
Authentication, form-validation, etc. are all handled by Kratos.
Kratos only provides an API and not UI itself.
Kratos provides an admin API as well,
which is only used from the server-side flask app to create/delete users.
- **MariaDB**: The login application, as well as Hydra and Kratos, need to store data.
This is done in a MariaDB database server.
There is one instance with three databases.
As all databases are very small we do not foresee resource limitation problems.
If Hydra hits a new session/user, it has to know if this user has access.
To do so, the user has to login through a login application.
This application is developed by the Stackspin team (Greenhost)
and is part of this repository.
It is a Python Flask application
The application follows flows defined in Kratos,
and as such a lot of the interaction is done in the web-browser,
rather then server-side.
As a result,
the login application has a UI component which relies heavily on JavaScript.
As this is a relatively small application,
it is based on traditional Bootstrap + JQuery.
# Development
To develop the Dashboard,
you need a Stackspin cluster that is set up as a development environment.
Follow the instructions [in the dashboard-dev-overrides
repository](https://open.greenhost.net/stackspin/dashboard-dev-overrides#dashboard-dev-overrides)
in order to set up a development-capable cluster.
The end-points for the Dashboard,
as well as Kratos and Hydra, will point to `localhost` in that cluster.
As a result, you can run those components locally, and still log into Stackspin
applications that run on the cluster.
## Setting up the local development environment
After this process is finished, the following will run locally:
- The [dashboard](https://open.greenhost.net/stackspin/dashboard)
- The
[dashboard-backend](https://open.greenhost.net/stackspin/dashboard-backend)
The following will be available on localhost through a proxy and port-forwards:
- Hydra
- Kratos
- The MariaDB database connections
These need to be available locally, because Kratos wants to run on the same
domain as the front-end that serves the login interface.
### 1. Setup port forwards
To be able to work on the dashboard,
we have to configure our development system to access all the remote services and endpoints.
A helper script is available in this directory
to setup and redirect the relevant ports locally.
It will open ports 8000, 8080, 4445, 3306 to get access to all APIs.
To use it, you'll need `kubectl` access to the cluster:
1. Install `kubectl` (available through `snap` on Linux)
2. Download the kubeconfig: `scp root@stackspin.example.com:/etc/rancher/k3s/k3s.yaml kube_config_stackspin.example.com.yaml`
3. Set `kubectl` to use the kubeconfig: `export KUBECONFIG=$PWD/kube_config_stackspin.example.com.yaml`.
4. Test if it works:
```
kubectl get ingress -n stackspin
```
Should return something like:
```
NAME CLASS HOSTS ADDRESS PORTS AGE
hydra-public <none> sso.stackspin.example.com 213.108.110.5 80, 443 39d
dashboard <none> dashboard.stackspin.example.com 213.108.110.5 80, 443 150d
kube-prometheus-stack-grafana <none> grafana.stackspin.example.com 213.108.110.5 80, 443 108d
kube-prometheus-stack-alertmanager <none> alertmanager.stackspin.example.com 213.108.110.5 80, 443 108d
kube-prometheus-stack-prometheus <none> prometheus.stackspin.example.com 213.108.110.5 80, 443 108d
```
5. Run the script to forward ports of the services to your local setup:
```
./set-port-forward.sh
```
As long as the script runs, your connection stays open.
End the script by pressing `ctrl + c` and your port-forwards will end as well.
### 2. Configure a local proxy
Because of strict CORS headers,
we have to map the public Kratos API and login app,
which we will run locally with a local proxy.
This can be done with any proxy server, here we use `nginx`.
Be sure you have NGINX installed and listening on port 80 locally:
`sudo apt-get install nginx` should be enough.
Now configure NGINX with this configuration in `/etc/nginx/sites-enabled/default`
```
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
index index.html;
server_name _;
# Flask app and dashboard-backend
location / {
proxy_pass http://127.0.0.1:5000/;
proxy_redirect default;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
# Kratos APIs
location /kratos/ {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect default;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
```
Reload your NGINX:
```
sudo systemctl reload nginx.service
```
### 3. Run FLASK app
Now it is time to start the flask app.
Please make sure you are using Python 3 in your environment.
And install the required dependencies:
```
pip3 install -r requirements.txt
```
Then copy `run_app.sh` to `run_app.local.sh` and change the secrets defined in it.
You can now start the app by running
```
./run_app.local.sh
```
Lastly, start the [dashboard front-end app](https://open.greenhost.net/stackspin/dashboard/#yarn-start)

3
run_app.sh
View file

@ -11,7 +11,8 @@ else
echo -e "${RED}**********************************************************${NC}" echo -e "${RED}**********************************************************${NC}"
echo -e "${RED}WARNING! It looks like the Kratos Admin port NOT available${NC}" echo -e "${RED}WARNING! It looks like the Kratos Admin port NOT available${NC}"
echo -e "${RED}please run in a seperate terminal: ${NC}" echo -e "${RED}please run in a seperate terminal: ${NC}"
echo -e "${RED}./set-ssh-tunnel.sh init.stackspin.net ${NC}" echo -e "${RED} ${NC}"
echo -e "${RED}./set-port-forward.sh ${NC}"
echo -e "${RED} ${NC}" echo -e "${RED} ${NC}"
echo -e "${RED}We will continue to start the app after 5 seconds. ${NC}" echo -e "${RED}We will continue to start the app after 5 seconds. ${NC}"
echo -e "${RED}**********************************************************${NC}" echo -e "${RED}**********************************************************${NC}"

45
set-ssh-tunnel.sh
View file

@ -1,45 +0,0 @@
#!/bin/bash
host=$1
namespace=$2
if [ "x$host" == "x" ]
then
echo "Please give host of kubernetes master as argument. Optionally a
namespace can be provided. This defaults to 'stackspin'"
echo " "
echo $0 hostname [namespace]
exit 1
fi
if [ "x$namespace" == "x" ]
then
namespace="stackspin"
fi
admin=`ssh $host -lroot kubectl get service -n $namespace |grep kratos-admin | awk '{print $3'}`
public=`ssh $host -lroot kubectl get service -n $namespace |grep kratos-public | awk '{print $3}'`
hydra=`ssh $host -lroot kubectl get service -n $namespace |grep hydra-admin | awk '{print $3}'`
mysql=`ssh $host -lroot kubectl get service -n $namespace |grep single-sign-on-database-maria|grep -v headless | awk '{print $3}'`
if [ "x$admin" == 'x' ] || [ "x$public" == 'x' ] || [ "x$hydra" == 'x' ] || [ "x$mysql" == 'x' ]
then
echo "It seems we where not able find at least one of the remote services"
echo " please make sure that kubectl use the right namespace by default."
echo " normally this is 'stackspin'. If you use a different namespace"
echo " please provide this as second argument"
exit 1
fi
echo "
kratos admin port will be at localhost: 8000
kratos public port will be at localhost: 8080
hydra admin port will be at localhost: 4445
mysql port will be at localhost: 3306
"
ssh -L 8000:$admin:80 -L 8080:$public:80 -L 4445:$hydra:4445 -L 3306:$mysql:3306 root@$host