# PIL Dataflow · office facts

The office health read-out in three sections that fit one screen: what is running now, what is not protected yet, and what waits on you — one line each. Read-only, password gated. A second edition on pil-dataflow-prometheus.navonsimon.com runs twelve PromQL queries per page load, so every cluster number is live and the query behind it is printed on the page. A third edition runs its own dedicated Grafana on pil-dataflow-grafana.navonsimon.com with one provisioned dashboard over the same Prometheus.

**Status:** Live · Projects · 2026

- **Local checkout:** `/Users/simon/Developer/pil-dataflow` (branch `main`, last commit d0dbc84, 2026-11-08 — Bigger type on all three editions)
- **Repository:** https://github.com/ShimonNavon/pil-dataflow
- **On another machine:** `git clone https://github.com/ShimonNavon/pil-dataflow` — the paths below are Simon's Mac
- **Live:** https://pil-dataflow.navonsimon.com/?k=Nmpil777
- **Prometheus edition:** https://pil-dataflow-prometheus.navonsimon.com/?k=Nmpil777
- **Grafana edition:** https://pil-dataflow-grafana.navonsimon.com/
- **PIL office:** https://projects.navonsimon.com/p/pil-office
- **Where it runs:** The app is a stdlib Python server serving one static page; it reads no office system and holds no office credentials. The register service that will feed the numbers reads a PostgreSQL schema in the databases fleet and accepts run records from the win01 collector with an ingest token.
- **Note:** Read-only oversight · three editions (hand-written, Prometheus, Grafana) · register service next

## Stack

Python 3.12 (stdlib only: password-gated server, no dependencies), Static single-file oversight page (no build step, six tabs), Kubernetes (namespace pil-dataflow, NodePort 30132), Cloudflare Tunnel + edge nginx on ubuntu01, PostgreSQL schema backup in the databases fleet (planned for the register service), Read-only

## Start here

- `site/index.html` — the health page: eight system tiles, each with a state word, one number, one sentence, the time it was checked and a closed details list
- `serve.py` — the cluster app: stdlib server with an in-app password gate, /healthz for probes, and the same file used for the local preview
- `deploy/app.yaml` — namespace pil-dataflow: page and server ConfigMaps, Deployment, Service NodePort 30132
- `deploy/deploy.sh` — builds the manifests with the page and server inlined, creates the access Secret from .env, applies, waits for the rollout
- `deploy/edge.conf` — edge nginx server for pil-dataflow.navonsimon.com proxying the NodePort, GET/HEAD only
- `deploy/route_domain.py` — Cloudflare tunnel route + proxied CNAME, idempotent, keeps every other route
- `tests/test_site.py` — page structure and safety tests: tabs, stages, the eight fields, the register tables, custody, four boundaries, no secrets or client-data keys
- `.github/workflows/ci.yml` — CI: syntax check, pytest, and a check that the page stays self-contained
- `prom/app.py` — the Prometheus edition: stdlib server that runs twelve PromQL queries per request, joins node-exporter series to kube_node_info so only the four cluster nodes count, prints the query behind each fact, and exposes /metrics for its own state
- `deploy/app-prometheus.yaml + deploy/deploy-prometheus.sh + deploy/edge-prometheus.conf` — namespace pil-dataflow-prometheus, NodePort 30133, the app and template as ConfigMaps, edge conf for the second name
- `grafana/dashboard-office-health.json` — the dashboard: stat panels for servers, containers, restarts, tightest disk, volume claims and scrape failures, per-node disk over time, and the three sections as tables
- `grafana/provisioning.yaml` — datasource (the cluster Prometheus) and the dashboard provider, both as ConfigMaps
- `deploy/app-grafana.yaml + deploy/deploy-grafana.sh + deploy/edge-grafana.conf` — namespace pil-dataflow-grafana, a 1Gi PVC, Grafana 11.3 on ubuntu02, NodePort 30134, and an edge conf with basic auth

## Commands

- **Local preview:** `cd ~/Developer/pil-dataflow && python3 serve.py --port 8484`
- **Tests:** `cd ~/Developer/pil-dataflow && .venv/bin/python -m pytest -q`
- **Deploy the app:** `cd ~/Developer/pil-dataflow && ./deploy/deploy.sh`
- **Deploy the app + edge conf:** `cd ~/Developer/pil-dataflow && ./deploy/deploy.sh edge`
- **Route the name:** `cd ~/Developer/pil-dataflow && python3 deploy/route_domain.py`
- **Check the gate:** `curl -s -o /dev/null -w '%{http_code}\n' https://pil-dataflow.navonsimon.com/`
- **Redeploy the Prometheus edition:** `cd ~/Developer/pil-dataflow && ./deploy/deploy-prometheus.sh`
- **Redeploy the Grafana edition:** `cd ~/Developer/pil-dataflow && ./deploy/deploy-grafana.sh`

## Deploy and where it runs

- Kubernetes namespace pil-dataflow (NodePort 30132) behind the Cloudflare Tunnel via the edge nginx on ubuntu01; the access password lives in a cluster Secret built from .env by deploy.sh, so it never enters git.

## Current state

Built 2026-11-08: one-screen health page (eight tiles, plain state words, closed details), namespace pil-dataflow live on ubuntu03 at NodePort 30132, name routed through the tunnel, password gate verified. Register service and the win01 collector push are next. The Prometheus edition is live too, on pil-dataflow-prometheus.navonsimon.com (NodePort 30133), reading the cluster's Prometheus directly. The Grafana edition is live as well, on pil-dataflow-grafana.navonsimon.com — a dedicated Grafana (not the shared monitoring one) with one provisioned dashboard, edge basic auth, anonymous viewer inside.

## Watch out

- The Windows nightly job still writes a 95 GB dump onto the disk it protects, with 33 GB free — the first concrete thing this page is meant to replace.
- All cluster volumes sit on ubuntu04 (37 of 40) and ubuntu03 (3); the other two nodes hold none.
- The page must never grow client documents or identity: counts, sizes, ages and outcomes only.
- The password is in .env (gitignored) and the cluster Secret; rotate by editing .env and re-running deploy.sh.

## Secrets — locations only, never print values

- ~/Developer/pil-dataflow/.env (gitignored): PIL_DATAFLOW_PASSWORD
- cluster Secret pil-dataflow/access (key: password)
- CLOUDFLARE_API_TOKEN in the Mac shell env for the tunnel route
- cluster Secret pil-dataflow-grafana/grafana-admin (Grafana admin 'simon'); edge htpasswd file on ubuntu01 for the same password

## Links

- [README](file:///Users/simon/Developer/pil-dataflow/README.md)
- [AGENTS.md](file:///Users/simon/Developer/pil-dataflow/AGENTS.md)
- [Open](https://pil-dataflow.navonsimon.com/?k=Nmpil777)

## Tags

Python, Kubernetes, Health

---

Brief generated 2026-11-08 from `~/Developer/my-projects/context/`. Canonical URL: https://projects.navonsimon.com/context/pil-dataflow.md
