250 lines
19 KiB
Markdown
250 lines
19 KiB
Markdown
# Bot Notes — oko project observations
|
||
|
||
Notes from a full read-through of the repo (every source file, configs, tests, git history) on 2026-08-23.
|
||
|
||
---
|
||
|
||
## 1. What this project is
|
||
|
||
**oko** is a self-hosted, multi-stage video surveillance pipeline written in Go (GPL-3.0 licensed). It captures footage from local V4L2 cameras (webcams), runs motion detection and object classification over it, stores clips with metadata, and serves a live MJPEG stream to browsers. It is designed to run as separate cooperating processes connected by raw TCP + `encoding/gob`, with optional AES-256-GCM encryption of all pixel data in transit/at rest. Deployment targets appear to include Raspberry Pi-class hardware ("Pi Zero W friendly" comment, 5 fps defaults).
|
||
|
||
The README is minimal (`# oko` / "The full oko stack"); the real documentation lives in this file (which absorbed `bots_readme.md` — see the appendix for the mofin-stage changelog), `terp/readme.md`, and `coordinator/readme.md`.
|
||
|
||
## 2. Architecture
|
||
|
||
```
|
||
clips (gob/TCP)
|
||
runCam ─────────────────▶ mofin ──── has motion ────▶ terp ──▶ coordinator ──▶ storage/ + MySQL
|
||
│ │ (clip.mp4 + clip.json)
|
||
│ └──── no motion ────────▶ coordinator (bypasses classification)
|
||
└── single frames @ N fps ──▶ livestream-cache ──HTTPS MJPEG──▶ browsers (:8443)
|
||
|
||
oko-run: orchestrator that builds missing binaries and launches nodes listed in config.yaml
|
||
```
|
||
|
||
### Module layout (each directory is its own Go module)
|
||
|
||
| Directory | Role | Key deps |
|
||
|---|---|---|
|
||
| `frame/` | Shared library: core types, crypto, networking, motion, detection, camera capture | gocv v0.43.0, x/crypto |
|
||
| `runCam/` | Camera daemon: capture, clip building, hotplug rescan, livestream send | frame, gocv |
|
||
| `mofin/` | Motion gate between runCam and terp | frame, gocv |
|
||
| `terp/` | Classifier ("interpreter"): Haar cascade detection per frame | frame, gocv |
|
||
| `coordinator/` | Storage sink: mp4+json files, optional MySQL index | frame, gocv, go-sql-driver/mysql |
|
||
| `livestream-cache/` | Live-view backend: TCP ingest → frame cache → HTTPS MJPEG out | frame, yaml (gocv only indirect!) |
|
||
| `oko-run/` | Pipeline supervisor driven by YAML node list | yaml only (no gocv — builds anywhere) |
|
||
|
||
All modules pin `go 1.26.3` and use `replace frame => ../frame`.
|
||
|
||
### Ports / addressing convention
|
||
- `:8083` mofin ingest
|
||
- `:8081` terp ingest
|
||
- `:8082` coordinator ingest
|
||
- `:8085` livestream-cache TCP ingest
|
||
- `:8443` livestream-cache HTTPS MJPEG (`/stream/<user>--<camera>/<idx>`, `/health`)
|
||
|
||
### Data conventions
|
||
- **SourceData**: `<username>--<cameraID>/<v4lIndex>`, e.g. `oko--front/0`. Coordinator splits on first `--`.
|
||
- **GUIDs**: 16 random bytes (crypto/rand) with RFC-4122 version-4 bits set manually (`frame/cameraFunctions.go`).
|
||
- **Timestamps**: `uint64` UnixNano.
|
||
- **Wire format**: one gob-encoded `Clip` per TCP connection for pipeline hops; livestream sender keeps one connection open and streams many single-frame Clips through a shared `gob.Decoder`.
|
||
- **Detection map keys** are classifier identifiers, currently file paths: e.g. `"classifiers/haarcascade_fullbody.xml"` or `"yolo++best.onnx"`.
|
||
|
||
## 3. Stage-by-stage details
|
||
|
||
### `frame` (shared library)
|
||
- `Frame`: raw pixel bytes + dims + gocv MatType + channels + GUID + SourceData + Timestamp + Detections + Errors.
|
||
- `Clip`: parallel slices (`PixelMats [][]byte`, `Guids [][]byte`, `Timestamps []uint64`) plus shared dims/type. `Sublimate()` converts Clip → []Frame views (shares underlying pixel slices).
|
||
- **Crypto** (`frame/frame.go`): AES-256-GCM with PBKDF2-SHA256 key derivation, 600k iterations, per-frame random 16-byte salt + 12-byte nonce prepended. Empty PixelBytes is a no-op. Note: 600k PBKDF2 iterations *per frame* is CPU-heavy — at ~50 captured fps × 10s clips that's ~500 KDF runs per clip on every encrypting hop (and mofin decrypts then re-encrypts everything).
|
||
- **Motion** (`frame/motion.go`): `AbsDiff` → grayscale if multi-channel → binary threshold → `CountNonZero`. Stored as `Comparison{Guid1, Guid2, PixelsChanged, Threshold}` per adjacent pair. `GetHighestMotion()` returns worst-pair percent of changed pixels.
|
||
- **Detection** (`frame/detect.go`):
|
||
- `Detect(classifierLocations)` — parallel Haar cascades via `gocv.NewCascadeClassifier`, certainty hardcoded 1.0.
|
||
- `DetectYolo(modelPath)` — ONNX inference via `gocv.ReadNet`, input 416², conf 0.25; decodes YOLO-style output rows. **Newest commit ("Prelimenary work on integrating YOLO") adds this but nothing calls it yet**, and its helpers `filterNMS`/`iou` exist while `filterNMS` is never invoked from `detect()` — NMS is effectively unwired.
|
||
- Hardcoded default `modelPath = "/home/nolan/Pictures/chickens/fin/chicken-detector/weights/best.onnx"` — reveals the project's origin (chicken-coop monitoring) and embeds an absolute user path.
|
||
- **Networking** (`frame/network.go`): host:port regex validation; `Sendoff`/`SendFrames` return the unusual `(formatErr, dialErr, encodeErr)` triple. `Clip.Send` (used everywhere in the pipeline) does *not* validate addresses. `FrameListener`/`ClipListener` accept loops have no shutdown path (goroutine leak if used).
|
||
- **Capture** (`frame/cameraFunctions.go`): two generations coexist:
|
||
- Legacy: `RunCamera` (recursive self-restart on read failure, capped at depth 10), `generateFrames`, `preframeWrapper`, `CfgData.PassHash` (which gets concatenated into SourceData — would leak a password hash into stored metadata; luckily unused by current binaries).
|
||
- Current: `Capture(ctx, v4lIndex, output)` — context-driven, drops frames via non-blocking channel send when downstream is slow.
|
||
- `frame/utils.go` similarly has old `ScanCams` and newer `Scan`/`isCaptureDevice` (used by runCam).
|
||
- `frame/broadcast.go`: fan-out with drop-on-full policy and drop logging every 1000.
|
||
|
||
### `runCam`
|
||
- Flags for mofin address, cam scan range, clip duration (10s), rescan interval, user/camera-id, passphrase, livestream backend + fps.
|
||
- Main loop = hotplug rebalance ticker: `frame.Scan` every 10s, starts/stops per-camera goroutines tracked by `map[int]context.CancelFunc`.
|
||
- Per camera: capture → `Broadcast` to (a) clip builder and (b) livestream sender.
|
||
- Clip builder buffers frames, flushes on ticker; encrypts per-frame when passphrase set (dropping frames that fail to encrypt); sends whole clip to mofin.
|
||
- Livestream sender (`livestream.go`): holds latest frame under mutex, emits single-frame Clips at FPS ticks, reconnect-with-drop-count logic, optional encryption.
|
||
|
||
### `mofin`
|
||
- Decrypts (if passphrase), `CountChangedPixels(30)`, re-encrypts, routes: no-motion (`GetHighestMotion()==0`) → coordinator directly, else terp. Retry send with exponential backoff (6 tries, cap 500ms), then drop.
|
||
- `-motion-percent` flag is deprecated (gating moved to terp). `-coordinator` empty ⇒ everything goes to terp.
|
||
- Unbounded goroutine-per-clip processing (`procWg.Add` per incoming clip, no semaphore) — a burst could spike memory.
|
||
- Success-send logging is literally a stub: `// ...success logging...` (mofin/main.go:177).
|
||
|
||
### `terp`
|
||
- Loads classifiers via glob (default `classifiers/*.xml`; ships `haarcascade_fullbody.xml`).
|
||
- Recomputes motion itself (`CountChangedPixels(30)`) even though mofin already did — duplicated work by design ("trust but verify"?).
|
||
- Motion-percent gate, then per-frame parallel Haar detection, then **drops clips entirely if no detections** (motion but nothing recognized ⇒ data discarded, not archived).
|
||
- Forwards survivors to coordinator via buffered channel + single sender goroutine.
|
||
|
||
### `coordinator`
|
||
- Writes `storage/<user>/<camera>/<YYYY-MM-DD>/<firstTs>_<firstGuidHex>/clip.mp4` + `clip.json`.
|
||
- VideoWriter codec fallback chain: avc1 → H264 → MJPG.
|
||
- `computeFPS` derives fps from first timestamp delta (clamped 0–120, default 30).
|
||
- Optional MySQL (`clips` + `detections` tables auto-created; FK cascade; per-detection inserts). Degrades gracefully to file-only mode if DB unreachable.
|
||
- Sample committed output exists: 10 real clips from 2026-06-30, 320×240 BGR, ~51fps source, with genuine fullbody-person Haar hits visible in clip.json.
|
||
|
||
### `livestream-cache`
|
||
- YAML-configured (cameras/passphrases keyed by SourceData), TCP ingest :8085, cache of latest frame per camera with TTL janitor (30s default), HTTPS MJPEG server :8443 with TLS 1.2 min, generated gray "offline" placeholder JPEG at quality 60.
|
||
- Pure-Go pixel conversion (BGR→RGBA loop, no OpenCV needed at runtime despite importing gocv transitively).
|
||
- Contains a hand-rolled `flagString` parser duplicating stdlib `flag` behavior.
|
||
- Only takes `PixelMats[0]` of each incoming clip — fine for single-frame livestream clips.
|
||
|
||
### `oko-run`
|
||
- Reads `nodes:` list from YAML; if `<rootDir>/<binary>/<binary>` doesn't exist, runs `go build -o <binary> .` inside that module dir; launches each node once with flags marshaled from a `map[string]interface{}` (note: Go map iteration order ⇒ flag order nondeterministic, harmless here).
|
||
- No restart/supervision of crashed nodes (single-shot `cmd.Run()`); shutdown = signal → CommandContext kill.
|
||
|
||
## 4. Configuration files
|
||
|
||
- `config.yaml` — full 4-node pipeline (cam1 front_door, mofin1, terp1 motion-percent 5, coord1 storage).
|
||
- `cam-only.yaml` — just runCam pointed at livestream backend :8085 (camera-id "front").
|
||
- `config.example.yaml` — documented template for the livestram side (mentions Tailscale for transport).
|
||
- `livestream-cache/config.yaml` — live backend config (passphrase currently empty).
|
||
- Note: oko-run does not read `config.example.yaml`; that one belongs to livestream-cache which loads `config.yaml` in its own CWD — two different `config.yaml` semantics depending on CWD, mildly confusing.
|
||
|
||
## 5. Tests
|
||
|
||
~79 test functions total, table-driven style, mostly in `frame`:
|
||
|
||
| File | Count | Coverage |
|
||
|---|---|---|
|
||
| `frame/frame_test.go` | 19 | ToMat roundtrip, encrypt/decrypt roundtrips & failures, unique salt, Sublimate, CheckLenCorrelations, Clip.Send TCP roundtrip, zero-value fields |
|
||
| `frame/motion_test.go` | 14 | compareTo edge cases (identical/partial/multi-channel/mismatched dims), CountChangedPixels incl. reset behavior |
|
||
| `frame/utils_test.go` | 15 | ScanCams bounds/dedup/parallel-safety, Scan integration, isCaptureDevice; hardware probes skip with `-short` |
|
||
| `mofin/mofin_test.go` | 6 | routing matrix (motion→terp, none→coordinator), passphrase decrypt/re-encrypt integrity, invalid-address timeout |
|
||
| `terp/terp_test.go` | 7 | motion gate, no-detection drop, passphrase wrong-key, timeout |
|
||
| `coordinator/coordinator_test.go` | 11 | splitSource table, computeFPS edges, video writer/json writing |
|
||
| `runCam/runcam_test.go` | 7 | contains(), livestream sender lifecycle/sends/unreachable |
|
||
|
||
Documented invocation: `go test -short ./...` per module (skips V4L hardware probes).
|
||
|
||
## 6. Build health on THIS machine (updated 2026-08-23)
|
||
|
||
- Toolchain: go1.27.0 present on host; modules demand go ≥ 1.26.3 (fine).
|
||
- **Host Arch Linux now ships OpenCV 5.0** (`opencv5.pc`); gocv v0.43.0 (latest) requires OpenCV **4.12** and does not compile against OpenCV 5 (massive API drift: module namespaces, removed `TrackerGOTURN`/`readNetFromCaffe`/etc.) nor against older 4.x distro builds (Ubuntu 24.04's 4.6 lacks `dnn::DataLayout`, `ImagePaddingMode`, new `FaceDetectorYN` overload). Distro with matching version: **Fedora 44 ships OpenCV 4.13 + contrib headers**, which gocv compiles against cleanly.
|
||
- **Solution in place**: `oko-dev` distrobox (Fedora 44) with `opencv-devel`, `gcc-c++`, `golang`, `git`. Use the repo-root wrapper: `./dev.sh <command...>` (e.g. `./dev.sh go test -short ./...`). All seven modules build, vet, and pass `-short` tests inside the box.
|
||
- Binaries link against container libs — run them inside the box too (`./dev.sh ../oko-run/oko-run ...`).
|
||
|
||
## 7. Repo hygiene / issues worth flagging
|
||
|
||
1. **TLS private key committed**: `livestream-cache/key.pem` (+ `cert.pem`) are tracked in git. Self-signed, but keys don't belong in history — should be gitignored/regenerated and rotated if ever reused.
|
||
2. **Build artifacts committed**: compiled binaries `terp/terp` (~large ELF) and `livestream-cache/livestream-cache` (13 MB) are tracked. `.gitignore` lists them (plus runCam/mofin/coordinator/oko-run binaries) but was added after they were committed, so ignore rules don't untrack them. `git rm --cached` needed.
|
||
3. **955 KB runtime log committed**: `livestream-cache/livestream.log`.
|
||
4. **Real surveillance footage committed**: 10 clips (mp4+json) under `coordinator/storage/oko/front/...` — presumably intentional test fixtures, but they bloat the repo and contain identifiable imagery.
|
||
5. **Hardcoded absolute path** in `frame/detect.go` modelPath (`/home/nolan/Pictures/chickens/...`) and a hardcoded retry limit/TODOs about infinite camera-restart loops in legacy `RunCamera`.
|
||
6. **Dead/legacy code in frame**: `RunCamera`, `generateFrames`, `preframeWrapper`, `CfgData`, `ScanCams`, `Sendoff`, `SendFrames`, `FrameListener`, `ClipListener`, `filterNMS`, `DetectYolo` (unwired), `Frame.Errors` field (defined, never populated outside tests).
|
||
7. **API quirks**: `Sendoff`'s `(err, err, err)` triple return; inconsistent address validation (regex in Sendoff/SendFrames, none in Send); success-path logging stub in mofin.
|
||
8. **Performance notes**: PBKDF2-600k per frame encryption is expensive; motion recomputation happens twice (mofin + terp); coordinator computes fps from only the first frame-pair delta.
|
||
9. **Git remote** is a LAN Gitea instance (`http://gitea:3000/nolan/oko`), branch `main` clean and up to date.
|
||
10. Commit messages are informal/casual ("SOmething. I don't remember", "worthless", "I don't know, kid stole my laptop") — history is hard to mine for intent; the appendix below preserves the mofin-stage writeup that compensated for this.
|
||
|
||
## 7b. Fixes made this session (2026-08-23)
|
||
|
||
1. **`livestream-cache/main_test.go` repaired**: fixed corrupted import `" .. /frame"` → `"frame"` and added the missing `fakeClip(n)` helper so the package compiles.
|
||
2. **Live gob protocol bug found & fixed** (`runCam/livestream.go`): `Clip.SendConn` created a fresh `gob.Encoder` per frame; each new Encoder re-sends type definitions, which the cache's shared decoder rejects ("gob: duplicate type received"). The committed `livestream.log` shows **5685** such errors — production livestream only updated once per reconnect cycle. Fix: one persistent `*gob.Encoder` per connection in `StartLivestreamSender`. Test client updated to match (single encoder), test now passes and validates the multi-clip-per-connection pattern.
|
||
|
||
## 7c. Future work: cgo-free capture nodes (TODO before Pi deployment)
|
||
|
||
- runCam's own source never calls gocv — it links OpenCV only transitively via the `frame`
|
||
package, forcing every capture binary to carry libopencv ≥4.12 runtime deps. That rules
|
||
out trivial cross-compilation to Raspberry Pis (Pi OS ships OpenCV 4.6; containers/
|
||
distrobox are a poor fit for headless camera nodes, and impossible on ARMv6 Zero W).
|
||
- **Planned fix**: split wire/data types (`Frame`, `Clip`, gob transport, AES-GCM crypto)
|
||
out of `frame` into a cgo-free subpackage (e.g. `framewire`); leave all gocv/CV code in
|
||
`frame`. Then build capture binaries with `CGO_ENABLED=0 GOARM=6/7` or `GOARCH=arm64` —
|
||
static single-binary deploys with zero runtime requirements.
|
||
- Mechanical refactor (~1h): update imports across runCam (+ tests), adjust `replace`
|
||
directives in all go.mod files, keep mofin/terp/coordinator on full `frame`.
|
||
|
||
## 8. Where things seem headed
|
||
|
||
- YOLO ONNX integration is mid-flight (`DetectYolo` + decode logic landed, wiring/NMS/class-name mapping still missing; Detection struct already has version/postfix fields seemingly designed for richer classifier identity).
|
||
- Livestream stack (runCam sender + livestream-cache + browser MJPEG over Tailscale) is the newest subsystem and looks near-complete.
|
||
- MySQL indexing in coordinator is complete but optional; the file hierarchy remains the source of truth.
|
||
|
||
---
|
||
|
||
## Appendix — Historical changelog: mofin motion-detection stage
|
||
|
||
*(Absorbed from `bots_readme.md`, written by the session that added mofin. Note it
|
||
predates livestream-cache and oko-run — "five modules" then, seven now.)*
|
||
|
||
### Pipeline before / after
|
||
|
||
```
|
||
Old: runCam ──TCP──▶ terp ──TCP──▶ coordinator
|
||
|
||
New: runCam ──TCP──▶ mofin ──── has motion ────▶ terp ──▶ coordinator
|
||
└──── no motion ─────▶ coordinator (bypasses classification)
|
||
```
|
||
|
||
`mofin` runs pixel-level motion detection on each clip. Clips with motion are forwarded
|
||
to `terp` for classification; clips without motion go straight to `coordinator` for storage.
|
||
|
||
### Files created
|
||
|
||
**`frame/motion.go`**
|
||
- `Comparison{Guid1, Guid2 []byte; PixelsChanged int; Threshold uint8}` — result of one
|
||
adjacent-frame comparison.
|
||
- `(*Frame).compareTo(next *Frame, threshold uint8) (int, error)` — ToMat both frames →
|
||
`AbsDiff` → grayscale if multi-channel → binary `Threshold` → `CountNonZero`.
|
||
- `(*Clip).CountChangedPixels(threshold uint8)` — iterates adjacent pairs, appends a
|
||
`Comparison` per pair, logs errors from lightweight Frame views (pre-existing
|
||
Comparisons are cleared first).
|
||
|
||
**`mofin/main.go`**
|
||
|
||
| Flag | Default | Purpose |
|
||
|---|---|---|
|
||
| `-listen` | `:8083` | TCP address to receive clips from runCam |
|
||
| `-terp` | `localhost:8081` | Forward address for clips with motion |
|
||
| `-coordinator` | `localhost:8082` | Forward address for clips without motion (empty ⇒ all to terp) |
|
||
| `-threshold` | `5000` | Minimum `PixelsChanged` to count as motion |
|
||
| `-passphrase` | `""` | Decrypt/re-encrypt passphrase |
|
||
|
||
Concurrency: goroutine per incoming clip tracked by `procWg`; shutdown closes listener,
|
||
drains decodes via `acceptWg`, closes `clipChan`, waits on `procWg`.
|
||
`processClip` flow: decrypt → `CountChangedPixels(30)` → gate on any
|
||
`Comparison.PixelsChanged >= threshold` → re-encrypt → forward to terp or coordinator.
|
||
|
||
**`mofin/go.mod`** — standard module + `replace frame => ../frame`.
|
||
|
||
### Files modified
|
||
|
||
- **`frame/frame.go`**: added `Frame.Errors []string` and `Clip.Comparisons []Comparison`;
|
||
removed a stale duplicate `Detect` declaration.
|
||
- **`runCam/main.go`**: renamed `-terp` flag to `-mofin` (`localhost:8083`);
|
||
`terpAddr` → `mofinAddr` throughout.
|
||
- **`config.yaml`**: inserted `mofin1` node between `cam1` and `terp1`;
|
||
`cam1` repointed at mofin.
|
||
|
||
### Test inventory at time of writing
|
||
|
||
`frame/frame_test.go` — 19 tests: ToMat roundtrip/non-nil/empty-bytes;
|
||
Encrypt/Decrypt roundtrip, wrong-passphrase, empty no-op, truncated input, unique salt
|
||
per call; Sublimate basic/empty/detections-carryover; CheckLenCorrelations
|
||
match/PixelMats-Guids mismatch/Timestamps mismatch/empty; Clip.Send TCP roundtrip +
|
||
invalid address; zero-value Frame/Clip fields.
|
||
|
||
`frame/motion_test.go` — 14 tests: compareTo identical/all-changed/partial/threshold
|
||
filtering/multi-channel/dimension-mismatch/channel-mismatch/identical-pixel-data;
|
||
CountChangedPixels basic/single-frame/empty/reset-behavior/threshold-parameter;
|
||
Comparison struct fields.
|
||
|
||
`frame/utils_test.go` — 16 tests: ScanCams bounds/dedup/parallel-safety, Scan
|
||
integration, isCaptureDevice, pure-logic helpers; camera-dependent cases skip under
|
||
`-short`.
|
||
|
||
Plus per-module suites (mofin routing matrix, terp gates, coordinator writers, runCam
|
||
lifecycle) — full current counts in §5. Invocation: `go test -short ./...` per module.
|