nolan/oko_public
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2998b478ca2a1da3f448604786dcfc98140e0a9c
oko_public/bot_notes.md
T
nolan 2998b478ca INIT
2026-09-09 21:44:05 -05:00

19 KiB
Raw Blame History

Bot Notes — oko project observations

Notes from a full read-through of the repo (every source file, configs, tests, git history) on 2026-08-23.

1. What this project is

oko is a self-hosted, multi-stage video surveillance pipeline written in Go (GPL-3.0 licensed). It captures footage from local V4L2 cameras (webcams), runs motion detection and object classification over it, stores clips with metadata, and serves a live MJPEG stream to browsers. It is designed to run as separate cooperating processes connected by raw TCP + encoding/gob, with optional AES-256-GCM encryption of all pixel data in transit/at rest. Deployment targets appear to include Raspberry Pi-class hardware ("Pi Zero W friendly" comment, 5 fps defaults).

The README is minimal (# oko / "The full oko stack"); the real documentation lives in this file (which absorbed bots_readme.md — see the appendix for the mofin-stage changelog), terp/readme.md, and coordinator/readme.md.

2. Architecture

                        clips (gob/TCP)
runCam ─────────────────▶ mofin ──── has motion ────▶ terp ──▶ coordinator ──▶ storage/ + MySQL
  │                       │                                                    (clip.mp4 + clip.json)
  │                       └──── no motion ────────▶ coordinator (bypasses classification)
  └── single frames @ N fps ──▶ livestream-cache ──HTTPS MJPEG──▶ browsers (:8443)

oko-run: orchestrator that builds missing binaries and launches nodes listed in config.yaml

Module layout (each directory is its own Go module)

Directory Role Key deps
frame/ Shared library: core types, crypto, networking, motion, detection, camera capture gocv v0.43.0, x/crypto
runCam/ Camera daemon: capture, clip building, hotplug rescan, livestream send frame, gocv
mofin/ Motion gate between runCam and terp frame, gocv
terp/ Classifier ("interpreter"): Haar cascade detection per frame frame, gocv
coordinator/ Storage sink: mp4+json files, optional MySQL index frame, gocv, go-sql-driver/mysql
livestream-cache/ Live-view backend: TCP ingest → frame cache → HTTPS MJPEG out frame, yaml (gocv only indirect!)
oko-run/ Pipeline supervisor driven by YAML node list yaml only (no gocv — builds anywhere)

All modules pin go 1.26.3 and use replace frame => ../frame.

Ports / addressing convention

  • :8083 mofin ingest
  • :8081 terp ingest
  • :8082 coordinator ingest
  • :8085 livestream-cache TCP ingest
  • :8443 livestream-cache HTTPS MJPEG (/stream/<user>--<camera>/<idx>, /health)

Data conventions

  • SourceData: <username>--<cameraID>/<v4lIndex>, e.g. oko--front/0. Coordinator splits on first --.
  • GUIDs: 16 random bytes (crypto/rand) with RFC-4122 version-4 bits set manually (frame/cameraFunctions.go).
  • Timestamps: uint64 UnixNano.
  • Wire format: one gob-encoded Clip per TCP connection for pipeline hops; livestream sender keeps one connection open and streams many single-frame Clips through a shared gob.Decoder.
  • Detection map keys are classifier identifiers, currently file paths: e.g. "classifiers/haarcascade_fullbody.xml" or "yolo++best.onnx".

3. Stage-by-stage details

frame (shared library)

  • Frame: raw pixel bytes + dims + gocv MatType + channels + GUID + SourceData + Timestamp + Detections + Errors.
  • Clip: parallel slices (PixelMats [][]byte, Guids [][]byte, Timestamps []uint64) plus shared dims/type. Sublimate() converts Clip → []Frame views (shares underlying pixel slices).
  • Crypto (frame/frame.go): AES-256-GCM with PBKDF2-SHA256 key derivation, 600k iterations, per-frame random 16-byte salt + 12-byte nonce prepended. Empty PixelBytes is a no-op. Note: 600k PBKDF2 iterations per frame is CPU-heavy — at ~50 captured fps × 10s clips that's ~500 KDF runs per clip on every encrypting hop (and mofin decrypts then re-encrypts everything).
  • Motion (frame/motion.go): AbsDiff → grayscale if multi-channel → binary threshold → CountNonZero. Stored as Comparison{Guid1, Guid2, PixelsChanged, Threshold} per adjacent pair. GetHighestMotion() returns worst-pair percent of changed pixels.
  • Detection (frame/detect.go):
    • Detect(classifierLocations) — parallel Haar cascades via gocv.NewCascadeClassifier, certainty hardcoded 1.0.
    • DetectYolo(modelPath) — ONNX inference via gocv.ReadNet, input 416², conf 0.25; decodes YOLO-style output rows. Newest commit ("Prelimenary work on integrating YOLO") adds this but nothing calls it yet, and its helpers filterNMS/iou exist while filterNMS is never invoked from detect() — NMS is effectively unwired.
    • Hardcoded default modelPath = "/home/nolan/Pictures/chickens/fin/chicken-detector/weights/best.onnx" — reveals the project's origin (chicken-coop monitoring) and embeds an absolute user path.
  • Networking (frame/network.go): host:port regex validation; Sendoff/SendFrames return the unusual (formatErr, dialErr, encodeErr) triple. Clip.Send (used everywhere in the pipeline) does not validate addresses. FrameListener/ClipListener accept loops have no shutdown path (goroutine leak if used).
  • Capture (frame/cameraFunctions.go): two generations coexist:
    • Legacy: RunCamera (recursive self-restart on read failure, capped at depth 10), generateFrames, preframeWrapper, CfgData.PassHash (which gets concatenated into SourceData — would leak a password hash into stored metadata; luckily unused by current binaries).
    • Current: Capture(ctx, v4lIndex, output) — context-driven, drops frames via non-blocking channel send when downstream is slow.
    • frame/utils.go similarly has old ScanCams and newer Scan/isCaptureDevice (used by runCam).
  • frame/broadcast.go: fan-out with drop-on-full policy and drop logging every 1000.

runCam

  • Flags for mofin address, cam scan range, clip duration (10s), rescan interval, user/camera-id, passphrase, livestream backend + fps.
  • Main loop = hotplug rebalance ticker: frame.Scan every 10s, starts/stops per-camera goroutines tracked by map[int]context.CancelFunc.
  • Per camera: capture → Broadcast to (a) clip builder and (b) livestream sender.
  • Clip builder buffers frames, flushes on ticker; encrypts per-frame when passphrase set (dropping frames that fail to encrypt); sends whole clip to mofin.
  • Livestream sender (livestream.go): holds latest frame under mutex, emits single-frame Clips at FPS ticks, reconnect-with-drop-count logic, optional encryption.

mofin

  • Decrypts (if passphrase), CountChangedPixels(30), re-encrypts, routes: no-motion (GetHighestMotion()==0) → coordinator directly, else terp. Retry send with exponential backoff (6 tries, cap 500ms), then drop.
  • -motion-percent flag is deprecated (gating moved to terp). -coordinator empty ⇒ everything goes to terp.
  • Unbounded goroutine-per-clip processing (procWg.Add per incoming clip, no semaphore) — a burst could spike memory.
  • Success-send logging is literally a stub: // ...success logging... (mofin/main.go:177).

terp

  • Loads classifiers via glob (default classifiers/*.xml; ships haarcascade_fullbody.xml).
  • Recomputes motion itself (CountChangedPixels(30)) even though mofin already did — duplicated work by design ("trust but verify"?).
  • Motion-percent gate, then per-frame parallel Haar detection, then drops clips entirely if no detections (motion but nothing recognized ⇒ data discarded, not archived).
  • Forwards survivors to coordinator via buffered channel + single sender goroutine.

coordinator

  • Writes storage/<user>/<camera>/<YYYY-MM-DD>/<firstTs>_<firstGuidHex>/clip.mp4 + clip.json.
  • VideoWriter codec fallback chain: avc1 → H264 → MJPG.
  • computeFPS derives fps from first timestamp delta (clamped 0–120, default 30).
  • Optional MySQL (clips + detections tables auto-created; FK cascade; per-detection inserts). Degrades gracefully to file-only mode if DB unreachable.
  • Sample committed output exists: 10 real clips from 2026-06-30, 320×240 BGR, ~51fps source, with genuine fullbody-person Haar hits visible in clip.json.

livestream-cache

  • YAML-configured (cameras/passphrases keyed by SourceData), TCP ingest :8085, cache of latest frame per camera with TTL janitor (30s default), HTTPS MJPEG server :8443 with TLS 1.2 min, generated gray "offline" placeholder JPEG at quality 60.
  • Pure-Go pixel conversion (BGR→RGBA loop, no OpenCV needed at runtime despite importing gocv transitively).
  • Contains a hand-rolled flagString parser duplicating stdlib flag behavior.
  • Only takes PixelMats[0] of each incoming clip — fine for single-frame livestream clips.

oko-run

  • Reads nodes: list from YAML; if <rootDir>/<binary>/<binary> doesn't exist, runs go build -o <binary> . inside that module dir; launches each node once with flags marshaled from a map[string]interface{} (note: Go map iteration order ⇒ flag order nondeterministic, harmless here).
  • No restart/supervision of crashed nodes (single-shot cmd.Run()); shutdown = signal → CommandContext kill.

4. Configuration files

  • config.yaml — full 4-node pipeline (cam1 front_door, mofin1, terp1 motion-percent 5, coord1 storage).
  • cam-only.yaml — just runCam pointed at livestream backend :8085 (camera-id "front").
  • config.example.yaml — documented template for the livestram side (mentions Tailscale for transport).
  • livestream-cache/config.yaml — live backend config (passphrase currently empty).
  • Note: oko-run does not read config.example.yaml; that one belongs to livestream-cache which loads config.yaml in its own CWD — two different config.yaml semantics depending on CWD, mildly confusing.

5. Tests

~79 test functions total, table-driven style, mostly in frame:

File Count Coverage
frame/frame_test.go 19 ToMat roundtrip, encrypt/decrypt roundtrips & failures, unique salt, Sublimate, CheckLenCorrelations, Clip.Send TCP roundtrip, zero-value fields
frame/motion_test.go 14 compareTo edge cases (identical/partial/multi-channel/mismatched dims), CountChangedPixels incl. reset behavior
frame/utils_test.go 15 ScanCams bounds/dedup/parallel-safety, Scan integration, isCaptureDevice; hardware probes skip with -short
mofin/mofin_test.go 6 routing matrix (motion→terp, none→coordinator), passphrase decrypt/re-encrypt integrity, invalid-address timeout
terp/terp_test.go 7 motion gate, no-detection drop, passphrase wrong-key, timeout
coordinator/coordinator_test.go 11 splitSource table, computeFPS edges, video writer/json writing
runCam/runcam_test.go 7 contains(), livestream sender lifecycle/sends/unreachable

Documented invocation: go test -short ./... per module (skips V4L hardware probes).

6. Build health on THIS machine (updated 2026-08-23)

  • Toolchain: go1.27.0 present on host; modules demand go ≥ 1.26.3 (fine).
  • Host Arch Linux now ships OpenCV 5.0 (opencv5.pc); gocv v0.43.0 (latest) requires OpenCV 4.12 and does not compile against OpenCV 5 (massive API drift: module namespaces, removed TrackerGOTURN/readNetFromCaffe/etc.) nor against older 4.x distro builds (Ubuntu 24.04's 4.6 lacks dnn::DataLayout, ImagePaddingMode, new FaceDetectorYN overload). Distro with matching version: Fedora 44 ships OpenCV 4.13 + contrib headers, which gocv compiles against cleanly.
  • Solution in place: oko-dev distrobox (Fedora 44) with opencv-devel, gcc-c++, golang, git. Use the repo-root wrapper: ./dev.sh <command...> (e.g. ./dev.sh go test -short ./...). All seven modules build, vet, and pass -short tests inside the box.
  • Binaries link against container libs — run them inside the box too (./dev.sh ../oko-run/oko-run ...).

7. Repo hygiene / issues worth flagging

  1. TLS private key committed: livestream-cache/key.pem (+ cert.pem) are tracked in git. Self-signed, but keys don't belong in history — should be gitignored/regenerated and rotated if ever reused.
  2. Build artifacts committed: compiled binaries terp/terp (~large ELF) and livestream-cache/livestream-cache (13 MB) are tracked. .gitignore lists them (plus runCam/mofin/coordinator/oko-run binaries) but was added after they were committed, so ignore rules don't untrack them. git rm --cached needed.
  3. 955 KB runtime log committed: livestream-cache/livestream.log.
  4. Real surveillance footage committed: 10 clips (mp4+json) under coordinator/storage/oko/front/... — presumably intentional test fixtures, but they bloat the repo and contain identifiable imagery.
  5. Hardcoded absolute path in frame/detect.go modelPath (/home/nolan/Pictures/chickens/...) and a hardcoded retry limit/TODOs about infinite camera-restart loops in legacy RunCamera.
  6. Dead/legacy code in frame: RunCamera, generateFrames, preframeWrapper, CfgData, ScanCams, Sendoff, SendFrames, FrameListener, ClipListener, filterNMS, DetectYolo (unwired), Frame.Errors field (defined, never populated outside tests).
  7. API quirks: Sendoff's (err, err, err) triple return; inconsistent address validation (regex in Sendoff/SendFrames, none in Send); success-path logging stub in mofin.
  8. Performance notes: PBKDF2-600k per frame encryption is expensive; motion recomputation happens twice (mofin + terp); coordinator computes fps from only the first frame-pair delta.
  9. Git remote is a LAN Gitea instance (http://gitea:3000/nolan/oko), branch main clean and up to date.
  10. Commit messages are informal/casual ("SOmething. I don't remember", "worthless", "I don't know, kid stole my laptop") — history is hard to mine for intent; the appendix below preserves the mofin-stage writeup that compensated for this.

7b. Fixes made this session (2026-08-23)

  1. livestream-cache/main_test.go repaired: fixed corrupted import " .. /frame" → "frame" and added the missing fakeClip(n) helper so the package compiles.
  2. Live gob protocol bug found & fixed (runCam/livestream.go): Clip.SendConn created a fresh gob.Encoder per frame; each new Encoder re-sends type definitions, which the cache's shared decoder rejects ("gob: duplicate type received"). The committed livestream.log shows 5685 such errors — production livestream only updated once per reconnect cycle. Fix: one persistent *gob.Encoder per connection in StartLivestreamSender. Test client updated to match (single encoder), test now passes and validates the multi-clip-per-connection pattern.

7c. Future work: cgo-free capture nodes (TODO before Pi deployment)

  • runCam's own source never calls gocv — it links OpenCV only transitively via the frame package, forcing every capture binary to carry libopencv ≥4.12 runtime deps. That rules out trivial cross-compilation to Raspberry Pis (Pi OS ships OpenCV 4.6; containers/ distrobox are a poor fit for headless camera nodes, and impossible on ARMv6 Zero W).
  • Planned fix: split wire/data types (Frame, Clip, gob transport, AES-GCM crypto) out of frame into a cgo-free subpackage (e.g. framewire); leave all gocv/CV code in frame. Then build capture binaries with CGO_ENABLED=0 GOARM=6/7 or GOARCH=arm64 — static single-binary deploys with zero runtime requirements.
  • Mechanical refactor (~1h): update imports across runCam (+ tests), adjust replace directives in all go.mod files, keep mofin/terp/coordinator on full frame.

8. Where things seem headed

  • YOLO ONNX integration is mid-flight (DetectYolo + decode logic landed, wiring/NMS/class-name mapping still missing; Detection struct already has version/postfix fields seemingly designed for richer classifier identity).
  • Livestream stack (runCam sender + livestream-cache + browser MJPEG over Tailscale) is the newest subsystem and looks near-complete.
  • MySQL indexing in coordinator is complete but optional; the file hierarchy remains the source of truth.

Appendix — Historical changelog: mofin motion-detection stage

(Absorbed from bots_readme.md, written by the session that added mofin. Note it predates livestream-cache and oko-run — "five modules" then, seven now.)

Pipeline before / after

Old: runCam ──TCP──▶ terp ──TCP──▶ coordinator

New: runCam ──TCP──▶ mofin ──── has motion ────▶ terp ──▶ coordinator
                          └──── no motion ─────▶ coordinator (bypasses classification)

mofin runs pixel-level motion detection on each clip. Clips with motion are forwarded to terp for classification; clips without motion go straight to coordinator for storage.

Files created

frame/motion.go

  • Comparison{Guid1, Guid2 []byte; PixelsChanged int; Threshold uint8} — result of one adjacent-frame comparison.
  • (*Frame).compareTo(next *Frame, threshold uint8) (int, error) — ToMat both frames → AbsDiff → grayscale if multi-channel → binary Threshold → CountNonZero.
  • (*Clip).CountChangedPixels(threshold uint8) — iterates adjacent pairs, appends a Comparison per pair, logs errors from lightweight Frame views (pre-existing Comparisons are cleared first).

mofin/main.go

Flag Default Purpose
-listen :8083 TCP address to receive clips from runCam
-terp localhost:8081 Forward address for clips with motion
-coordinator localhost:8082 Forward address for clips without motion (empty ⇒ all to terp)
-threshold 5000 Minimum PixelsChanged to count as motion
-passphrase "" Decrypt/re-encrypt passphrase

Concurrency: goroutine per incoming clip tracked by procWg; shutdown closes listener, drains decodes via acceptWg, closes clipChan, waits on procWg. processClip flow: decrypt → CountChangedPixels(30) → gate on any Comparison.PixelsChanged >= threshold → re-encrypt → forward to terp or coordinator.

mofin/go.mod — standard module + replace frame => ../frame.

Files modified

  • frame/frame.go: added Frame.Errors []string and Clip.Comparisons []Comparison; removed a stale duplicate Detect declaration.
  • runCam/main.go: renamed -terp flag to -mofin (localhost:8083); terpAddr → mofinAddr throughout.
  • config.yaml: inserted mofin1 node between cam1 and terp1; cam1 repointed at mofin.

Test inventory at time of writing

frame/frame_test.go — 19 tests: ToMat roundtrip/non-nil/empty-bytes; Encrypt/Decrypt roundtrip, wrong-passphrase, empty no-op, truncated input, unique salt per call; Sublimate basic/empty/detections-carryover; CheckLenCorrelations match/PixelMats-Guids mismatch/Timestamps mismatch/empty; Clip.Send TCP roundtrip + invalid address; zero-value Frame/Clip fields.

frame/motion_test.go — 14 tests: compareTo identical/all-changed/partial/threshold filtering/multi-channel/dimension-mismatch/channel-mismatch/identical-pixel-data; CountChangedPixels basic/single-frame/empty/reset-behavior/threshold-parameter; Comparison struct fields.

frame/utils_test.go — 16 tests: ScanCams bounds/dedup/parallel-safety, Scan integration, isCaptureDevice, pure-logic helpers; camera-dependent cases skip under -short.

Plus per-module suites (mofin routing matrix, terp gates, coordinator writers, runCam lifecycle) — full current counts in §5. Invocation: go test -short ./... per module.

Reference in New Issue View Git Blame Copy Permalink