feat: HuggingFace cybersec model integration — pretrained classifier on ingested entries #9

New issue

Closed

opened 2026-05-09 15:31:23 -07:00 by pyr0ball · 1 comment

pyr0ball commented

2026-05-09 15:31:23 -07:00

Owner

Integrate a pretrained cybersec/log-anomaly model from HuggingFace (candidates: markusbayer/log-anomaly-detection, datasets like SecBench, BETH, or LANL auth logs models). Run inference on newly ingested entries, persist a ml_score and ml_label column. Local inference via Ollama or direct transformers.

Integrate a pretrained cybersec/log-anomaly model from HuggingFace (candidates: `markusbayer/log-anomaly-detection`, datasets like `SecBench`, `BETH`, or LANL auth logs models). Run inference on newly ingested entries, persist a `ml_score` and `ml_label` column. Local inference via Ollama or direct transformers.

pyr0ball added this to the beta milestone 2026-05-09 15:31:23 -07:00

pyr0ball commented

2026-06-10 15:01:36 -07:00

Author

Owner

Closing — cybersec zero-shot scoring pipeline is complete and verified.

First clean run: 320 entries scored, 0 detections (as expected for a fresh baseline), last_error: null, 38.6 min CPU inference time. The blocking "database is locked" contention (163,975 errors) was resolved in the same session by removing per-flush FTS sync from watcher.py (see commit ccc9a9e). The security alerts UI for displaying detections was shipped in commit a33d983.

Model: MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli, threshold: 0.60, device: cpu.

Closing — cybersec zero-shot scoring pipeline is complete and verified. First clean run: 320 entries scored, 0 detections (as expected for a fresh baseline), `last_error: null`, 38.6 min CPU inference time. The blocking "database is locked" contention (163,975 errors) was resolved in the same session by removing per-flush FTS sync from `watcher.py` (see commit `ccc9a9e`). The security alerts UI for displaying detections was shipped in commit `a33d983`. Model: `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli`, threshold: 0.60, device: cpu.

pyr0ball closed this issue

2026-06-10 15:01:43 -07:00

pyr0ball referenced this issue from a commit

2026-06-11 18:38:01 -07:00

feat: cybersec zero-shot scoring pipeline (#9)

pyr0ball referenced this issue from a commit

2026-06-11 18:38:01 -07:00

fix(watcher): remove per-flush FTS sync to eliminate SQLite write lock contention

pyr0ball referenced this issue from a commit

2026-06-13 21:54:47 -07:00

feat: cybersec zero-shot scoring pipeline (#9)

pyr0ball referenced this issue from a commit