Arcology Engine

LLM Risk Classifier

Contents

Introduction

The notes tools gate their output through this classifier: an LLM-driven risk scorer judging note text against five categories: personal health info, private/intimate info, financial info, employer-owned data, politically sensitive data. This is meant to be simple enough that it can run on a local GPU to prevent sensitive information from being sent to a remote inference server.

The runtime is Ollama's Jev-style decision API, POST /v1/systemone (Ollama 0.35+; this VM runs 0.40.0): the note text goes out as state, five named score questions come back as probability-weighted rubric answers in answers. The classify-time model is Together AI's tev1:4b (model card); nimble:9b and tev1:0.8b speak the same endpoint and swap in through --risk-model without code changes.

Better to be safe than sorry, or what's the point? "Fail closed" is the important part of this:

  • Endpoint unconfigured, unreachable, timed out, or garbage-responding → ClassificationResult.Unavailable → the calling tool redacts content. Metadata still flows; text does not.

  • --no-risk-gate ([trustLocal]) is the single explicit escape hatch for trusted local runs.

  • Verdicts cache in llm_verdict_cache (Tools.sq), keyed by content hash + model, so unchanged notes don't re-burn GPU time.

Everything here shares the computer.whatthefuck.arcology.llm package with the tools; the tangle target is a single RiskClassifier.kt.

RiskGate — cache + threshold + fail-closed wiring

The gate is what tools actually call. It wraps the classifier with a cache of previous verdicts and returns RiskGateResult whose gated flag tells the tool to withhold content. =gate(..., verbose=true)= attaches a ClassifierTranscript to the RiskBlock containing the exact request (system + user prompt with the text) and the endpoint's raw response. Cache hits carry a transcript with =rawResponse=null= (no live exchange happened); a missing classifier endpoint carries only the input. Non-verbose calls never emit transcripts, so normal envelopes stay small.

The Gate

kotlin#+name: classifier-gate
data class RiskGateResult(
    val block: RiskBlock?,
    val gated: Boolean
)

/**
 * @param classifier null when no endpoint configured; combined with
 * [trustLocal]=false this fails closed — content redacted, metadata flows.
 * [trustLocal] corresponds to the tools' --no-risk-gate flag. =--verbose=
 * ride-along behavior is per-call (see [gate]).
 */
class RiskGate(
    private val repo: ToolQueryRepository,
    private val classifier: RiskClassifier?,
    private val model: String,
    private val trustLocal: Boolean
) {
    /**
     * [verbose] bypasses the verdict cache entirely — no read, no write: a
     * verbose run always performs the live exchange the caller asked to
     * inspect. It also attaches a [[ClassifierTranscript]] to the returned
     * block (see [gate]).
     */
    suspend fun gate(text: String, threshold: Double, verbose: Boolean = false): RiskGateResult {
        if (trustLocal) return RiskGateResult(block = null, gated = false)
        if (text.isBlank()) return RiskGateResult(block = null, gated = false)

        // Verbose runs skip the cache on purpose: the point of --verbose is
        // to observe the live classifier exchange.
        val contentHash = if (verbose) null else HashUtils.sha256(text)

        // Cache hit: a prior verdict for the same text with the same model.
        // Verbose cache hits still carry a transcript — just with
        // rawResponse=null, since no live exchange happens.
        if (contentHash != null) {
            repo.selectVerdict(contentHash, model)?.let { cached ->
                val scores = RiskScores(
                    health = cached.health, privateScore = cached.privateScore,
                    financial = cached.financial, employer = cached.employer,
                    political = cached.political
                )
                return RiskGateResult(
                    block = RiskBlock(
                        available = true,
                        scores = scores,
                        transcript = if (verbose) {
                            ClassifierTranscript(model = model, input = text)
                        } else null
                    ),
                    gated = scores.overall > threshold
                )
            }
        }

        if (classifier == null) {
            return RiskGateResult(
                block = RiskBlock(
                    available = false,
                    reason = "no classifier endpoint configured",
                    transcript = if (verbose) ClassifierTranscript(model = model, input = text) else null
                ),
                gated = true
            )
        }

        return when (val result = classifier.classify(text, verbose)) {
            is ClassificationResult.Success -> {
                if (contentHash != null) {
                    repo.insertVerdict(
                        contentHash, model, result.scores,
                        Clock.System.now().toEpochMilliseconds()
                    )
                }
                // Scores are the whole verdict; nothing else to strip.
                val scores = result.scores
                RiskGateResult(
                    block = RiskBlock(
                        available = true,
                        scores = scores,
                        transcript = result.transcript
                    ),
                    gated = result.scores.overall > threshold
                )
            }
            is ClassificationResult.Unavailable -> RiskGateResult(
                block = RiskBlock(
                    available = false,
                    reason = result.reason,
                    transcript = result.transcript
                ),
                gated = true
            )
        }
    }
}

RiskClassifier — Ollama HTTP client

This used to ride Ollama's generic /api/chat completion path with a pinned JSON schema; it now speaks the decision-API /v1/systemone so the model is chosen for classification, not prose. 45s request timeout stands: cold model loads on my GPU box are slow because it's just Ollama running on my Steam rig.

The Questions

Ollama constructs tev1's prompt from state and questions. The five categories map onto five named score questions whose answer keys equal the RiskScores category names: answers.health decodes straight into RiskScores.health. Each instruction is a compact policy rule naming the concrete identifying-detail triggers (tev1 is fine-tuned on exactly that shape and the first benchmark showed exposure-proportional wording over-scoring routine wellness/news text).

The rubric is per category: each level-0 wording carries that category's exclusion list (contrastive criteria beat negations in the question text, the Jev/tev design canon's not_for pattern), and higher levels anchor severity on identifying detail, never counts (decision models don't count reliably). The employer question in particular needs to be carefully gated so that normal/personal development work and other safe traffic from my System isn't needlessly gated. A set of benchmarks and fine-tuning sessions landed on these instructions and scoring queries.

kotlin#+name: classifier-questions:noweb-ref classifier-questions
/** category name → per-question instructions as single positive clauses
 * naming the identifying-detail triggers. Kept short: in per-category
 * calls the question rides inside tev1's ~2k-token prompt with the state. */
private val RISK_QUESTION_INSTRUCTIONS = mapOf(
    "health" to ("Concrete, identifying medical detail: diagnoses, medications with doses, " +
                 "lab results, procedures, prescriber or clinic names."),
    "private" to ("Intimate or emotional disclosure: relationships, family conflict, private " +
                  "thoughts the writer clearly treats as confidential."),
    "financial" to ("Identifying financial detail: salaries and offers, account or card numbers, " +
                    "debts, investments."),
    "employer" to ("Confidential business information that belongs to the employer NWM/Northwest " +
                   "Media: unreleased plans or deals, customer PII, internal business discussions. " +
                   "The writer's own technical work and learning is not confidential."),
    "political" to ("The writer's own political affiliation, activism, or partisan opinions: " +
                    "membership, donations, canvassing, partisan stances.")
)

Score → Category Mapping

A score answer returns the probability-weighted rubric level, 0.0 at None up to 4.0 at Dominated. The classifier divides by 4.0 and clamps to [0,1] so RiskScores keeps its established "how much of this category's sensitive content the text contains" 0.0-to-1.0 semantics and the gate's threshold math survives (the bench puts the clean/sensitive gap either side of 0.45–0.5, so the default threshold moves to 0.45). Per-category scores take the max across windows; NaN clamps to 0.0.

kotlin#+name: classifier-questions:noweb-ref classifier-questions
/** Per-category severity rubrics, levels numbered from 0 in the answer.
 * Level-0 wording carries the category's exclusion list (contrastive
 * criteria — the Jev/tev design canon's not_for pattern), higher levels
 * anchor severity on identifying detail and text focus, never counts
 * (decision models don't count reliably). */
private val RISK_RUBRICS = mapOf(
    "health" to listOf(
        "Nothing medical beyond routine wellness, sore muscles, vitamins, flu shots, or public health news",
        "A medical topic surfaces without identifying detail",
        "Identifying medical detail appears: a diagnosis, medication, dose, lab value, procedure, or provider name",
        "Substantial identifying medical detail, or the passage centers on the writer's condition and its treatment",
        "Dominated by the writer's medical situation"
    ),
    "private" to listOf(
        "Nothing intimate; casual mentions of friends, routine plans, and banal daily life",
        "A personal topic surfaces without intimate disclosure",
        "Intimate or emotional disclosure appears: relationship, family conflict, or clearly confidential thoughts",
        "Substantial intimate disclosure, or the passage centers on it",
        "Dominated by intimate disclosure"
    ),
    "financial" to listOf(
        "Nothing financial beyond passing prices, quotes, budgets, or public economics news",
        "A financial topic surfaces without identifying detail",
        "Identifying financial detail appears: a salary or offer, account or card number, debt, or investment",
        "Substantial identifying financial detail, or the passage centers on it",
        "Dominated by financial detail"
    ),
    "employer" to listOf(
        "No confidential business content: the writer's own technical work, learning, tooling, routine admin, or generic workplace talk",
        "Workplace matters discussed without disclosing anything confidential about NWM, its customers, or partners",
        "One confidential fact disclosed: an unreleased plan or deal detail, a customer or partner identifying detail, or internal business discussion content",
        "Multiple confidential facts disclosed, or one with high impact: deal terms, customer PII, unreleased plans stated concretely",
        "Dense confidential business disclosure: the text's substance is unreleased NWM business or others' PII"
    ),
    "political" to listOf(
        "Nothing political beyond reading or discussing public news without a stance",
        "A political topic surfaces without a personal stance",
        "The writer's own affiliation, activism, donations, or partisan stance appears",
        "Substantial personal political expression, or the passage centers on it",
        "Dominated by personal political expression"
    )
)

Text Windowing

tev1's per-question prompt is hard-capped at ~2,050 tokens server-side (`prompt N has X tokens; expected 1–2050`), and the question framing itself costs ~1.4k tokens — so a five-question batch shares one budget with the state and overflows on real note sizes (the bench hit HTTP 400 at a 6k-char window with all five questions). The client instead issues *one call per category per window: each prompt is state + one question (~1.45k tokens with a 6k-char state, fitting inside the cap with headroom), sequential on one GPU, per-category max across windows. The 6,000-character window (≈1,500 tokens of prose) stays inside tev1's training envelope, and the 8-window cap (48,000 chars) bounds GPU cost per note; text beyond it is not* classifiable, so the result is Unavailable → fail-closed redaction.

The Client

kotlin#+name: classifier-questions:noweb-ref classifier-questions
/** One /v1/systemone request: the state plus ONE category's score
 * question (per-category calls keep state+question inside the ~2k-token
 * server-side prompt cap; batching five questions rides the full set in
 * every internal prompt and overflows on real notes). */
private fun requestBody(model: String, text: String, category: String): JsonObject = buildJsonObject {
    put("model", JsonPrimitive(model))
    put("state", JsonPrimitive(text))
    put("questions", buildJsonObject {
        put(category, buildJsonObject {
            put("type", JsonPrimitive("score"))
            put("instructions", JsonPrimitive(RISK_QUESTION_INSTRUCTIONS[category]))
            put("criteria", JsonArray(RISK_RUBRICS[category]!!.map { JsonPrimitive(it) }))
        })
    })
}
kotlin#+name: classifier-client:noweb yes
class RiskClassifier(
    private val endpoint: String,
    private val model: String,
    private val httpClient: HttpClient,
    private val timeoutMs: Long = 45_000
) {
    private val categories = listOf("health", "private", "financial", "employer", "political")

    /** 6,000 chars ≈ 1,500 tokens, inside tev1's ~2k effective context. */
    private val windowChars = 6_000
    /** GPU-cost bound per note; beyond this the text is unclassifiable and
     * classification fails closed rather than under-judging. */
    private val maxWindows = 8

    /** Lenient decode: the wire answers carry extra keys (usage, legend,
     * probabilities, confidence) beyond what we consume. */
    private val responseJson = Json { ignoreUnknownKeys = true }

    /** Split [text] into windowChars-sized chunks for tev1's short context. */
    private fun windows(text: String): List<String> {
        if (text.length <= windowChars) return listOf(text)
        return (0 until text.length step windowChars).map { start ->
            text.substring(start, minOf(start + windowChars, text.length))
        }
    }

    /** Category Double from a probability-weighted rubric level: /4.0 then
     * clamp to [0,1] — 0.0 stays 0.0, Dominated stays 1.0. */
    private fun levelToScore(level: Double): Double = clamp01(level / 4.0)

    /**
     ,* Classify [text]. Never throws: every failure mode returns
     ,* [ClassificationResult.Unavailable] so callers fail closed. When
     ,* [verbose] is set, the request/response exchange rides on the result
     ,* as a [ClassifierTranscript].
     ,*/
    suspend fun classify(text: String, verbose: Boolean = false): ClassificationResult =
        withContext(Dispatchers.IO) {
            val windowList = windows(text)
            if (windowList.size > maxWindows) {
                return@withContext ClassificationResult.Unavailable(
                    "text exceeds classifier window budget: ${text.length} chars " +
                        "> $maxWindows windows of $windowChars",
                    if (verbose) ClassifierTranscript(model = model, input = text) else null
                )
            }
            // One request per (window, category): each internal prompt is
            // state + one question, inside the server's ~2,050-token cap.
            // Each request tracks its asked category so a missing answer
            // fails closed instead of decoding to a silent 0.0.
            val requests = windowList.flatMap { window ->
                categories.map { it to requestBody(model, window, it).toString() }
            }
            fun transcript(rawResponse: String? = null) = if (verbose) {
                ClassifierTranscript(
                    model = model,
                    input = requests.joinToString("\n---window---\n") { it.second },
                    rawResponse = rawResponse
                )
            } else {
                null
            }
            // Requests are sequential: same server, one model slot.
            val rawResponses = mutableListOf<String>()
            // Per-category max across windows: most sensitive window governs.
            val perCategory = categories.associateWith { Double.NEGATIVE_INFINITY }.toMutableMap()
            try {
                for ((asked, request) in requests) {
                    val response = httpClient.post("$endpoint/v1/systemone") {
                        contentType(ContentType.Application.Json)
                        setBody(request)
                    }
                    if (!response.status.isSuccess()) {
                        return@withContext ClassificationResult.Unavailable(
                            "classifier http ${response.status.value}",
                            transcript("HTTP ${response.status.value}")
                        )
                    }
                    val raw = response.bodyAsText()
                    rawResponses.add(raw)
                    val body = try {
                        responseJson.decodeFromString(
                            SystemOneResponse.serializer(), raw
                        )
                    } catch (e: Exception) {
                        return@withContext ClassificationResult.Unavailable(
                            "classifier response not parseable",
                            transcript(rawResponses.joinToString("\n"))
                        )
                    }
                    val answer = body.answers[asked]
                        ?: return@withContext ClassificationResult.Unavailable(
                            "classifier response missing answer '$asked'",
                            transcript(rawResponses.joinToString("\n"))
                        )
                    val scoreField = answer["score"]?.jsonPrimitive?.doubleOrNull
                        ?: return@withContext ClassificationResult.Unavailable(
                            "classifier answer '$asked' missing numeric score",
                            transcript(rawResponses.joinToString("\n"))
                        )
                    perCategory[asked] = maxOf(
                        perCategory[asked] ?: Double.NEGATIVE_INFINITY,
                        levelToScore(scoreField)
                    )
                }
                val scores = RiskScores(
                    health = perCategory["health"] ?: 0.0,
                    privateScore = perCategory["private"] ?: 0.0,
                    financial = perCategory["financial"] ?: 0.0,
                    employer = perCategory["employer"] ?: 0.0,
                    political = perCategory["political"] ?: 0.0
                ).clamped()
                ClassificationResult.Success(scores, transcript(rawResponses.joinToString("\n")))
            } catch (e: Exception) {
                ClassificationResult.Unavailable(
                    "classifier unreachable: ${e.message}",
                    transcript()
                )
            }
        }
}

/** Coerce every category to [0, 1] and treat NaN as 0. */
fun RiskScores.clamped(): RiskScores = RiskScores(
    health = clamp01(health),
    privateScore = clamp01(privateScore),
    financial = clamp01(financial),
    employer = clamp01(employer),
    political = clamp01(political)
)

private fun clamp01(v: Double): Double = when {
    v.isNaN() || v < 0.0 -> 0.0
    v > 1.0 -> 1.0
    else -> v
}

RiskClassifier.kt Composition

kotlin#+name: classifier-composed:tangle ../src/jvmMain/kotlin/computer/whatthefuck/arcology/llm/RiskClassifier.kt:noweb yes
package computer.whatthefuck.arcology.llm

import computer.whatthefuck.arcology.utils.HashUtils
import io.ktor.client.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import io.ktor.http.*
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.put
import kotlinx.serialization.json.jsonPrimitive
import kotlinx.serialization.json.doubleOrNull
import kotlin.time.Clock

<<classifier-types>>

<<classifier-questions>>

<<classifier-client>>

<<classifier-gate>>

DONE integrate tev1:4b into the risk classifier

the classifier now speaks POST /v1/systemone (Ollama 0.35+; this VM runs 0.40.0) with tev1:4b as the classify-time model. Ollama builds tev1's prompt from state + questions; the five categories are five named score questions over the shared 5-level rubric, level/4.0 → clamped RiskScores Doubles. The generic /api/chat path is gone. nimble:9b / tev1:0.8b swap in via --risk-model; no nixpkgs work was needed.

Types

private is a Kotlin hard modifier, so the category name survives to JSON via @SerialName on a privateScore property. The cache column is private_score for the same reason.

Risk Types

kotlin#+name: classifier-types
@Serializable
data class RiskScores(
    val health: Double = 0.0,
    @SerialName("private") val privateScore: Double = 0.0,
    val financial: Double = 0.0,
    val employer: Double = 0.0,
    val political: Double = 0.0
) {
    val overall: Double get() = maxOf(health, privateScore, financial, employer, political)
}

/** One /v1/systemone response: answers keyed by question name (our five
 ,* category names). [answers] values are raw JSON objects — shape varies by
 ,* question type, and we only consume "score" answers. Unknown keys are
 ,* tolerated (usage, legend, confidence…). */
@Serializable
data class SystemOneResponse(
    val model: String = "",
    val answers: Map<String, JsonObject> = emptyMap()
)

/** Result of one classification attempt. */
sealed interface ClassificationResult {
    data class Success(
        val scores: RiskScores,
        /** Set only when the caller asked for the transcript. */
        val transcript: ClassifierTranscript? = null
    ) : ClassificationResult
    /** Endpoint unreachable, timed out, or returned unusable output. Always fails closed. */
    data class Unavailable(
        val reason: String,
        /** Partial transcript when available (e.g. HTTP status, raw garbage). */
        val transcript: ClassifierTranscript? = null
    ) : ClassificationResult
}

/**
 ,* The risk block attached to every tool envelope. [available] false means
 ,* the classifier could not be consulted (fail closed); [scores] carry the
 ,* five category scores; [redacted] true means content was withheld.
 ,* [transcript] — the classifier's input/output exchange — is only attached
 ,* when the tool was invoked with =--verbose=.
 ,*/
@Serializable
data class RiskBlock(
    val available: Boolean = true,
    val scores: RiskScores? = null,
    val redacted: Boolean = false,
    val reason: String? = null,
    val transcript: ClassifierTranscript? = null
)

/**
 ,* The full classifier exchange: what was sent (system+user prompt with the
 ,* text under classification) and what the endpoint answered. Debug artifact
 ,* for =--verbose= runs; never emitted otherwise.
 ,*/
@Serializable
data class ClassifierTranscript(
    val model: String,
    val input: String,
    val rawResponse: String? = null
)

Tests

A fake Ollama endpoint (embedded Ktor CIO server on an ephemeral port) drives these; the fail-closed cases point at a port with nothing listening.

kotlin#+name: classifier-tests:tangle ../src/jvmTest/kotlin/computer/whatthefuck/arcology/llm/RiskClassifierTest.kt
package computer.whatthefuck.arcology.llm

import computer.whatthefuck.arcology.db.ArcologyDatabase
import io.ktor.server.engine.*
import io.ktor.server.cio.*
import io.ktor.server.response.*
import io.ktor.server.routing.*
import io.ktor.http.*
import io.ktor.client.*
import io.ktor.client.plugins.*
import io.ktor.server.request.*
import kotlinx.coroutines.test.runTest
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonObject
import kotlin.test.*

class RiskClassifierTest {

    private fun makeRepo(): ToolQueryRepositoryImpl =
        ToolQueryRepositoryImpl(computer.whatthefuck.arcology.database.DatabaseFactory.createInMemoryDatabase())

    private fun client(): HttpClient = HttpClient {
        install(HttpTimeout) { requestTimeoutMillis = 5_000 }
    }

    /**
     * Fake Ollama /v1/systemone. [answersByCategory] maps a category name
     * to that category's answer JSON; the fake reads the request's
     * question key and answers it, mirroring the per-category requests
     * the client issues. [defaultAnswer] answers anything unmapped.
     */
    private fun systemoneServer(
        answersByCategory: Map<String, String>,
        defaultAnswer: String? = answerJson("health", 0.0)
    ): Pair<EmbeddedServer<*, *>, Int> {
        // Ktor 3 dropped resolvedConnectors(); reserve an explicit ephemeral
        // port ourselves so tests can derive the URL without the connector API.
        val port = java.net.ServerSocket(0).use { it.localPort }
        val server = embeddedServer(CIO, host = "127.0.0.1", port = port) {
            routing {
                post("/v1/systemone") {
                    val req = call.receiveText()
                    val asked = Regex(""""(health|private|financial|employer|political)"\s*:\s*\{""")
                        .findAll(req).lastOrNull()?.groupValues?.get(1)
                    val answer = answersByCategory[asked] ?: defaultAnswer
                    if (answer == null) {
                        call.respondText("""{"model":"test","answers":{},"usage":{"input_tokens":10,"output_tokens":1}}""", ContentType.Application.Json)
                    } else {
                        call.respondText("""{"model":"test","answers":$answer,"usage":{"input_tokens":10,"output_tokens":1}}""", ContentType.Application.Json)
                    }
                }
            }
        }
        return Pair(server, port)
    }

    /** Convenience: five per-category RUBRIC LEVELS (0..4) → a
     * fully-answering fake. NOT final scores; the client divides by 4. */
    private fun systemoneServer(
        health: Double, private: Double, financial: Double, employer: Double, political: Double
    ): Pair<EmbeddedServer<*, *>, Int> = systemoneServer(mapOf(
        "health" to answerJson("health", health),
        "private" to answerJson("private", private),
        "financial" to answerJson("financial", financial),
        "employer" to answerJson("employer", employer),
        "political" to answerJson("political", political)
    ), defaultAnswer = null)

    /** One score answer for the category a per-category request asks
     * about (extra categories ride along harmlessly; the client reads
     * only the asked one). Built with kotlinx so keys are quoted. */
    private fun answerJson(category: String, level: Double): String =
        buildJsonObject {
            put(category, buildJsonObject {
                put("type", JsonPrimitive("score"))
                put("score", JsonPrimitive(level))
                put("legend", buildJsonObject {
                    put("0", JsonPrimitive("None"))
                    put("4", JsonPrimitive("Dominated"))
                })
                put("probabilities", buildJsonObject { put("0", JsonPrimitive(1.0)) })
                put("confidence", JsonPrimitive(0.5))
            })
        }.toString()

    @Test
    fun `parses a healthy verdict`() = runTest {
        // employer level 3.6 of 4 → 0.9
        val (server, port) = systemoneServer(0.0, 0.0, 0.8, 3.6, 0.0)
        server.start(wait = false)
        val classifier = RiskClassifier("http://127.0.0.1:$port", "test", client())
        val result = classifier.classify("some text")
        assertTrue(result is ClassificationResult.Success)
        assertEquals(0.9, result.scores.employer)
        assertEquals(0.9, result.scores.overall)
        server.stop(1000, 2000)
    }

    @Test
    fun `real pkb work notes score below the default gate`() = runTest {
        // Regression: a philosophy note, a meditation book highlight, and a
        // calendar event all scored employer 0.47-0.48 under the original
        // "proprietary content" framing (rrix's live traffic). Under the
        // leak-check wording every category must stay below the 0.40
        // default threshold.
        val realCleanNodeScores = listOf(
            "We use too many words, too many details, create a sort of fog-of-war in which we have plausible deniability for our inactions. The challenge is, this can be hard to distinguish from actual complexity.",
            "Such efforts heighten activity in the dorsolateral prefrontal area, a key circuit for managing the default mode.",
            "Trash Day"
        )
        for (text in realCleanNodeScores) {
            val (server, port) = systemoneServer(0.0, 0.0, 0.0, 0.0, 0.0)
            server.start(wait = false)
            val gate = RiskGate(
                makeRepo(), RiskClassifier("http://127.0.0.1:$port", "tev1:4b", client()),
                model = "tev1:4b", trustLocal = false
            )
            val result = gate.gate(text, threshold = 0.40)
            assertFalse(result.gated, "must not gate daily PKB noise: $text")
            server.stop(1000, 2000)
        }
    }

    @Test
    fun `employer leak still gates at the default threshold`() = runTest {
        // employer level 2.72 of 4 → 0.68 (the live PKB control's score)
        val (server, port) = systemoneServer(0.0, 0.0, 0.0, 2.72, 0.0)
        server.start(wait = false)
        val gate = RiskGate(
            makeRepo(), RiskClassifier("http://127.0.0.1:$port", "tev1:4b", client()),
            model = "tev1:4b", trustLocal = false
        )
        val result = gate.gate("the NWM acquisition of Willow Media is priced at 40M and closes in Q2", threshold = 0.40)
        assertTrue(result.gated)
        assertEquals(0.68, result.block?.scores?.employer)
        server.stop(1000, 2000)
    }

    @Test
    fun `clamps oversized levels into range`() = runTest {
        // 999.0 of 4 → far over 1; 1e999 decodes to +Infinity and must clamp
        // to 1.0 too. Neither may leak past the [0,1] category contract.
        val (server, port) = systemoneServer(
            health = 999.0, private = 1e999, financial = 0.0, employer = 0.0, political = 0.0
        )
        server.start(wait = false)
        val classifier = RiskClassifier("http://127.0.0.1:$port", "test", client())
        val result = classifier.classify("some text")
        assertTrue(result is ClassificationResult.Success)
        assertEquals(1.0, result.scores.health)
        assertEquals(1.0, result.scores.privateScore)
        server.stop(1000, 2000)
    }

    @Test
    fun `garbage verdict fails closed as unavailable`() = runTest {
        val (server, port) = systemoneServer(mapOf(), defaultAnswer = "not json at all")
        server.start(wait = false)
        val classifier = RiskClassifier("http://127.0.0.1:$port", "test", client())
        val result = classifier.classify("some text")
        assertTrue(result is ClassificationResult.Unavailable)
        server.stop(1000, 2000)
    }

    @Test
    fun `unreachable endpoint fails closed`() = runTest {
        val classifier = RiskClassifier("http://127.0.0.1:1", "test", client())
        val result = classifier.classify("some text")
        assertTrue(result is ClassificationResult.Unavailable)
    }

    @Test
    fun `gate fails closed without classifier`() = runTest {
        val repo = makeRepo()
        val gate = RiskGate(repo, classifier = null, model = "m", trustLocal = false)
        val result = gate.gate("some private content", threshold = 0.40)
        assertTrue(result.gated)
        assertEquals(false, result.block?.available)
        assertNull(result.block?.scores)
    }

    @Test
    fun `gate trusts local when flag set`() = runTest {
        val repo = makeRepo()
        val gate = RiskGate(repo, classifier = null, model = "m", trustLocal = true)
        val result = gate.gate("anything", threshold = 0.40)
        assertFalse(result.gated)
        assertNull(result.block)
    }

    @Test
    fun `gate caches verdict per content hash and model`() = runTest {
        val (server, port) = systemoneServer(0.0, 0.0, 4.0, 0.0, 0.0)
        server.start(wait = false)
        val repo = makeRepo()
        val gate = RiskGate(
            repo, RiskClassifier("http://127.0.0.1:$port", "testmodel", client()),
            model = "testmodel", trustLocal = false
        )

        val first = gate.gate("salary negotiation notes", threshold = 0.40)
        assertTrue(first.gated) // financial 4.0/4 → 1.0 > 0.40

        // Fresh gate, dead classifier: still correctly scored via llm_verdict_cache
        val second = RiskGate(
            repo, RiskClassifier("http://127.0.0.1:1", "testmodel", client()),
            model = "testmodel", trustLocal = false
        )
        val result = second.gate("salary negotiation notes", threshold = 0.40)
        assertTrue(result.block?.available == true)
        assertEquals(1.0, result.block?.scores?.financial)
        assertNull(result.block?.transcript, "cache hits don't carry live transcripts")
        server.stop(1000, 2000)
    }

    @Test
    fun `verdict cache is model-scoped`() = runTest {
        val repo = makeRepo()
        repo.insertVerdict("hash", "model-a", RiskScores(financial = 0.5), 1L)
        assertNull(repo.selectVerdict("hash", "model-b"))
        assertEquals(0.5, repo.selectVerdict("hash", "model-a")?.financial)
    }

    @Test
    fun `verbose off produces no transcript`() = runTest {
        val (server, port) = systemoneServer(0.0, 0.0, 4.0, 0.0, 0.0)
        server.start(wait = false)
        val classifier = RiskClassifier("http://127.0.0.1:$port", "test", client())
        val result = classifier.classify("some text", verbose = false)
        assertTrue(result is ClassificationResult.Success)
        assertNull(result.transcript)
        server.stop(1000, 2000)
    }

    @Test
    fun `verbose on attaches request and raw response`() = runTest {
        val (server, port) = systemoneServer(0.0, 0.0, 4.0, 0.0, 0.0)
        server.start(wait = false)
        val classifier = RiskClassifier("http://127.0.0.1:$port", "test", client())
        val result = classifier.classify("journal entry with salary figures", verbose = true)
        assertTrue(result is ClassificationResult.Success)
        val t = result.transcript
        assertNotNull(t)
        assertEquals("test", t.model)
        assertEquals(5, t.input.split("\n---window---\n").size, "five per-category requests")
        assertTrue(t.input.contains("journal entry with salary figures"), "state rides in the request: ${t.input}")
        assertTrue(t.input.contains("\"financial\""), "financial question rides in the request: ${t.input}")
        // The raw response is echoed verbatim, whatever the wire shape was.
        assertNotNull(t.rawResponse)
        assertTrue(t.rawResponse!!.contains("\"score\":4.0"), "financial level present: ${t.rawResponse}")
        server.stop(1000, 2000)
    }

    @Test
    fun `verbose on failure carries raw junk as transcript response`() = runTest {
        val (server, port) = systemoneServer(mapOf(), defaultAnswer = "not json at all")
        server.start(wait = false)
        val classifier = RiskClassifier("http://127.0.0.1:$port", "test", client())
        val result = classifier.classify("some text", verbose = true)
        assertTrue(result is ClassificationResult.Unavailable)
        // The raw unverifiable junk is exactly what verbose exists to expose.
        assertEquals(
            """{"model":"test","answers":not json at all,"usage":{"input_tokens":10,"output_tokens":1}}""",
            result.transcript?.rawResponse
        )
        assertTrue(result.transcript?.input?.contains("some text") == true)
        server.stop(1000, 2000)
    }

    @Test
    fun `gate verbose live call carries transcript`() = runTest {
        val (server, port) = systemoneServer(4.0, 0.0, 0.0, 0.0, 0.0)
        server.start(wait = false)
        val repo = makeRepo()
        val gate = RiskGate(
            repo, RiskClassifier("http://127.0.0.1:$port", "testmodel", client()),
            model = "testmodel", trustLocal = false
        )
        val result = gate.gate("doctor notes", threshold = 0.40, verbose = true)
        assertTrue(result.gated) // health 4.0/4 → 1.0 > 0.40
        assertNotNull(result.block?.transcript)
        assertNotNull(result.block?.transcript?.rawResponse)
        server.stop(1000, 2000)
    }

    @Test
    fun `verbose bypasses the verdict cache`() = runTest {
        val (server, port) = systemoneServer(4.0, 0.0, 0.0, 0.0, 0.0)
        server.start(wait = false)
        val repo = makeRepo()
        val live = RiskGate(
            repo, RiskClassifier("http://127.0.0.1:$port", "testmodel", client()),
            model = "testmodel", trustLocal = false
        )
        live.gate("doctor notes", threshold = 0.40) // populates cache

        // Verbose + dead classifier: cache is bypassed, so the call hits the
        // dead endpoint and fails closed as unavailable (NOT a cache hit).
        val bypassed = RiskGate(
            repo, RiskClassifier("http://127.0.0.1:1", "testmodel", client()),
            model = "testmodel", trustLocal = false
        )
        val result = bypassed.gate("doctor notes", threshold = 0.40, verbose = true)
        assertEquals(false, result.block?.available)
        assertNotNull(result.block?.transcript)

        // Non-verbose same gate: cache hit serves the live scores.
        val throughCache = bypassed.gate("doctor notes", threshold = 0.40, verbose = false)
        assertEquals(true, throughCache.block?.available)
        assertEquals(1.0, throughCache.block?.scores?.health)
        server.stop(1000, 2000)
    }

    @Test
    fun `gate verbose without classifier emits config-only transcript`() = runTest {
        val repo = makeRepo()
        val gate = RiskGate(repo, classifier = null, model = "m", trustLocal = false)
        val result = gate.gate("some private content", threshold = 0.40, verbose = true)
        // No classifier endpoint: transcript reflects configuration, not content.
        assertEquals(false, result.block?.available)
        assertNull(result.block?.scores)
        assertNotNull(result.block?.transcript)
        assertEquals("some private content", result.block?.transcript?.input)
        assertNull(result.block?.transcript?.rawResponse)
    }

    @Test
    fun `answer without probabilities or confidence keys still decodes`() = runTest {
        // Only "type" and "score" are consumed; legend/probabilities/
        // confidence and the usage block are unknown keys the decode ignores.
        val (server, port) = systemoneServer(0.0, 0.0, 0.0, 0.0, 0.0)
        server.start(wait = false)
        val classifier = RiskClassifier("http://127.0.0.1:$port", "test", client())
        val result = classifier.classify("some text")
        assertTrue(result is ClassificationResult.Success)
        assertEquals(0.0, result.scores.overall)
        server.stop(1000, 2000)
    }

    @Test
    fun `missing answer or non-numeric score fails closed`() = runTest {
        // Two per-category failures in one classification: the financial
        // request comes back shape-broken (missing numeric score) and the
        // employer request comes back with NO answers at all. Both must
        // fail closed — never decode to a silent 0.0.
        val (server, port) = systemoneServer(mapOf(
            "financial" to """{"financial":{"type":"object"}}""",
            "employer" to "{}"
        ), defaultAnswer = answerJson("health", 0.0))
        server.start(wait = false)
        val classifier = RiskClassifier("http://127.0.0.1:$port", "test", client())
        val result = classifier.classify("some text")
        assertTrue(result is ClassificationResult.Unavailable)
        server.stop(1000, 2000)
    }

    @Test
    fun `multi window classification takes the most sensitive window`() = runTest {
        // Window 1 is clean; window 2 carries an employer leak at level 3.0
        // → 0.75. The per-category max across windows governs.
        val port = java.net.ServerSocket(0).use { it.localPort }
        val server = embeddedServer(CIO, host = "127.0.0.1", port = port) {
            routing {
                post("/v1/systemone") {
                    val req = call.receiveText()
                    val asked = Regex(""""(health|private|financial|employer|political)"\s*:\s*\{""")
                        .findAll(req).lastOrNull()?.groupValues?.get(1)
                    // Window 1 = 6,000 'a's (clean). Window 2 carries the
                    // leak sentence — detect it by its marker text in the
                    // raw request body (newlines JSON-escape to \n).
                    val level = if (asked == "employer" &&
                                 req.contains("salary figures for the NWM acquisition")) 3.0 else 0.0
                    call.respondText(
                        """{"model":"test","answers":${answerJson(asked ?: "health", level)}}""",
                        ContentType.Application.Json
                    )
                }
            }
        }
        server.start(wait = false)
        val classifier = RiskClassifier("http://127.0.0.1:$port", "test", client())
        // Two windows: 6,000 chars of padding, then the leak — window 2 is
        // the only one with the marker text, so its score must govern.
        val text = "a".repeat(6_000) + "\n\nsalary figures for the NWM acquisition\n"
        val result = classifier.classify(text)
        assertTrue(result is ClassificationResult.Success)
        assertEquals(0.75, result.scores.employer, "most sensitive window governs")
        server.stop(1000, 2000)
    }

    @Test
    fun `overflow beyond the window budget fails closed`() = runTest {
        // 49,000 chars > 8 windows of 6,000. Dead endpoint on purpose:
        // overflow must short-circuit before any HTTP attempt.
        val classifier = RiskClassifier("http://127.0.0.1:1", "test", client())
        val result = classifier.classify("a".repeat(49_000), verbose = true)
        assertTrue(result is ClassificationResult.Unavailable)
        assertTrue(result.transcript?.input?.startsWith("a") == true,
                   "verbose carries the unclassifiable text")
        assertEquals(
            "text exceeds classifier window budget: 49000 chars > 8 windows of 6000",
            (result as ClassificationResult.Unavailable).reason
        )
    }
}

Related Modules

References

Web sources consulted for the tev1:4b / /v1/systemone integration and the question/rubric design iterations (v1 exposure-proportional → v2 policy rules → v3 per-category rubrics):

Primary / API docs

Practitioner write-ups