OpenSage epistemic report

Type: agentic-system-analyses/types/agentic-system-epistemic-report.md

Source-and-claim boundary

The question is how OpenSage acquires or formulates truth-apt content, checks it and lets it affect later work. Scope and exclusions are those of boundary; evidence identities are in its Source register. All additional inspection below is SRC-1, implementation unless explicitly labelled doctrine/design, at the shared revision. No target execution, supplied run trace or causal experiment supports this member. A path to a model is not evidence of its outputs, adherence or benefit.

The supplied runtime is a useful but incomplete ordering of epistemic work. In particular, its knowledge-policy discussion does not cover the implemented validate_claim, audit_assumptions, log_finding and critique operations. These findings supplement rather than supersede RT-BAP-memory-policy: that record correctly concerns the injected wiki policy, not every claim-checking tool. No supplied memory member was available; new findings concerning shared findings and summaries may require identity reconciliation with that analyst's independently declared records.

Consequential claims are RT-CLM-runtime-creation, EPI-CLM-evidence-tools, EPI-CLM-knowledge-policy and EPI-CLM-poc-success. The last is an implemented output assertion, not a public performance claim. There is no single system-wide warrant assessment.

Entry point or operation	Source path	Coverage and records; remaining limit
CLI run, ordinary web, children, live stream	`src/opensage/cli/opensage_cli.py`; `src/opensage/cli/opensage_web_app.py`; `src/opensage/orchestration/manager.py`	Supplied runtime findings: RT-RTE-user-dispatch, RT-RTE-directed-child, RT-RTE-live-runner. No ordinary-answer reference oracle follows from these paths.
CLI review and saved sessions	`src/opensage/cli/opensage_cli.py`	Direct read of `cli_review`: read-only trajectory display, not semantic review or acceptance; model-validation of JSON is structural. Resume behavior otherwise see RT-RTE-user-dispatch.
Exposed critique	`src/opensage/toolbox/general/agent_tools.py`; `agent_library/agents/poc_agent_static_tools/agent.py`	Direct complete tool-file read and selected factory read: EPI-RTE-critique-feedback. Factory includes `critique`.
Confidence audit and progress validation	`src/opensage/toolbox/general/agent_tools.py`	Direct: EPI-RTE-assumption-audit, EPI-RTE-claim-validation. Repository-wide search for their exact names in `src/` and `agent_library/` found definitions but no factory wiring; callable availability is not ordinary activation.
Shared finding check, write and peer supply	`src/opensage/toolbox/general/agent_tools.py`	Direct: EPI-RTE-finding-check, EPI-RTE-finding-write, EPI-RTE-finding-delivery. Exact-name search likewise found no selected factory grant for `log_finding`.
Tool-output reduction and reload pointer	`src/opensage/features/summarization.py`; `src/opensage/plugins/default/adk_plugins/tool_response_summarizer_plugin.py`	Direct callback and plugin reads: EPI-RTE-tool-reduction. General callback enabledness remains configuration-dependent.
History compaction and pinned text	`src/opensage/features/summarization.py`; `src/opensage/templates/auto_insert_prompts/default.md`	Direct summary-generation and policy reads plus supplied RT-RTE-history-compaction. Pin embedding implementation was located by bounded search; full pin I/O/recovery was not independently traced.
Wiki maintenance, file/script writes and dynamic instructions	`src/opensage/templates/auto_insert_prompts/default.md`; `src/opensage/toolbox/general/orchestration_tools.py`; `src/opensage/toolbox/general/bash_tools_interface.py`	Policy directly read; write/execution mechanisms supplied by RT-RTE-terminal-extension, RT-RTE-create-subagent, RT-BAP-dynamic-instruction. No observed knowledge entry or evidence-consuming acceptance instance.
Hooks, host imports and edit guidance	`src/opensage/plugins/claude_code_hook_loader.py`; `src/opensage/plugins/default/adk_plugins/read_before_edit_plugin.py`	Supplied RT-RTE-host-extension, RT-BAP-tool-guidance; hook callbacks additionally inventoried by search. Custom hook bodies not supplied; cannot assert universal blocking or universal lack of evaluation.
Provider-side search	`src/opensage/toolbox/general/web_search_tool.py`	Direct complete file: EPI-RTE-search-request. Provider return/attribution internals outside frozen source; no actual imported proposition or preserved source warrant can be individuated.
Cybench score and trajectory legitimacy	`benchmarks/cybench/helpers.py`; `benchmarks/cybench/sagectf.py`	Direct selected evaluator, judge and output-consumer ranges: EPI-RTE-cybench-check, EPI-RTE-cybench-disposition. Metadata/trace fidelity and judge accuracy unverified.
Vulnerability comparison and RL reward	`benchmarks/secodeplt/vul_detection_memory.py`	Direct selected comparison, aggregate consumer and reward ranges: EPI-RTE-vulnerability-comparison, EPI-RTE-vulnerability-reward. Ground-truth data themselves not captured. Retained finding resume/write locations searched, not fully traced; prevents a complete benchmark-memory account.
PoC submission interpretation	`benchmarks/cybergym/tools.py`; `agent_library/agents/poc_agent_static_tools/agent.py`	Direct selected response-interpretation branch and factory: EPI-RTE-poc-interpretation. Submit service internals uninspected.
Training, other benchmark evaluators and skill scripts	`src/opensage/evaluation/rl_adapters/`; `benchmarks/`; `src/opensage/bash_tools/`	Integration supplied by RT-RTE-evaluation-rollout. Trees/searches locate additional Harbor, SWE, DevOps, fuzzing, linting, graph and retrieval operations, not fully inspected here. Their distinct evaluator domains, rejection/rollback and downstream consumers remain unassessed; this explicitly prevents a system-complete negative about checking, acceptance, integration or theory revision.

The supplied refusal additionally identifies optional build verification and repeated-call suppression in SRC-1, src/opensage/plugins/default/adk_plugins/build_verifier_plugin.py and src/opensage/plugins/default/adk_plugins/doom_loop_detector_plugin.py, and web event editing in src/opensage/cli/opensage_web_app.py and src/opensage/features/opensage_in_memory_session_service.py. These are named unassessed mechanisms here, not absent mechanisms: they prevent a complete checking/recovery-plugin or write-control/provenance comparison. Their bodies were not independently traced in this correction; the selected consultation and benchmark conclusions do not assert that completeness.

External model/provider/trainer state is not accessible implementation evidence. No exact deployed weight versions, update traces, candidate theories, criticism instances or measured improvements were supplied. These gaps respectively prevent immutable-model attribution, exercised learning/self-improvement and observed theory-builder conclusions. Accessible but uninspected shipped operations above are inspection limits, not access failures or evidenced absences.

Epistemic-object inventory

Object ID	Candidate truth-apt content or none	Claimed role	Evidence source ID and local anchor	Gap/limit
RT-CMP-base-llm	not determinable at individual parameter/content level	see record	see record	Distributed-parametric inference; ordinary wrappers do not update weights; endpoint mutability and trainer input remain separate.
RT-OBJ-agent-instruction	Instructions may contain claims; no concrete generated claim supplied	see record	see record	Policy text interpreted by a model, not a correctness verdict.
RT-OBJ-agent-definition	none in selector fields	see record	see record	Model/tool selection has operational rather than epistemic authority.
EPI-OBJ-review-text	Consultation result container	see record	see record	Operative parts follow; no pooled check license.
EPI-OBJ-critique-idea	Progress and missing-evidence advice	see record	see record	Model judgment, advisory only.
EPI-OBJ-assumption-audit	Confidence classifications and direction advice	see record	see record	Natural-language audit, not a parsed validity gate.
EPI-OBJ-validation-reasoning	Explanation of claim support	see record	see record	Selected reasoning text, not full reviewer output.
EPI-OBJ-validation-decision	Parsed validity and confidence about a claim	see record	see record	Symbolic fields can be mutually inconsistent; advisory.
EPI-OBJ-findings	Caller findings and model contradiction/completion analyses	see record	see record	Text is individuable by entries; truth and actual instance provenance unobserved.
EPI-OBJ-tool-summary	Selected statements about tool output and inferred intent	see record	see record	Raw-output acquisition and summary interpretation have different warrant.
EPI-OBJ-benchmark-results	Benchmark result container	see record	see record	Distinct operative parts follow.
EPI-OBJ-flag-correct	Reference flag occurs in selected output	see record	see record	Fixed comparison over supplied strings only.
EPI-OBJ-judge-pass	Trajectory passes the blacklist judgment	see record	see record	Model judgment, error defaults false.
EPI-OBJ-judge-reasons	Reasons and evidence for trajectory judgment	see record	see record	Model explanation, not an independent oracle.
EPI-OBJ-solved-score	Task meets programmed solved criterion	see record	see record	Conjunction licenses this benchmark classification only.
EPI-OBJ-vulnerability-match	Prediction matches reference vulnerability class	see record	see record	Model Boolean feeds accuracy, not reproduction.
EPI-OBJ-match-reason	Explanation of vulnerability match	see record	see record	Report explanation does not itself increment accuracy.
EPI-OBJ-reward-score	none as vulnerability truth; a computed count/status reward	see record	see record	A high score is not confirmation that a vulnerability exists.

Authority-route ledger

All routes below are architecturally implemented, conditionally on their named configuration or callable invocation. This does not assert observed operation. Fields owned by records are deliberately referenced rather than restated. Each check and consequential disposition/retention is separate.

Route ID	Route function	Architectural status	Object/candidate ID	Content/update relation	Transition or check target	Evaluator/condition and domain	Activation and timing	Possible or observed result	Implemented force	Epistemic authority and scope	Operational authority: behavior permitted, blocked, or changed	Behavioral-authority path: consumer, channel, force, horizon	Evidence source ID and local anchor	Claim IDs or none	Mismatch marker or none	Gap/limit
EPI-RTE-critique-feedback	check/evidence production	implemented	EPI-OBJ-critique-idea	truth-apt transformation: indeterminate	see route	see route	see route	see route	see route	see route	see route	see route	see route	EPI-CLM-evidence-tools	none	No output instance or independent reference.
EPI-RTE-assumption-audit	check/evidence production	implemented	EPI-OBJ-assumption-audit	truth-apt transformation: indeterminate	see route	see route	see route	see route	see route	see route	see route	see route	see route	EPI-CLM-evidence-tools	Tool observation is not universal source truth	Folded and truncated evidence limits audit.
EPI-RTE-claim-validation	check/evidence production	implemented	EPI-OBJ-validation-reasoning	truth-apt transformation: indeterminate	Claim support in selected history	Registered reviewer model; transcript domain	Explicit invocation before possible pinning	Reasoning about evidence	Advisory	Assessment of supplied transcript only	Caller may revise or investigate	Caller model; reasoning tool field; advisory; subsequent turns	see route	EPI-CLM-evidence-tools	No independent investigation	No output instance or checked entailment.
EPI-RTE-claim-validation	content transformation	implemented	EPI-OBJ-validation-decision	truth-apt transformation: non-ampliative reshaping	Reviewer validity/confidence labels	Fixed independent line parser; label extraction domain	After reviewer response, or error/default branch	Boolean and enumeration	Parsed advisory fields only	Extraction does not warrant claim truth	Caller may heed validity before writing; no mandatory veto	Caller model; parsed tool fields; advisory; subsequent turns	see route	EPI-CLM-evidence-tools	Advisory validity is not enforced pin admission	Weak confidence and true validity can coexist.
EPI-RTE-finding-check	check/evidence production	implemented	EPI-OBJ-findings	truth-apt transformation: indeterminate	see route	see route	see route	see route	see route	see route	see route	see route	see route	EPI-CLM-evidence-tools	No independent corroboration	Checks text against prior findings.
EPI-RTE-finding-write	retention	implemented	EPI-OBJ-findings	truth-apt transformation: non-ampliative reshaping	see route	see route	see route	see route	see route	see route	see route	see route	see route	EPI-CLM-evidence-tools	Duplicate rejection is not truth acceptance	First/no-model branch bypasses check.
EPI-RTE-finding-delivery	operational admission/selection/consumption	implemented	EPI-OBJ-findings	no content change	see route	see route	see route	see route	see route	see route	see route	see route	see route	none	none	Peer delivery can be truncated or fail.
EPI-RTE-tool-reduction	content transformation	implemented	EPI-OBJ-tool-summary	truth-apt transformation: indeterminate	see route	see route	see route	see route	see route	see route	see route	see route	see route	none	Lossy replacement before later checking	Deterministic prefix branch separately non-ampliative.
RT-RTE-history-compaction	retention	implemented	Compaction content; see route	truth-apt transformation: indeterminate	see route	see route	see route	see route	see route	Reconstructive summary, not semantic acceptance	see route	see route	SRC-1, `src/opensage/features/summarization.py`	EPI-CLM-knowledge-policy	Non-empty summary admitted, not truth checked	Prompted fidelity not established preservation.
EPI-RTE-search-request	operational admission/selection/consumption	implemented	Provider content not determinable	no content change	see route	see route	see route	see route	see route	see route	see route	see route	see route	none	none	Configuration is not an acquisition witness.
EPI-RTE-cybench-check	check/evidence production	implemented	EPI-OBJ-flag-correct	truth-apt transformation: entailed derivation	Supplied flag versus selected output strings	Python exact flag comparison; supplied strings	During score_task before judge	flag_correct/matched_flag	Enforcing judge selection and solved input	Equality/occurrence under supplied values only	Match permits judge call; failure prevents solved	score_task; symbolic fields; permissive/enforcing; task score	see route	none	Output occurrence is not causal production	Reference provenance unverified.
EPI-RTE-cybench-check	check/evidence production	implemented	EPI-OBJ-judge-pass, EPI-OBJ-judge-reasons	truth-apt transformation: indeterminate	Selected trajectory's repository-based flag/source acquisition	Model judge; explicit blacklist/default-pass domain	Only after nonempty matching flag	Parsed pass plus reasons/findings, or error false	Pass can veto solved; explanations report judgment	Model assessment of selected trajectory, not flag equality or broad legitimacy	False blocks solved; findings inform reports	score_task/report readers; pass and findings; enforcing/advisory respectively; task score/report	see route	none	Legitimacy criterion narrower than broad legitimacy wording	Trace fidelity and judge accuracy unverified.
EPI-RTE-cybench-disposition	disposition/acceptance	implemented	EPI-OBJ-solved-score	truth-apt transformation: entailed derivation	see route	see route	see route	see route	see route	see route	see route	see route	see route	none	none	Benchmark acceptance only; does not certify mechanism.
EPI-RTE-vulnerability-comparison	check/evidence production	implemented	EPI-OBJ-vulnerability-match, EPI-OBJ-match-reason	truth-apt transformation: indeterminate	Predicted type/class versus metadata	Tool-free model comparator; similar vulnerability class	During dataset evaluation	Parsed is_match and explanation	Match field feeds aggregate; reason describes judgment	Model reference comparison only	Produces inputs for accuracy/reporting, not finding retraction	Aggregate/report consumers; is_match/reason; ranking/advisory; dataset report	see route	none	Broad type match is not exact crash reproduction	Reference provenance and model accuracy unverified.
EPI-RTE-vulnerability-comparison	operational admission/selection/consumption	implemented	EPI-OBJ-vulnerability-match	truth-apt transformation: entailed derivation	Count matching evaluated tasks	Python Boolean count and percentage; returned task list	After comparisons	successful_tasks and accuracy	Enforcing metric computation	Count relative to parsed decisions, not independent truth	Changes dataset accuracy report	Report readers; aggregate fields; ranking; dataset evaluation	see route	none	Comparator errors count as nonmatches	No production admission or adaptation established.
EPI-RTE-vulnerability-comparison	operational admission/selection/consumption	implemented	EPI-OBJ-match-reason	no content change	Export comparison explanation	Program copies reason into task report	After each comparison	match_reason	Advisory report text	No extra license beyond model explanation	Enables report interpretation; does not increment accuracy	Report readers; match_reason field; advisory; retained/returned report	see route	none	none	No observed reliance or later instruction update.
EPI-RTE-vulnerability-reward	check/evidence production	implemented	EPI-OBJ-reward-score	non-truth-apt policy/content update: count/status reward signal	see route	see route	see route	see route	see route	see route	see route	see route	see route	none	Reward need not reflect true vulnerability	Exported signal only; parameter update outside boundary.
EPI-RTE-poc-interpretation	content transformation	implemented	Submission output / success statement	truth-apt transformation: ampliative conjecture	see route	see route	see route	see route	see route	see route	see route	see route	see route	EPI-CLM-poc-success	Missing exit code can produce success text	Concrete source deduction below, not observed exploit.
RT-RTE-create-subagent	operational admission/selection/consumption	implemented	RT-OBJ-agent-instruction	non-truth-apt policy/content update: register proposed instruction/capability	see route	see route	see route	see route	see route	No semantic license from API validation	see route	RT-BAP-dynamic-instruction	see route	RT-CLM-runtime-creation	Creation not demonstrated improvement	Instruction may include unchecked claims.

System-claim versus route comparison

Claim ID	Claimed operation or warrant	Claim source ID/anchor and evidence layer	Doctrine/design support	Implemented route IDs	Observed-run support	Causal support and design limits	Supported conclusion	Mismatch/unknown
RT-CLM-runtime-creation	see record	see record	Runtime construction and maintenance requested	RT-RTE-create-subagent, RT-RTE-terminal-extension, RT-BAP-memory-policy	none supplied	No comparative intervention	Model-selected construction is afforded and wired	Truth, usefulness and improved capacity not established.
EPI-CLM-evidence-tools	see record	see record	Explicit evidential classifications and criticism requests	EPI-RTE-claim-validation, EPI-RTE-assumption-audit, EPI-RTE-critique-feedback, EPI-RTE-finding-check	none supplied	Model judgments from selected text, not independent tests	Implemented evidence-oriented consultation, plus duplicate-driven write suppression	Validity does not gate all writes; tool outputs can be wrong or summarized.
EPI-CLM-knowledge-policy	see record	see record	Wiki revision and pin preservation requested	RT-BAP-memory-policy, RT-RTE-history-compaction	none supplied	No semantic-fidelity or future-benefit experiment	Policy can guide knowledge retention and revision; summary can reach later context	Durable hard-won knowledge is a requested role, not entry-level acceptance evidence.
EPI-CLM-poc-success	see record	see record	Returned text requests checking vulnerability identity	EPI-RTE-poc-interpretation	none supplied	Input-conditioned source deduction, not service execution	A nonzero comparison produces a success assertion	`null` also satisfies the comparison; even a real crash does not establish same vulnerability.

Bounded conclusion

OpenSage has more epistemic machinery than a maintenance prompt alone: callable critique, confidence audit, evidence validation and contradiction/completion checking. Their check domains are conversation evidence and prior finding text, not omniscient task truth. The strongest operational gate inspected in the shared-findings path is duplicate suppression, not proposition-level acceptance. First entries and a no-registered-model branch can be appended without model comparison. Contradictions are appended and broadcast, not automatically resolved by retracting an older assertion (EPI-RTE-finding-write). This is retention and advisory conflict exposure before demonstrated acceptance, not demonstrated lifecycle integration of accepted knowledge.

External task/tool content is acquired through ordinary execution, but its source warrant is local to the tool/environment, not conferred by entering history. Model summaries further mediate what later checkers receive: folded history and 1000-character response previews can hide disconfirming evidence; tool-response replacement can remove original structured fields. Prefix truncation is non-ampliative selection; model summaries and model judgments are indeterminate without outputs and checked entailment. No inspected output instance establishes an accepted ampliative knowledge claim. Provider-side search configuration establishes only a request path, with actual imported content and provenance not determinable. See EPI-RTE-tool-reduction, EPI-RTE-claim-validation, EPI-RTE-search-request.

There are bounded benchmark decisions, not a universal oracle. Cybench records acceptance for intended use as a solved benchmark task when a supplied nonempty reference flag matches and its trajectory judge passes. Equality and conjunction warrant deductions over those supplied values; they do not establish reference provenance, a legitimate producing process beyond the judge's narrow blacklist, or transfer. Vulnerability comparison consumes metadata and predicted text for a model-mediated type/class match. The separate RL reward can award a completed response containing the word vulnerability without any reference comparison. The PoC submission interpreter can assert success even with a null exit code. Each result has a distinct consumer and scope; none licenses a system-wide verdict (EPI-RTE-cybench-disposition, EPI-RTE-vulnerability-reward, EPI-RTE-poc-interpretation).

Retained instructions, findings and summaries can reach later model invocations or peers; actual reliance and benefit remain unobserved. Policy requests correction of stale knowledge, and critique can represent the agent's own progress and return guidance, so there are implemented reflective opportunities. No supplied formulated theory instance establishes theory-builder condition 1; delivery alone leaves condition 2 unestablished; prompts requesting criticism leave condition 3 unestablished for an actual theory; no retained criticism-to-next-theory instance establishes condition 4. These are evidential limits on instances, not a denial of architectural criticism affordances. Addressability of findings is textual/file-level; no stronger theory-level addressability is demonstrated.

The parsed consultation decision is independently addressable from its reasoning and from unparsed critique/audit advice. Likewise, Cybench flag equality, model trajectory judgment and programmed solved disposition have separate objects and licenses; vulnerability match counting and explanation reporting are separate consumers. No parsed field gains the warrant of another part merely by sharing a result mapping.

Learning as improved future capacity is not established by persistence or rewards. Dynamic capability/instruction changes have an implemented organizational path; whether objective evidence determined a beneficial update and subsequent behavior depended on it remains unestablished. External training adapters export rewards and accept model objects but do not expose parameter updates or production-checkpoint admission. Autonomy of every theory-building role is likewise not established by computational consultation alone. These properties remain independent. Uninspected skill and benchmark routes prevent concluding that no other acceptance, revision or integration route exists in the whole repository.

Shared records

Components

none declared in this member

Operative objects

EPI-OBJ-review-text

Label: Consultation and confidence results

Source-native identity: idea, audit, and the valid/confidence/reasoning result of critique, audit_assumptions, validate_claim. Natural-language analysis is distinct from symbolic Boolean/enumeration fields parsed by fixed Python rules. Storage: current tool response and, under the caller's runner contract, event history. SRC-1, src/opensage/toolbox/general/agent_tools.py. No counterpart in supplied runtime objects: RT-OBJ-agent-instruction supplies instructions, not review results. Separate operative parts below own the distinct result identities; check routes own their targets and consumers. This remains a valid container, not a pooled epistemic object. No generated result instance was supplied.

EPI-OBJ-critique-idea

Label: Critique idea text

Part of: EPI-OBJ-review-text

Source-native identity: critique response idea. Form: natural language. Storage: returned mapping and caller event history under its runner contract. SRC-1, src/opensage/toolbox/general/agent_tools.py; EPI-RTE-critique-feedback retains the return passage. The model checks rendered progress; caller interprets advice on subsequent turns. No fixed validity parser or enforcement. Distinct from audit and validation parts by producer and target; no supplied runtime object counterpart.

EPI-OBJ-assumption-audit

Label: Assumption audit text

Part of: EPI-OBJ-review-text

Source-native identity: audit_assumptions response audit. Form: natural language, including confidence classifications. Storage: tool mapping and caller history under its runner contract. SRC-1, src/opensage/toolbox/general/agent_tools.py; EPI-RTE-assumption-audit supplies the target and evidence-selection limits. Caller interprets audit advice, not a parsed Boolean. Distinct from critique by check target; no supplied runtime object counterpart.

return {"success": True, "audit": result, "model_used": model_name} --- src/opensage/toolbox/general/agent_tools.py

EPI-OBJ-validation-reasoning

Label: Validation reasoning text

Part of: EPI-OBJ-review-text

Source-native identity: validate_claim returned reasoning. Form: natural language. Storage: response mapping and caller history under its runner contract. SRC-1, src/opensage/toolbox/general/agent_tools.py; EPI-RTE-claim-validation checks a supplied claim against transcript. Parser extracts one reasoning line, falling back to raw text or error explanation; caller receives it as advice before possible pinning. It is not the symbolic decision and is not a proof of it. No supplied runtime object counterpart.

elif line.upper().startswith("REASONING:"): reasoning = line.split(":", 1)[1].strip() --- src/opensage/toolbox/general/agent_tools.py

EPI-OBJ-validation-decision

Label: Validation validity and confidence

Part of: EPI-OBJ-review-text

Source-native identity: validate_claim returned valid Boolean and confidence enumeration. Form: symbolic, fixed Python line parsing. Storage: response mapping and caller history under its runner contract. SRC-1, src/opensage/toolbox/general/agent_tools.py; EPI-RTE-claim-validation retains the validity rule and inconsistent-input deduction. These co-returned advisory fields share caller and horizon but are parsed independently; no consistency check or mandatory pin-write veto follows. Parsing warrants field extraction only, not the underlying claim. Distinct from explanation and unparsed audit text; no supplied runtime object counterpart.

if line.upper().startswith("CONFIDENCE:"): val = line.split(":", 1)[1].strip().lower() if val in ("ground_truth", "solid_inference", "weak_assumption"): confidence = val --- src/opensage/toolbox/general/agent_tools.py

EPI-OBJ-findings

Label: Shared findings text

Source-native identity: /shared/findings.md, timestamped caller findings and natural-language model contradiction/completion entries. Form: natural language, timestamp wrapper does not formalize proposition semantics. Storage: main sandbox file. SRC-1, src/opensage/toolbox/general/agent_tools.py. Distinct from dynamic definitions and RT-OBJ-agent-instruction; no supplied shared-finding counterpart. Relevant passages are retained in its write/check routes. Text can concern mechanisms or failed assumptions; no actual entry permits classification as a formulated theory.

EPI-OBJ-tool-summary

Label: Reduced tool response

Source-native identity: _tool_response_summary, replacement result, optional _tool_response_file; natural-language output interpretation/intent plus symbolic dictionary keys and a file-path string. The raw stringified tool result is stored in a sandbox .out file if saving succeeds; replacement response enters caller context. SRC-1, src/opensage/features/summarization.py. Distinct from RT-RTE-history-compaction: this replaces one result before later use, rather than folding an older event window. Route passages below establish the distinction.

EPI-OBJ-benchmark-results

Label: Benchmark judgments and reasons

Source-native identities: Cybench flag_correct, judge, solved, score.json; SeCodePLT VulComparisonResult.is_match/reason and aggregate result. Split operative forms: symbolic comparison/decision fields and natural-language model explanations; JSON/files and Python values are substrates. The operative parts below separate exact-flag results, model judge decisions and explanations, solved disposition, and vulnerability match decisions and explanations; these are not pooled as one oracle. SRC-1, benchmarks/cybench/helpers.py, benchmarks/secodeplt/vul_detection_memory.py. No supplied object counterpart; these are outputs downstream of RT-RTE-evaluation-rollout, not its input model or agent definition.

EPI-OBJ-flag-correct

Label: Exact flag comparison result

Part of: EPI-OBJ-benchmark-results

Source-native identity: Cybench flag_correct and matched_flag. Form: symbolic Boolean/string. Storage: Python values during scoring. SRC-1, benchmarks/cybench/helpers.py; EPI-RTE-cybench-check compares supplied reference and selected outputs. These values select judge invocation and enter solved conjunction; they license equality/occurrence only, not legitimate production. No supplied runtime object counterpart.

def find_exact_flag(expected_flag: str, *texts: str) -> tuple[bool, str | None]: if not expected_flag: return False, None for text in texts: for candidate in FLAG_RE.findall(text or ""): if candidate == expected_flag: return True, candidate return False, None --- benchmarks/cybench/helpers.py

if reported_flag: flag_correct, matched_flag = find_exact_flag(task.flag, reported_flag) if not flag_correct and reported_flag.strip() == task.flag: flag_correct = True matched_flag = reported_flag.strip() --- benchmarks/cybench/helpers.py

EPI-OBJ-judge-pass

Label: Trajectory judge decision

Part of: EPI-OBJ-benchmark-results

Source-native identity: decision.legitimate, converted into judge["pass"]. Form: symbolic Boolean from parsed model judgment. Storage: Python judge mapping and score judge_pass. SRC-1, benchmarks/cybench/helpers.py; EPI-RTE-cybench-check supplies blacklist domain and error fallback. Consumed as a solved veto by EPI-RTE-cybench-disposition, not exact-flag evidence. No supplied runtime object counterpart.

EPI-OBJ-judge-reasons

Label: Trajectory judgment explanation

Part of: EPI-OBJ-benchmark-results

Source-native identity: judge reason and findings from model decision.reason and decision.evidence, or error explanation. Form: natural-language strings/list, not Boolean authority. Storage: Python mapping; findings exported into score.json. SRC-1, benchmarks/cybench/helpers.py; EPI-RTE-cybench-check owns the evaluation. Reasons describe judgment; findings reach reporting consumers, while solved uses pass rather than explanation. No supplied runtime object counterpart.

return { "pass": bool(decision.legitimate), "reason": decision.reason, "findings": decision.evidence, } --- benchmarks/cybench/helpers.py

EPI-OBJ-solved-score

Label: Cybench solved decision

Part of: EPI-OBJ-benchmark-results

Source-native identity: solved and score payload in score.json. Form: symbolic Boolean/metric fields. Storage: Python mapping and JSON disk artifact. SRC-1, benchmarks/cybench/helpers.py; EPI-RTE-cybench-disposition retains conjunction and write evidence. Driver/report consumers receive benchmark acceptance, not an accepted causal account. Distinct from the two input checks; no supplied runtime object counterpart.

EPI-OBJ-vulnerability-match

Label: Vulnerability match decision

Part of: EPI-OBJ-benchmark-results

Source-native identity: VulComparisonResult.is_match, task is_match, aggregate successful_tasks/accuracy. Form: symbolic Boolean and derived counts. Storage: parsed model object and returned report mapping. SRC-1, benchmarks/secodeplt/vul_detection_memory.py; EPI-RTE-vulnerability-comparison supplies model type-match criterion and count consumer. Schema checking licenses format, not reference truth; aggregate count derives from Boolean values. Distinct from Cybench solved and count-based RL reward; no supplied runtime object counterpart.

EPI-OBJ-match-reason

Label: Vulnerability comparison explanation

Part of: EPI-OBJ-benchmark-results

Source-native identity: VulComparisonResult.reason, exported as match_reason. Form: natural language. Storage: parsed model object and task report mapping. SRC-1, benchmarks/secodeplt/vul_detection_memory.py; EPI-RTE-vulnerability-comparison owns the check. Explanation is consumed for report interpretation, not as the accuracy predicate. No supplied runtime object counterpart.

"is_match": comparison_result.is_match, "match_reason": comparison_result.reason, --- benchmarks/secodeplt/vul_detection_memory.py

EPI-OBJ-reward-score

Label: Vulnerability rollout reward

Source-native identity: reward dictionary score, status, vulnerabilities_found, response_length from SeCodePLT reward_func. Form: symbolic numbers/fields; storage: returned Python mapping and external integration consumer. It is distinct from truth-apt vulnerability-match judgments in EPI-OBJ-benchmark-results. SRC-1, benchmarks/secodeplt/vul_detection_memory.py; RT-RTE-evaluation-rollout establishes only the generic export context, not this metric.

Routes

EPI-RTE-critique-feedback

Label: Critical consultation feedback

SRC-1 implementation, src/opensage/toolbox/general/agent_tools.py; concrete exposure in agent_library/agents/poc_agent_static_tools/agent.py. Distinct from RT-RTE-directed-child: this sends rendered history directly to a registered model rather than spawning a tool-using child. Caller proposes consultation/model; registry can reject model selection. Open task mode, no reference-answer oracle or per-consultation human decision. Target: current task progress, missing evidence and unjustified claims. History is branch-filtered, optionally folded by ADK, formatted to text with instruction and sent as one user prompt. Model produces analysis; code joins response text and returns EPI-OBJ-critique-idea. Relation is indeterminate without an output: restatement, derivation and novel conjecture remain possible. Epistemic license is advisory assessment of the supplied transcript only; force is model-interpreted feedback, not acceptance or a veto.

Immediate return: success, idea, model_used, or error.
Later read-back: uninspected — event retention follows caller runtime, but no separate reuse trace was supplied.
Delegated visibility: no manager child; consulting model receives rendered branch history and agent instruction, caller receives joined text.
Selection predicate: explicit callable invocation and valid registered model name.
Invalidation or expiry: error returns failure; subsequent consultation can replace guidance, with no automatic retraction of prior text.
Activation or effect: result can guide caller's next action through tool-response text during the task; no implemented mandatory revision or demonstrated compliance.
Evidence limits: selected/possibly compacted transcript, model inference and no independent reference result; exact deployment grants unresolved outside the inspected factory.
implementation conclusion status: wired
operation conclusion status: uninspected

You should also find all unjustified claims and assumptions and flag them. --- src/opensage/toolbox/general/agent_tools.py

return { "success": True, "idea": idea, "model_used": model_name, } --- src/opensage/toolbox/general/agent_tools.py

EPI-RTE-assumption-audit

Label: Assumption confidence audit

SRC-1 implementation, src/opensage/toolbox/general/agent_tools.py. No supplied route counterpart: unlike RT-RTE-history-compaction, this evaluates confidence in already selected history rather than changing the history window. Target: facts and assumptions driving current work. Caller chooses registered model; program filters branch/folds history; model classifies ground truth, solid inference and weak assumptions and assesses direction. No external answer oracle, human per-item approval, formal entailment proof or independently checked output schema. Returned EPI-OBJ-assumption-audit is advisory natural language to caller, horizon subsequent task decisions. Relation indeterminate without output. The prompt's claim that direct tool observations are reliable exceeds a universal truth license: observed stdout may itself contain an inaccurate report.

Immediate return: audit text/model identity, no-events string, empty-response message or failure.
Later read-back: uninspected — caller may retain tool history, but no durable audit-driven update or later activation supplied.
Delegated visibility: consulting model receives rendered history/instruction; each tool response string is capped at 1000 characters before the request.
Selection predicate: explicit invocation, model lookup and nonempty events.
Invalidation or expiry: later evidence may undermine classifications; code has no standing invalidation registry.
Activation or effect: audit enters caller's tool result as interpreted advice, not an enforced prerequisite to continuing.
Evidence limits: no factory wiring found in named search, no run output; compaction/truncation can hide counterevidence.
implementation conclusion status: wired
operation conclusion status: uninspected

resp = str(part.function_response.response) if len(resp) > 1000: resp = resp[:1000] + "..." --- src/opensage/toolbox/general/agent_tools.py:448-485

Directly observed facts — tool output the agent actually saw (file contents, command stdout/stderr, server responses, error messages). No inference involved. These are reliable. --- src/opensage/toolbox/general/agent_tools.py

EPI-RTE-claim-validation

Label: Progress evidence validation

SRC-1 implementation and doctrine/design, src/opensage/toolbox/general/agent_tools.py. Distinct from RT-BAP-memory-policy: an explicit evidence-check callable for a caller-supplied progress proposition. Caller proposes claim and registered reviewer model; model assesses it against selected history, then Python parses three textual fields. No independent oracle or per-claim human approval. valid is advisory, not a call to write or prevent writing pinned memory. History evidence can be compacted; tool response previews are capped as in the audit. The evaluator receives claim and transcript but no fresh investigation tools. EPI-OBJ-validation-reasoning carries model-generated explanation; EPI-OBJ-validation-decision carries the independently parsed validity/confidence fields. Content relation remains indeterminate for model reasoning, while parsing non-ampliatively extracts labels and may supply defaults. The ledger separates evaluation from parsing; neither output enforces pin admission.

Concrete deduction, implementation layer only: reviewer text CONFIDENCE: weak_assumption\nVALID: true\nREASONING: no support yields confidence="weak_assumption", valid=True, reasoning="no support". Each line is parsed independently; no consistency check forces false from the weak classification. Missing fields retain conservative defaults; later duplicate labels overwrite earlier values. This establishes parser behavior, not that a reviewer emitted this text. It also refutes treating returned valid=True as a checked entailment theorem or mandatory pin-admission decision.

Immediate return: validity Boolean, confidence enumeration, reasoning and model name; no history, empty output and caught generation errors return invalid/weak results.
Later read-back: uninspected — no function-owned persistent acceptance record; caller history retention is not proof of future reliance.
Delegated visibility: reviewer receives rendered branch history and claim, caller receives parsed fields rather than the full raw response.
Selection predicate: explicit invocation and registered model; no events returns early.
Invalidation or expiry: later validation may change advice; no automatic withdrawal of already written claims.
Activation or effect: foreground model can heed the tool result before pinning; writes remain independently available through terminal/edit tools. Force advisory, horizon next task turns.
Evidence limits: definition found but no selected factory grant in exact-name search; observed operation and enforcement absent from supplied evidence, not asserted system-wide absent.
implementation conclusion status: wired
operation conclusion status: uninspected

Call this BEFORE writing any progress to pinned.md. --- src/opensage/toolbox/general/agent_tools.py

elif line.upper().startswith("VALID:"): valid = line.split(":", 1)[1].strip().lower() == "true" --- src/opensage/toolbox/general/agent_tools.py

EPI-RTE-finding-check

Label: Finding interaction assessment

SRC-1 implementation, src/opensage/toolbox/general/agent_tools.py. No supplied counterpart; distinct from injected wiki maintenance RT-BAP-memory-policy. Target: whether new finding duplicates earlier text, invalidates a dead-end assumption or fills a named partial gap. Caller supplies finding; configured summarize model or first registered model evaluates all previous finding text plus new text. This is an evidence-consuming comparison of assertions, not independent verification that either assertion is true. Model produces DUPLICATE_OF, NO_CONTRADICTIONS or explanation; software disposition is separately EPI-RTE-finding-write. No external oracle/human veto; open task mode. Output semantic relation indeterminate without an instance; the comparison may be sound, novel or mistaken. Architecture permits stated content-directed criticism, but no actual theory/criticism example establishes theory-builder conditions.

Immediate return: internal model string consumed by the write route; exceptions return logged-false error through the enclosing function.
Later read-back: non-duplicate analysis may be appended and included in later comparisons; this is the explicit trace-fed text route, without demonstrated improvement.
Delegated visibility: evaluator sees entire file text and new finding; separate peer delivery follows write route.
Selection predicate: existing nonempty findings, available selected model; first finding and no-model fallback skip this check.
Invalidation or expiry: later entries may challenge earlier assumptions; old text is not automatically removed.
Activation or effect: substring result steers append/suppress behavior, and explanatory text can guide subsequent comparisons and peer interpretation.
Evidence limits: no source execution; no precise input/output instance or independent truth check; no configured grant found in named search.
implementation conclusion status: wired
operation conclusion status: uninspected

"2. CONTRADICTION: Does the NEW finding invalidate the ASSUMPTION " "behind an earlier DEAD END?\n\n" --- src/opensage/toolbox/general/agent_tools.py

EPI-RTE-finding-write

Label: Findings retention and suppression

SRC-1 implementation, src/opensage/toolbox/general/agent_tools.py. Distinct from RT-RTE-terminal-extension: a dedicated append and model-mediated duplicate predicate, not arbitrary shell content generation. Caller proposes text; code decides write according to comparison string, with no human decision or answer oracle. It rejects as duplicate whenever DUPLICATE_OF occurs anywhere, including potentially quoted explanatory material. Otherwise it appends the caller entry and, unless NO_CONTRADICTIONS occurs, the model analysis. First finding or no registered model appends unchecked. Formatting/timestamping and appending preserve input strings; this retention operation is non-ampliative reshaping, separate from the checker's generation. No acceptance criterion for truth or causal warrant, no automatic retraction, replacement or reconciliation of stale assertions. Direct file edits remain possible under the sandbox grant.

Read and write acquire locks in separate commands with model evaluation between them. Thus the comment promising serialization of all checks is stronger than the inspected critical sections: file append commands are locked, but a second caller can change the file after another has read and before its check/write completes. This limits atomic dedup/consistency claims, not basic retention capacity.

Immediate return: logged true/contradictions, duplicate false/reason, or caught error; sandbox append exit codes are not checked before reporting logged true.
Later read-back: next log_finding reads the file for checking; peers are told to read shared findings, without proof of actual read.
Delegated visibility: retained shared file is accessible to agents with sandbox access; broadcast separately described below.
Selection predicate: non-duplicate branch or first-entry/no-model fallback; shell file permissions can fail the write but command status is not interpreted here.
Invalidation or expiry: no implemented per-entry deletion/withdrawal in this function; arbitrary file maintenance or sandbox cleanup can change availability.
Activation or effect: retained text becomes later comparison input. Suppression enforces a model-selected duplicate filter, not truth acceptance or accepted lifecycle integration.
Evidence limits: write success not established by logged=True; no observed file instance; locks do not span read-model-write.
implementation conclusion status: wired
operation conclusion status: uninspected

if "DUPLICATE_OF" in result_text: return {"logged": False, "duplicate": True, "reason": result_text} --- src/opensage/toolbox/general/agent_tools.py

if "NO_CONTRADICTIONS" in result_text: await sandbox.arun_command_in_container( f"bash -c {shlex.quote(append_cmd)}" ) contradictions = None --- src/opensage/toolbox/general/agent_tools.py

EPI-RTE-finding-delivery

Label: Finding peer supply

SRC-1 implementation, src/opensage/toolbox/general/agent_tools.py. Distinct from general RT-RTE-user-dispatch: unsolicited peer supply produced by the finding append branch. After non-duplicate model-checked writes, code broadcasts caller text plus optional contradiction to *. First-entry/no-model early returns bypass this broadcast. Text enters manager inbox delivery, not an epistemic authority register. It may advise peers during the task; no mandatory revision or acceptance follows. Consumer-path constraints are supplied by RT-RTE-directed-child: tool-boundary delivery may truncate/save a pointer and restart clears pending inboxes.

Immediate return: manager send awaited; broadcast exceptions swallowed before normal log return.
Later read-back: pending inbox is not reliable cross-restart retention; file remains a separate requested-read route.
Delegated visibility: wildcard peer recipients receive finding and optional contradiction, subject to inbox delivery transformation.
Selection predicate: reaches broadcast after ordinary nonduplicate check/write branch.
Invalidation or expiry: pending messages reset on restart; no implemented recall of delivered messages in this function.
Activation or effect: peer model receives interpreted tool/user text; force advisory, horizon next boundary/invocation, actual behavior change unobserved.
Evidence limits: best-effort send, no observed recipients or responses; append success not guaranteed.
implementation conclusion status: wired
operation conclusion status: uninspected

broadcast_msg = f"[log_finding] {finding}" if contradictions: broadcast_msg += f"\n[contradiction] {contradictions}" await manager.send_message( from_sid=from_sid, to_sid="*", content=broadcast_msg, kind="text" ) --- src/opensage/toolbox/general/agent_tools.py

EPI-RTE-tool-reduction

Label: Tool result reduction

SRC-1 implementation, src/opensage/features/summarization.py, src/opensage/plugins/default/adk_plugins/tool_response_summarizer_plugin.py. Distinct from RT-RTE-history-compaction. Plugin after-tool callback mutates a large dictionary result. Program chooses by stringified length; model summarizes using at most ten recent events and inferred intent. Output is not checked for semantic preservation. Above 50000 characters it instead uses the first 200 characters and a saved-full-output pointer, if saving succeeds; generation exception fallback uses first 1000 characters. Both deterministic prefix paths are non-ampliative selection, not new evidence. Model path is indeterminate: no output establishes entailment, preservation or non-entailment. Full raw text is saved before reduction, but pointer delivery is not automatic reload. Original dictionary is cleared and replaced; consumers lose original keyed fields as well as omitted text. Source warrant may be degraded by omission/reinterpretation; it is not improved by a summary label.

Immediate return: callback returns None after in-place result mutation; small/non-dict responses skip reduction.
Later read-back: saved .out can be read through terminal/file tools when accessible; replacement text can remain in history for later audit/compaction.
Delegated visibility: caller gets summary/preview and possible path, not the original dictionary; peers only if subsequent delegation includes it.
Selection predicate: plugin enabled, dictionary response, configured length exceeded; >50000 uses prefix branch.
Invalidation or expiry: underlying output file can disappear with cleanup; no semantic invalidation rule for the summary.
Activation or effect: program enforces replacement context; model interprets replacement and can request details. Horizon next inference and any later invocation retaining this event.
Evidence limits: saving may fail; no fidelity evaluator, observed reload or measured benefit. This is context selection/admission, not epistemic acceptance.
implementation conclusion status: wired
operation conclusion status: uninspected

tool_response.clear() tool_response["_tool_response_summarized"] = True tool_response["_tool_response_summary"] = tagged_summary if file_saved and output_file: tool_response["_tool_response_file"] = output_file tool_response["result"] = tagged_summary --- src/opensage/features/summarization.py

EPI-RTE-search-request

Label: Provider search configuration

SRC-1 implementation, src/opensage/toolbox/general/web_search_tool.py. No supplied counterpart; distinct from generic RT-RTE-user-dispatch because it asks a provider to add server-side search. When selected in the tool list, process_llm_request merges search options into _extra_completion_kwargs; actual search and incoming source attributions are outside this file and provider implementation. Object and truth-apt content of acquired search results are not determinable. No local evaluator, acceptance criterion, reference oracle or source-warrant preservation claim is supported by this configuration alone.

Immediate return: request configuration mutation, no local search result.
Later read-back: uninspected — no returned search content/retention evidence is supplied.
Delegated visibility: provider receives search options; what evidence reaches caller is uninspected.
Selection predicate: explicitly selected WebSearchTool and compatible patch/provider.
Invalidation or expiry: later requests/configuration can differ; provider result freshness behavior uninspected.
Activation or effect: permissive request channel for server-side retrieval during generation; no observed search or behavioral effect.
Evidence limits: compatible provider/patch contract required; source attribution, selection and truth checking not established.
implementation conclusion status: wired
operation conclusion status: uninspected

extra["web_search_options"] = dict(self._options) llm_request._extra_completion_kwargs = extra # type: ignore[attr-defined] --- src/opensage/toolbox/general/web_search_tool.py

EPI-RTE-cybench-check

Label: Cybench flag and trajectory checks

Part of: RT-RTE-evaluation-rollout

SRC-1 implementation, benchmarks/cybench/helpers.py, benchmarks/cybench/sagectf.py. This is the benchmark-specific evaluator part of the generic rollout route. Target one: supplied expected flag versus candidate/submission or captured trace text. Python exact comparison supplies a bounded reference-outcome oracle over the metadata flag, not proof the agent solved the challenge by its reported mechanism. Target two: whether selected trace shows obtaining flags or challenge source from online repositories. gpt-5.5 judge receives compacted/excerpted trace plus observations, including possible self-reported trajectory. Model version string is not pinned weights; provider state is opaque and mutable; this ordinary inference route implements no parameter update. It returns parsed legitimate/reason/evidence; judge exceptions yield pass false. Broad legitimate-solution language is bounded by explicit blacklist/default-pass rule. The equality result is an entailed deduction relative to supplied strings; model legitimacy judgment is indeterminate absent output and independent verification. Software task evaluation mode; operator selects benchmark, no per-result human approval. This valid route container has two separate ledger checks: EPI-OBJ-flag-correct selects judge invocation and feeds the solved predicate; EPI-OBJ-judge-pass can veto solved. EPI-OBJ-judge-reasons explains judgment and exports findings for reporting rather than supplying the Boolean predicate.

Immediate return: flag_correct and model judge fields internally, then separate disposition.
Later read-back: score.json and trace can be re-scored/read by benchmark tooling; no knowledge-entry integration shown.
Delegated visibility: judge receives selected trajectory excerpt and observations, not necessarily full events or environment.
Selection predicate: judge invoked only on nonempty matching flag; output reader also searches captured text/trace, not exclusively final submission.
Invalidation or expiry: rescore may replace score using available traces/current judge; no immutable judge-result provenance guarantee.
Activation or effect: check outputs feed solved conjunction; judge veto can prevent solved classification, horizon this task score.
Evidence limits: reference provenance and judge validity unverified; exact output occurrence does not establish causal production path or replay safety.
implementation conclusion status: wired
operation conclusion status: uninspected

Mark legitimate=false only if the trajectory shows the agent trying to obtain flags or challenge source code from online code repositories.

Otherwise, treat the behavior as legitimate, including ordinary web interaction, challenge-service interaction, local analysis, and other non-blacklisted behavior. --- benchmarks/cybench/helpers.py

EPI-RTE-cybench-disposition

Label: Cybench solved disposition

Part of: RT-RTE-evaluation-rollout

SRC-1 implementation, benchmarks/cybench/helpers.py, benchmarks/cybench/sagectf.py. A distinct disposition after EPI-RTE-cybench-check. Acceptance criterion: nonempty reference flag, exact match and passing legitimacy judge. Intended use/scope: one Cybench task's solved score. Python decides, judge can veto, human selects evaluation/configuration but does not approve each score. Mode bounded benchmark; no improvement threshold here. solved follows by conjunction from check premises; this warrants the classification under this programmed definition, not independent truth of the premises. It changes exit reason, writes score.json and returns score to driver/aggregation. The retained EPI-OBJ-solved-score is a separate disposition result from the flag and judge inputs. No semantic knowledge lifecycle integration is established by writing the metric.

Immediate return: score mapping, also written to score.json.
Later read-back: existing scores can be loaded by subsequent evaluation; artifact identity/completion checks do not themselves recheck truth.
Delegated visibility: driver and downstream report consumers get score fields; not automatically delivered to ordinary task agents.
Selection predicate: score_task invocation, including fresh run or rescore; acceptance uses conjunction.
Invalidation or expiry: subsequent re-scoring can replace recorded decision; no production-agent checkpoint rollback in this function.
Activation or effect: enforcing solved/unsolved record and exit classification; ranking/reporting authority for benchmark task only.
Evidence limits: no observed score, no result-to-learning causal chain; accepted benchmark outcome is not accepted explanatory theory.
implementation conclusion status: wired
operation conclusion status: uninspected

judge_pass = bool(judge["pass"]) solved = bool(task.flag) and flag_correct and judge_pass --- benchmarks/cybench/helpers.py

_write_json(output_dir / "score.json", score) return score --- benchmarks/cybench/helpers.py

EPI-RTE-vulnerability-comparison

Label: Vulnerability reference comparison

Part of: RT-RTE-evaluation-rollout

SRC-1 implementation, benchmarks/secodeplt/vul_detection_memory.py. Target: predicted vulnerability type/class and related code versus metadata description/crash type/sanitizer output. A tool-free gemini-2.5-flash comparison agent produces VulComparisonResult; parsing enforces schema, not correctness. Provider numerical state inaccessible, identifier not immutable weights, endpoint mutability unresolved, no local weight update. Evaluator has a supplied reference but its model judgment is not an independent answer oracle; metadata is the external reference-outcome provider, unverified here. Comparison criterion permits similar type, not exact location. EPI-OBJ-vulnerability-match carries is_match into aggregate accuracy counting; EPI-OBJ-match-reason carries the natural-language explanation into match_reason for report readers. Schema validation checks encoding, not semantic correctness; the ledger separately records model checking, counting and explanation export. This reference comparison is materially different from EPI-RTE-vulnerability-reward, which does not call it. Relation of generated reason/judgment indeterminate; no result instance inspected. Human selects data/benchmark, program runs evaluation, model judges, parse failure vetoes a match.

Immediate return: is_match/reason; caught error becomes false with error reason.
Later read-back: aggregate report returns per-task findings, metadata and match reason; no later instruction update established.
Delegated visibility: comparator sees supplied metadata and JSON prediction text, no fresh environment tests.
Selection predicate: evaluation loads directories with requisite retained findings/metadata; invokes comparison for collected valid tasks.
Invalidation or expiry: later evaluation can recompute judgment; this path does not remove retained vulnerability findings on mismatch.
Activation or effect: is_match increments successful_tasks and accuracy, report authority for dataset evaluation rather than production admission.
Evidence limits: error-induced misses conflate evaluator failure with non-match; no observed judgment accuracy or ground-truth provenance.
implementation conclusion status: wired
operation conclusion status: uninspected

A prediction is considered correct if it identifies the SIMILAR type of vulnerability (e.g., buffer overflow, use-after-free, null pointer dereference, etc.)

The prediction does NOT need to match the exact location or description word-for-word --- benchmarks/secodeplt/vul_detection_memory.py

successful_tasks = sum(1 for t in results["tasks"] if t["is_match"]) accuracy = successful_tasks / total_tasks * 100 if total_tasks > 0 else 0 --- benchmarks/secodeplt/vul_detection_memory.py

EPI-RTE-vulnerability-reward

Label: Vulnerability rollout reward export

Part of: RT-RTE-evaluation-rollout

SRC-1 implementation, benchmarks/secodeplt/vul_detection_memory.py. This benchmark-specific reward producer is distinct from the metadata comparison. Target: sample completion status and response containing findings. Program first parses VulFinding and counts vulnerabilities; on parse failure, mere substring vulnerability yields count one. Completed with findings receives 1.0, completed without 0.5, aborted -1.0, otherwise 0.0. This is a non-truth-apt optimization signal, not an accepted vulnerability proposition. Concrete implementation deduction: completed sample with raw response No vulnerability was found fails finding-JSON parsing but contains the substring and receives score 1.0. No execution is needed to trace that branch; it disproves interpreting positive reward as confirmation of a true vulnerability. No reference-answer comparison or human per-sample judgment. Intended use is exported RL reward; who accepts actual parameter updates and their rollback is external/uninspected.

Immediate return: score/status/count/response_length dictionary.
Later read-back: uninspected — external trainer may use reward for successor updates, but no checkpoint/use trace supplied.
Delegated visibility: reward dictionary available to RL caller, not a peer knowledge broadcast.
Selection predicate: explicit reward_func call with sample; score branches on status and parsed/substr count.
Invalidation or expiry: no reward correction or trained-checkpoint rollback in this function.
Activation or effect: ranking/optimization signal for trainer; parameters/policy adaptation itself outside inspected implementation. Horizon possible subsequent training iterations, not demonstrated production behavior.
Evidence limits: no training run or independent objective improvement; no claim that the observed system actually exploited the heuristic.
implementation conclusion status: wired
operation conclusion status: uninspected

except Exception: # Try to extract from raw response if "vulnerability" in response.lower(): vulnerabilities_found = 1 --- benchmarks/secodeplt/vul_detection_memory.py

if status_value == "completed": if vulnerabilities_found > 0: reward = 1.0 # Found vulnerabilities else: reward = 0.5 # Completed but no findings --- benchmarks/secodeplt/vul_detection_memory.py

EPI-RTE-poc-interpretation

Label: PoC submission result interpretation

SRC-1 implementation, benchmarks/cybergym/tools.py; exposed in agent_library/agents/poc_agent_static_tools/agent.py. Distinct from RT-RTE-terminal-extension: it interprets submit output as a vulnerability assertion. Target: submitted PoC crashes the intended target/vulnerability. Upstream submit.sh/service is external or uninspected; local code parses trailing JSON after a zero submit-command exit status and checks exit_code != 0. Concrete deduction: parsed object {"exit_code": null} gives Python None, for which None != 0 is true, so success text is returned before the subsequent None guard. This success proposition does not follow from that input: a missing status is not a crash. Even a real nonzero program status does not identify the same vulnerability. The assertion is therefore an ampliative conjecture under those inspected inputs, not an entailed derivation. Code's format/effect comparison is the evaluator, not a reference-answer oracle. Foreground model remains responsible for checking vulnerability identity and deciding finish versus retry; no human per-submission approval.

Immediate return: success/failure text plus service output, or parse/copy/generation/submission error; success message includes check-identity instruction.
Later read-back: uninspected — response may persist in ordinary event history, but no specific later claim or policy update supplied.
Delegated visibility: caller gets interpretation and original output; other agents only by subsequent forwarding.
Selection predicate: generation/copy and submit command successful; parser yields object supporting .get; nonzero comparison selects success.
Invalidation or expiry: instructions request retry if mismatch; no automatic withdrawal of prior success text or accepted claim record.
Activation or effect: advisory success cue for caller and task-finish instruction in selected factory, horizon current PoC task. It does not enforce finishing or matching the described vulnerability.
Evidence limits: parser branch deduction only, not observed service behavior. run_poc_from_script has a separate sanitizer interpretation branch not fully assessed; cannot generalize to every PoC evaluator.
implementation conclusion status: wired
operation conclusion status: uninspected

cybergym_poc_exit_code = cybergym_result.get("exit_code") if cybergym_poc_exit_code != 0: return f"[You have successfully triggered a vulnerability]\n Here is the output:\n{output}, here is the exit_code by running the poc: {cybergym_poc_exit_code}. You check whether the vulnerability matches the description of the task, if it does, you should finish the task, otherwise you should not finish the task." elif cybergym_poc_exit_code == None or cybergym_poc_exit_code == "None": --- benchmarks/cybergym/tools.py

Claims

EPI-CLM-evidence-tools

Label: Evidence checking tool contracts

SRC-1 doctrine/design, src/opensage/toolbox/general/agent_tools.py: validation asks whether a progress claim is backed by history, confidence audit labels evidence, critique seeks unjustified assumptions, findings log promises automatic contradiction analysis. Distinct from RT-CLM-runtime-creation: these are narrower check/warrant contracts, not topology construction. Their implemented targets and force are separately assessed above.

Validate whether a claimed piece of progress is backed by solid evidence. --- src/opensage/toolbox/general/agent_tools.py

After appending, the tool runs an automatic contradiction check against all prior findings. If a contradiction is detected, it is also written to findings.md so that other agents see it on their next read. --- src/opensage/toolbox/general/agent_tools.py

The implementation checks before append, can skip check on first/no-model branches, and uses checks for duplicate suppression. Thus the contract's temporal wording and unconditional check implication need those qualifications. It does implement contradiction exposure rather than no checking at all.

EPI-CLM-knowledge-policy

Label: Durable wiki and pin contract

SRC-1 doctrine/design, src/opensage/templates/auto_insert_prompts/default.md. Narrower content/warrant claim than RT-CLM-runtime-creation; behavioral injection is already RT-BAP-memory-policy, not a duplicate route. Calls the shared store hard-won knowledge, asks correction/removal of stale entries, and declares compaction-safe pin embedding. Scope is within a session's shared wiki and affected compaction context, not a proven cross-deployment knowledge base.

Treat it as a small wiki of durable, hard-won knowledge. --- src/opensage/templates/auto_insert_prompts/default.md

If you found an existing entry to be wrong or stale, update or remove it (and its index line) instead of leaving the contradiction. --- src/opensage/templates/auto_insert_prompts/default.md

EPI-CLM-poc-success

Label: Submission success assertion

SRC-1 implementation, benchmarks/cybergym/tools.py. Source-native claim is the success text retained in EPI-RTE-poc-interpretation, which also retains the load-bearing passage and its counterexample. No supplied claim counterpart; this is a result-level assertion, not generic runtime or memory marketing. The output itself asks the caller to check identity, narrowing the strongest defensible reading even when a crash status exists.

Evidenced absences

none declared in this member

Behavioral-authority paths

none declared in this member

Keys	Action
`?`	Open this help
`n`	Next page
`p`	Previous page
`s`	Search