OrangePro synthesis
Type: agentic-system-analyses/types/agentic-system-synthesis.md
Bounded synthesis
From checkout evidence to a bounded test workflow
OrangePro is a test-evidence tool mechanism with its own bounded orchestration, not an enclosing coding-agent runtime. It builds a local evidence graph, proposes tests, runs selected proof attempts and distinguishes candidate associations from mutation-based credit. This account covers the shipped repository at commit 039cd936e38eb7d52d0e03ded3b3168373d55d77, with an applicability cutoff of 2026-11-08. SRC-1 supplies implementation and doctrine/design evidence. No execution traces, provider replies or causal experiments were supplied, and the analysis did not execute the target. Thus the account establishes source wiring, not successful operation or benefit. The whole-artifact boundary includes local CLI/MCP tools, assisted verification, delivery machinery and hosted-service adapters; it does not reveal the hosted backend, model internals or enclosing agent's policies. See the boundary and evidence and Source register.
The representative workflow is opro start against a customer checkout. CLI and local MCP calls share operation code. Start first decides whether it can return a saved result; otherwise it analyzes source and documentation into graph state, optionally generates and applies AI associations, selects eligible targets, attempts proof, fills remaining draft slots and returns reports and next actions. Code owns these transitions and budgets. Models propose associations and test text rather than choose arbitrary subsequent shell actions. The surrounding coding agent owns any conversational continuation. This is a connected shipped workflow, not a sequence inferred by joining unrelated tools: RT-RTE-start-run and the workflow account.
The early reuse decision matters. A matching saved run can refresh report presentation without new analysis, proof or model calls. Its key incorporates selected code/configuration/ledger inputs, but ordinary workspace outputs are excluded from the code digest; large files can use size and modification time. Changed graph or candidate bytes alone are therefore not guaranteed to invalidate the shortcut. A caller can request fresh computation. See MEM-OBJ-run-key and MEM-RTE-start-reuse.
Selection and proposals are not proof
The local graph is filesystem JSON, not evidence of a graph database or embedding store. It combines symbolic identities, relationships and fingerprints with interpreted descriptions and acceptance text. Generation selects graph evidence by target and relationships, then reads current source excerpts. The inspected selector caps related files at four and excerpts at 8000 characters; this is not a total model-token budget. Weak candidate-edge reasons can reach the model as low-confidence context even though separate candidate flows are excluded from deterministic flow-chain selection. Context is assembled for individual requests, not compacted from the host conversation. See RT-OBJ-graph, MEM-RTE-context-selection and RT-RTE-candidate-admission.
Model proposals pass structural, identity and body checks. Those checks establish neither semantic relevance nor the truth of an assertion's expected behavior. Start applies candidate associations without per-proposal human approval; separate apply affords caller withholding. Draft storage is likewise software-controlled, even when a host later edits or rejects the returned test. Without actual replies, neither entailment nor non-entailment of model-produced assertions is established. Runnable code remains a candidate, not accepted general knowledge. See RT-RTE-test-generation and the epistemic conclusion.
Retained runnable drafts can be returned verbatim when their target fingerprint and flags permit reuse. Full reanalysis selectively preserves eligible drafts, whereas incremental update can retain affected drafts marked stale. Provider resolution precedes serving pinned drafts, so retained content does not establish no-provider availability. These are consumer-specific freshness rules, not a uniform dependency-wide validity invariant: RT-OBJ-generated-tests and its memory annotation.
Execution, rejection and public credit
Automatic proof tries existing associated tests before generating tests for remaining eligible gaps. Missing API credentials alone do not exclude generation: configured Ollama can be keyless, and explicit deterministic mode uses a symbolic offline renderer. No-AI suppresses automatic model generation when a non-deterministic provider resolves, but does not veto explicit deterministic generation; separate gap drafting still requires AI enabled. No-auto disables proof and gap drafting, not earlier AI association passes. These controls govern stages, not a global read-only authorization boundary. See RT-RTE-start-run.
Before automatic generated-test execution, code relocates relative imports, rechecks unresolved imports at the destination, rejects escaping or existing paths and writes admitted files under orangepro_generated/. A draft's non-runnable flag is not an absolute veto if relocation resolves its imports. Generated Go/Java tests lacking the required proof handoff are skipped in this branch; existing-test proof remains separate. Setup failures or red baselines do not earn credit. Classified project-wide blockers can suppress sibling attempts, while a surviving mutant does not cause the same quarantine. Proof exceptions can leave generated files on disk while start continues toward partial reports. See the start workflow annotation and WF-RTE-proof-recovery.
Public Proven credit has a stronger criterion than draft admission or a caller-reported pass. The inspected writer requires a green non-timeout baseline and an assertion-failing non-timeout mutant. It retains a dynamic certificate; the later traceability consumer independently requires closed certificate predicates and a current target fingerprint. Static record_run diagnostics cannot substitute. The resulting authority is narrow test-sensitivity credit and future worklist eligibility, not general correctness. A newer failed attempt need not displace an older matching success. Writable local ledger fields are not authenticated execution history. See RT-RTE-targeted-proof, MEM-OBJ-ledger and RT-BAP-proof-credit.
Execution ownership also qualifies OrangePro's host-owned write/run guidance. Start can write and prove automatically, and generic Go generation can temporarily write a test and start a package test program. In comparison mode, supported foreground, MCP and detached callers reach generation without forwarding static-only. Conditional on a suitable Go target/module, executable response and installed Go, the package-start check remains reachable despite the environment flag. This is source-traced reachability, not an observed bypass; it prevents a whole-product static-only guarantee. See RT-CLM-local-generation and RT-RTE-generation-compare.
Stronger artifact checks and optional delivery
A separate fault workflow checks witness differences, tests a candidate repeatedly on original and faulted copies, checks declared assertion position, restoration, text controls and suite conditions, and can feed classified failures into bounded setup repair. Oracle shape checking accepts proposed or approved metadata; it neither authenticates human approval nor checks that the candidate assertion expresses the oracle's expected value. Those premises remain review obligations. See EPI-RTE-fault-probe, EPI-OBJ-oracle and RT-RTE-draft-verification.
Export adds byte-lineage checks, recomputes the recorded verdict and replays the additive patch on fresh copies before issuing a scoped receipt. Separate publication checks receipt, base and managed branch state, requires confirmation, and opens a draft PR rather than merging. Actions replay uses a configured candidate or returns needs-test; it does not silently call the drafting model. Container test restrictions depend on Docker/OS enforcement and do not cover ordinary local setup or the distinct network-enabled installation lane. See EPI-RTE-export-replay and RT-RTE-actions-delivery.
Assessment at the supported comparison level
The strongest learning-related contribution is wired feedback persistence: execution results become durable proof decisions that affect later credit and selection, and failed candidate feedback can produce persisted repaired tests for subsequent verification/export. These qualify as trace-fed behavior-shaping artifacts without implying weight training or improved capacity. Repair feedback reaches the next model call within an invocation; cross-invocation state preserves attempt counts, not the previous repair conversation. No comparison demonstrates that these routes improve future action. See RT-RTE-draft-verification, RT-BAP-proof-credit and profile verification.
Comparison mode supplies two suites and quality scores, not evidence of causal graph benefit. Fixed lexical metrics have warrant within their string-computation domain; model judgments and fallback scores are quality proxies, not semantic correctness oracles. No winner is automatically installed as new prompt/provider policy. See EPI-RTE-comparison-scoring.
The accepted accounts establish bounded automatic sequencing, not autonomous theory building. They do not establish a formulated-theory loop: localized theory content, decisions depending on that content, content-directed criticism with blame and changed reliance, and retention shaping another round are each unestablished as theory-builder conditions here. Test repair is not itself evidence of those conditions. Nor is improved future capacity from criticism of a consumed theory established. Reflection, requiring a self-representation with both causal directions, and exercised self-improvement of OrangePro's own behavior-determining organization are independently unestablished. External test improvement is not automatically self-improvement. These are limits on conclusions, not system-wide absence findings: included operations remain incompletely assessed. See the epistemic conclusion and report verification.
For a checkout-level question about which selected tests respond to specified mutations, OrangePro supplies an implemented evidence-to-credit protocol and substantial rejection/replay checks. For questions about complete protection, correct expected behavior, production safety or improved agent capability, that evidence is insufficient. Supplied baseline/mutant and replay traces would establish exercised checks; independent semantic review and broader mutation comparisons would change the correctness assessment; a controlled intervention isolating graph context would change the benefit assessment. A traced, retained change to OrangePro's own organization with causally dependent later operation would be needed to establish exercised self-improvement.
Limitations
| Limitation | Affected SRC IDs and record citations | Inspected boundary | Conclusion prevented | Evidence that would resolve it |
|---|---|---|---|---|
| No supplied execution or intervention evidence | SRC-1; RT-RTE-start-run, EPI-RTE-comparison-scoring | Frozen implementation and doctrine, not customer runs | Successful operation, reliable model compliance, causal graph benefit or improved capacity | Linked execution artifacts and appropriately controlled comparisons |
| Mutation and reporter validity remain bounded | SRC-1; RT-RTE-targeted-proof, EPI-CLM-proof-scope | Mint/consumer predicates inspected; language mutators not audited end to end | General correctness, complete protection or independently authenticated proof | Runner/mutator audit, traceable mutation executions and evidence addressing equivalent mutants and reporter limits |
| Oracle semantics and approval provenance unresolved | SRC-1; EPI-OBJ-oracle, RT-RTE-draft-verification | Shape/identity and behavioral checks | That approval is human-authenticated or assertions express the intended expected value | Authenticated provenance and semantic review tied to the candidate |
| Reuse and freshness differ across consumers | SRC-1; MEM-RTE-start-reuse, MEM-OBJ-resolver-cache, RT-OBJ-generated-tests | Keys, preservation, stale marking and pinning; dependency invalidation not exhaustively audited | Universal byte-exact invalidation or dependency/expectation currency | Complete dependency tracing and adversarial invalidation evidence for each consumer |
| Static-only and isolation are route-specific | SRC-1; RT-RTE-generation-compare, RT-RTE-actions-delivery | Compare omission and container argument construction | Global non-execution, rollback or sandbox guarantees | Propagation/enforcement changes covering every executable route, plus deployment enforcement evidence |
| Host, provider and hosted backend remain opaque | SRC-1; RT-RTE-platform-call, RT-CMP-openai, RT-CMP-anthropic, RT-CMP-ollama | Adapter requests and configured endpoints | Backend memory/evaluator classifications, tenant isolation, host context guarantees or provider retention guarantees | Relevant pinned implementations/contracts and inspectable operation evidence |
| Delivery is not adoption | SRC-1; EPI-RTE-export-replay, RT-RTE-actions-delivery | Export receipt and draft-PR path | Merged tests, deployed benefit or per-test human approval | Linked merge/adoption records, approval provenance and downstream outcomes |
| Learning and theory-related inventory is incomplete | SRC-1; RT-RTE-draft-verification, EPI-RTE-comparison-scoring, RT-RTE-platform-call; epistemic coverage | Selected local routes; some coverage, recipe and renderer operations unassessed; backend unavailable | System-complete negatives about knowledge production, learning, reflection, autonomy or self-improvement | Records covering consequential alternatives and, for positive improvement attribution, objective-linked updates and dependent later operation |