SwarmScope Architecture

Why separate traces per agent

The naive approach -- one giant trace with every agent as a child span -- has two problems.

First, a 6-agent swarm running 20 tool calls each produces 120+ spans in one trace. The flamegraph is unreadable. Finding an agent's timeline means hunting through a wall of alternating spans, not reading a clean tree.

Second, and more seriously, a single trace cannot represent concurrency faithfully. OTel spans form a tree; concurrent peer agents are not in a parent-child relationship. Forcing them into one trace means either flattening them all under the run root (losing nesting) or picking an arbitrary hierarchy (lying about causality).

SwarmScope instead gives each agent its own trace. The agent root span carries a span link to the run root span:

links = [Link(parent_ctx, {"swarmscope.link": "swarm_root"})]
span = tracer.start_span(
    SPAN_AGENT,
    context=trace.set_span_in_context(trace.INVALID_SPAN),  # detached root
    links=links,
)

The detached root means the agent trace has its own trace ID and is independently readable. The span link means SigNoz can navigate from any agent span to the run root, and every span carries swarm.run_id so a single filter shows all agents in a run.

Result: per-agent flamegraphs stay readable; cross-agent correlation stays possible.

The resource_key contract

Contention is only computable if both agents name the same resource the same way. Without a canonical key, agent-1 calling edit_file("src/app.py") and agent-2 calling write_file("./src/app.py") are invisible to each other even though they fight over the same bytes.

resource_key is a required-if-you-want-detection attribute:

• file:src/app.py -- a filesystem resource, normalized path
• apikey:openai-main -- a rate-limited credential slot
• row:orders:42 -- a database row
• lock:cache-rebuild -- any named mutex

The SDK does not auto-infer keys. The agent code that calls tool_call(resource_key=...) is making an explicit claim about what it contends over. That is deliberate: inference would be fragile and produce false positives from incidental path matches. Explicit is also more useful as documentation.

args_hash is different: it is computed automatically from the tool arguments via args_hash(args) (SHA-1 of key-sorted JSON, first 16 hex chars). It does not need to match across agents -- two agents calling search_web with the same query will independently produce the same hash, enabling duplicate-work detection without any coordination.

Sweep-line overlap detection

The overlap detector (swarmscope.detect.engine._sweep_overlaps) finds all write-write and read-write overlapping pairs in O(n log n + k) time, where n is the number of spans in a (run_id, resource_key) group and k is the number of overlapping pairs.

Algorithm:

1. Sort spans by start time (ties broken by span_id for determinism).
2. Maintain a min-heap of currently-open spans, keyed by end time.
3. For each incoming span, evict all entries from the heap whose end_ns <= span.start_ns (half-open interval: touching boundary is not overlap).
4. For every remaining entry in the heap, classify the pair. Skip same-agent pairs (an agent nesting its own spans is not contention).

This is the standard sweep-line for interval intersection. The same pattern is used for duplicate-work detection, sweeping over a sliding time window per (run_id, tool_name, args_hash) group using a deque instead of a heap.

Correctness properties tested:

• Exact boundary touching is not overlap
• Nested spans (agent-B entirely inside agent-A's span) are detected
• Three-way overlaps count each pair exactly once
• Same-agent self-overlaps are skipped
• Different run_id or resource_key do not mix

Live ledger vs offline detector

There are two detection paths and they serve different purposes.

ContentionLedger (live, in-process)

swarmscope.sdk.ledger.ContentionLedger maintains a list of currently-held resource entries. On every enter(), it scans the held list for conflicts with the same resource_key and different agent_id. Collision found: emit swarm.contention.collisions metric immediately, annotate both spans with collision metadata.

This is O(n) per entry where n is the number of concurrently-held spans (typically small). It is not the same algorithm as the sweep-line -- it is a live scan of an active set, not a sort-and-sweep over a closed batch.

The live ledger exists because SigNoz metrics become visible in dashboards within a few seconds of emission. A demo without it would require waiting for spans to close, be exported, ingested by ClickHouse, and queried -- 15-30 seconds in a typical setup. The ledger makes collision bars appear in real time.

detect.engine (offline, batch)

swarmscope.detect.engine.detect(spans) is a pure function over a closed batch of SpanRecord objects. It is the authoritative detector used for:

• Post-run audit (feed it spans queried from SigNoz)
• Testing (no side effects, deterministic)
• The Warden's analysis phase (query SigNoz via MCP, convert to SpanRecord, run detect)

The offline detector finds collisions the ledger may have missed (e.g., a duplicate-work pair where one agent completed before the other started) and produces a summarize() dict for dashboard widgets.

MCP provisioning and healing loop

Why MCP for provisioning

SigNoz exposes a Model Context Protocol server that provides tools like create_dashboard, create_alert, and create_saved_view. Using MCP means:

• Dashboard and alert definitions live as code (in the provisioning calls), not as opaque JSON blobs imported through the UI
• The same client code that provisions also reads back alert state, closing the loop
• The Warden can discover what tools the server supports at runtime (tools/list) rather than hardcoding an API surface

The McpClient in swarmscope.warden.mcp_client is a minimal synchronous JSON-RPC 2.0 implementation over Streamable HTTP. It handles both application/json and text/event-stream responses, captures the Mcp-Session-Id header, and exposes call_tool, list_tools, list_resources, and read_resource.

Provisioning flow

swarmscope demo provision
    -> McpClient.initialize()
    -> McpClient.call_tool("create_dashboard", {...})  # contention heatmap
    -> McpClient.call_tool("create_dashboard", {...})  # cost / token dashboard
    -> McpClient.call_tool("create_alert", {...})      # collision rate > threshold
    -> McpClient.call_tool("create_alert", {...})      # agent starvation
    -> McpClient.call_tool("create_saved_view", {...}) # collision traces filter

Idempotent: provisioning checks for existing names before creating.

Healing loop

HealLoop polls on a configurable interval:

1. Query SigNoz (via MCP) for swarm.contention.collisions metric points in the last N seconds, grouped by (run_id, resource_key, kind).
2. For each active run with collisions above threshold, query the relevant traces to get (agent_id, span_id) pairs.
3. Convert to SpanRecord objects and run detect.detect() to classify.
4. Apply the smallest remediation that resolves the conflict:
   • write_write: pause the later-starting writer, requeue behind the current holder
   • duplicate_work: cancel the later duplicate, dedup-skip it
   • lease_starvation: extend or revoke the blocking lease
5. Emit a swarm.remediation span for each action, so the healing is visible in SigNoz.

The healing loop reads from SigNoz and writes back to the swarm -- SigNoz is not just a passive observer but an active participant in the control plane.