Make contention between parallel agents observable.
Two agents writing the same file. Six sharing one rate-limited API key. Duplicated identical tool calls. SwarmScope instruments swarms with OpenTelemetry, detects collisions with a sweep line, and ships the whole story to SigNoz.
same workload
Four failure modes single-agent traces cannot show
Each agent trace looks fine on its own. The bug lives between them, in the resource two agents held at the same moment. That correlation is never in one trace.
Concurrent same-file writes
Two agents write the same resource at once. One diff silently overwrites the other. Neither trace shows an error.
refactorer and dep-upgrader both write file:src/db.py
Read torn by a write
One agent reads a resource while another writes it. The reader acts on state that no longer exists a moment later.
tester reads file:src/app.py mid-write
Duplicated identical calls
Two agents run the same tool with identical args inside a window. Every call after the first is wasted tokens and cost.
linter and security-scanner both run scan_repo(depth=2)
Lease starvation
An agent waits past the threshold for a lease held by a long-running peer. From its own trace it just looks slow.
an agent blocked > 500ms behind the apikey lease holder
Three cooperating parts
An OTel SDK that names contended resources, a pure detector that finds overlaps, and a Warden that provisions SigNoz through MCP and heals the swarm.
OTel SDK for swarms
SDKswarmscope/sdk- swarm_run(name) opens a root span and yields a run_id
- agent(id, role) is a per-agent root in its own trace, span-linked to the run
- tool_call(resource_key=...) names the thing two agents can fight over
- ContentionLedger detects collisions live, before either span closes
Contention detector
Detectorswarmscope/detect- Pure function: same spans in, same collisions out, no I/O
- Interval sweep-line, O(n log n + k), not naive O(n squared)
- Four kinds: write_write, read_write, duplicate_work, lease_starvation
- Runs online via the ledger and offline over SigNoz query results
MCP-driven Warden
Wardenswarmscope/warden- Provisions dashboards and alerts through the SigNoz MCP server
- Polls SigNoz via MCP and classifies live contention
- Heals by writing .swarmscope/control.json: leases, dedup bans, concurrency caps
- Emits a swarm.remediation span so each healing action is observable
Replay the real chaos run, then guard it
Agent lanes down the side, time across the bottom. Each bar is a tool or LLM span. Dots are collisions, colored by kind. Toggle to the guarded run to watch the conflicts disappear. Hover any bar or dot for detail.
88 collisions: 66 write/write, 9 read/write, 13 duplicate
Footnote: the demo persisted only aggregate run totals, not per-span timings. This timeline is reconstructed from the real workload structure (6 agents, 4 tasks, one shared key) and a real half-open sweep-line detector, seeded so the per-kind and per-resource totals reproduce the detector output exactly: chaos 66 write/write, 9 read/write, 13 duplicate (88 total); guarded 6 duplicate, 3 prevented. The headline figures elsewhere on this page are the verified aggregates.
Genuine captures from the live instance
Traces, metrics, and the before/after all come from a self-hosted SigNoz recording real SwarmScope telemetry. Two dashboards and four alert rules were provisioned through the SigNoz MCP server.
SigNoz Traces Explorer, service.name = swarmscope-demo. Concurrent swarm.tool and swarm.llm spans share sub-second timestamps: real multi-agent parallelism, one trace per agent.
A swarm.llm span with swarm.agent_id, swarm.resource_key = apikey:openai-main and two swarm.collision events attached: agent-2:write vs agent-3:write on the shared key.
Metrics Summary lists all SwarmScope streams. swarm.contention.collisions leads with 252 samples across 123 time series, recording write conflicts in real time.
The swarm.contention.collisions detail panel. Both the chaos and guarded run ids sit in the same TSDB, split by resource_key and kind, with two alert rules attached.
Left: a chaos-run span dense with swarm.collision events on apikey:openai-main. Right: the guarded run root span with events: [] 0 items. The lease guard eliminated the conflicts.
Span and metric names, copied from attrs.py
These strings are the contract between the SDK, the detector, and the Warden. Changing one silently breaks a dashboard, so they live as constants.
| attribute | constant | meaning |
|---|---|---|
| swarm.run_id | A_RUN_ID | one id per swarm run |
| swarm.agent_id | A_AGENT_ID | stable per-agent id, e.g. agent-3 |
| swarm.agent_role | A_AGENT_ROLE | e.g. refactorer, tester |
| swarm.resource_key | A_RESOURCE_KEY | canonical contended resource |
| swarm.resource_op | A_RESOURCE_OP | read or write |
| swarm.tool_name | A_TOOL_NAME | tool invoked |
| swarm.tool_args_hash | A_ARGS_HASH | sha1[:16] of normalized args |
| swarm.attempt | A_ATTEMPT | retry attempt, 1-based |
| swarm.lease_wait_ms | A_LEASE_WAIT_MS | ms spent waiting for a lease |
| swarm.remediation_action | A_REMEDIATION_ACTION | action taken by the Warden |
| swarm.remediation_reason | A_REMEDIATION_REASON | why the action was taken |
| gen_ai.request.model | GEN_AI_MODEL | LLM model name |
| gen_ai.usage.input_tokens | GEN_AI_IN_TOKENS | input token count |
| gen_ai.usage.output_tokens | GEN_AI_OUT_TOKENS | output token count |
| gen_ai.usage.cost_usd | GEN_AI_COST | USD cost (SwarmScope extension) |
| span | emitted by |
|---|---|
| swarm.run | swarm_run() |
| swarm.agent | agent() |
| swarm.tool <name> | tool_call() |
| swarm.llm | llm_call() |
| swarm.remediation | remediation_span() |
| metric | type |
|---|---|
| swarm.agents.active | UpDownCounter |
| swarm.contention.collisions | Counter |
| swarm.resource.wait_ms | Histogram |
| swarm.tool.calls | Counter |
| swarm.cost.usd | Counter |
| swarm.tokens | Counter |
| swarm.remediation.actions | Counter |
One import makes a Python swarm observable
Name the resource each tool call contends over. The SDK, detector, and dashboards do the rest.
from swarmscope.sdk import agent, tool_call, llm_call
with agent("agent-3", role="refactorer"):
with tool_call(
"write_file",
resource_key="file:src/db.py", # the thing two agents fight over
resource_op="write",
args={"path": "src/db.py"},
ledger=ledger, # live collision detection
):
...
with llm_call("claude-sonnet", api_key_id="openai-main") as span:
span.record_usage(input_tokens=812, output_tokens=240, cost_usd=0.0043)# 1. Deploy SigNoz with the MCP server enabled
foundryctl apply deploy/casting.yaml # UI :8080 OTLP :4318 MCP :8000/mcp
# 2. Install and provision dashboards + alerts through MCP
uv sync
uv run python -m swarmscope.warden.provision
# 3. Run the swarm: chaos first, then guarded
swarmscope demo chaos # 6 agents, no coordination
swarmscope demo guarded # same workload, Warden active
# 4. Filter any SigNoz panel by the printed swarm.run_id