Making agent contention visible: what 6 parallel agents and one shared API key taught me about OpenTelemetry span links

Six agents finish "successfully" in SigNoz. The refactorer's diff is on disk, but half its lines are gone because the dep-upgrader wrote the same file a beat later. Nothing in the per-agent trace flags this. We built our own version of that failure and could not see it either, so we wrote SwarmScope.

Build note for the Agents of SigNoz track. Four things I did not expect: a blank OTEL_* env var that broke our exporter, why one big trace for 6 concurrent agents is worse than 6 linked traces, the span attribute that made cross-agent contention computable, and a metric-temporality gotcha in the Query Builder.

LLM calls in the demo are simulated from a static price table. asyncio.sleep for latency, tokens from a range, cost from a per-model multiplier. Everything else (asyncio concurrency, on-disk I/O, lock contention, OTel spans and metrics) is real.

What we built

An OTel SDK, a contention detector, and a Warden that provisions SigNoz artifacts through the SigNoz MCP server. The demo runs 6 agents with distinct roles (refactorer, tester, doc-writer, dep-upgrader, linter, security-scanner) against a small on-disk workspace and one shared "API key". The detector reports four collision kinds: write_write, read_write, duplicate_work, lease_starvation.

flowchart LR
  A[6 agents] -- spans, metrics --> C[OTel Collector :4318]
  C --> S[SigNoz UI :8080]
  W[Warden] -- JSON-RPC --> M[SigNoz MCP :8000]
  M -- create dashboards / alerts / views --> S
  W -- reads metrics via MCP --> M
  W -- writes control.json --> A

Things I would tell my past self

1. Blank OTEL_* env vars silently break the exporter

The first "no data in SigNoz" hunt ended here:

requests.exceptions.MissingSchema:
Invalid URL '/v1/traces': No scheme supplied. Perhaps you meant https:///v1/traces?

An agent harness two shells up had exported OTEL_EXPORTER_OTLP_ENDPOINT= (empty string, not unset). The exporter's default resolution kept the empty string, so the URL became "" + "/v1/traces". Fix, from swarmscope/sdk/tracing.py:

def _env(name: str) -> str | None:
    """os.getenv, but treats blank strings as unset."""
    value = os.getenv(name)
    return value.strip() if value and value.strip() else None

Every OTEL_* variable in the SDK goes through this. Treat empty strings as unset everywhere.

2. One giant trace with 6 concurrent agents is worse than 6 linked traces

Our first version made each agent a child of a swarm.run root. The flamegraph looked like a barcode: six parallel bars starting near t=0, all with children, and the eye kept reading vertical position as parent-child order (it is not, siblings just stack).

OpenTelemetry has a primitive for this: span links associate a span with spans in another trace without making it a child. Each agent root becomes its own trace, linked back to the run:

@contextmanager
def agent(agent_id: str, role: str):
    parent_ctx = _run_span_context.get()
    links = [Link(parent_ctx, {"swarmscope.link": "swarm_root"})] if parent_ctx else []
    span = tracer.start_span(
        SPAN_AGENT,
        context=trace.set_span_in_context(trace.INVALID_SPAN),
        attributes={A_RUN_ID: rid, A_AGENT_ID: agent_id, A_AGENT_ROLE: role},
        links=links,
    )

trace.set_span_in_context(trace.INVALID_SPAN) is the bit that detaches the span from any ambient parent. Each agent gets a readable flamegraph; swarm.run_id on every span keeps the fleet correlatable.

Six independent traces, one per agent, sharing the same swarm.run_id.

3. Contention is uncomputable without swarm.resource_key on every tool span

Vanilla OTel spans do not answer "did two agents fight over the same thing" because the spans have no shared name for the thing. We picked one attribute and made it mandatory:

with tool_call(
    "write_file",
    resource_key=f"file:{target}",        # or apikey:openai-main, row:orders:42
    resource_op="write",                  # read | write
    args={"path": target},
    ledger=self.ledger,
):
    ...

Downstream is a sweep line over spans grouped by (run_id, resource_key). Overlap where one op is write gives write_write or read_write. Same (tool, args_hash) from two agents inside 60s gives duplicate_work. lease_wait_ms > 500 gives lease_starvation. O(n log n + k), in swarmscope/detect/engine.py. A live in-process ContentionLedger twin runs during the swarm so the dashboard lights up in real time.

4. SigNoz returns HTTP 200 with aggregations: null on the wrong temporality

This is the debugging detail I wish I had found in the docs. Query a metric with the wrong temporality and the server returns 200 OK with no error, no warning, empty result. The OTel Python SDK exports counters as cumulative, so a Delta query silently returns nothing:

temporality = "Delta"        -> results[0].aggregations = null
temporality = "Cumulative"   -> series kind=write_write, values 8, 132

Same query, only that field changed. The fix: omit temporality from the builder query and the MCP tool auto-fetches it from the metric's metadata. The v5 response shape, for anyone parsing it by hand:

data.data.results[].aggregations[].series[].labels[].{key:{name}, value}
data.data.results[].aggregations[].series[].values[].{timestamp, value}

Self-hosting SigNoz with Foundry (and a Docker Desktop trap)

install.sh and deploy/docker-compose are deprecated as of v0.130.0. foundryctl is the supported path now. Whole install is one file:

# deploy/casting.yaml
apiVersion: v1alpha1
kind: Installation
metadata: { name: signoz }
spec:
  deployment: { flavor: compose, mode: docker }
  mcp: { spec: { enabled: true } }

foundryctl apply -f deploy/casting.yaml brings up SigNoz UI on :8080, OTLP HTTP on :4318, MCP on :8000/mcp.

Docker Desktop on macOS refused to mount our first working directory:

error while creating mount source path '/Users/.../Desktop/...':
mkdir /Users/.../Desktop: operation not permitted

~/Desktop is protected by TCC even when Docker has Full Disk Access. Moving the checkout to ~/swarmscope fixed it.

Auth chain in order

1. POST /api/v1/register with email, password, name. Creates the first admin.
2. POST /api/v2/sessions/email_password with email, password, and orgID in the body. Missing orgID returns a failure that reads like bad credentials.
3. POST /api/v1/service_accounts with name. Lowercase and hyphens only; underscores or camelCase are rejected.
4. POST /api/v1/service_accounts/{id}/keys for the key.
5. Attach the role via POST /api/v1/service_account_roles with {"serviceAccountId": ..., "roleId": ...}. The nested /service_accounts/{id}/roles route wants a different shape and returned invalid uuid for ours.

Subsequent requests use SIGNOZ-API-KEY: <key>.

Metrics, MCP, and the healing loop

Seven custom metrics, names fixed in swarmscope/sdk/attrs.py:

Dashboards, alerts, and saved Query Builder views are provisioned through the SigNoz MCP server. Write tools we call: signoz_create_dashboard, signoz_create_alert, signoz_create_view, signoz_create_notification_channel. Read tools (signoz_execute_builder_query, signoz_aggregate_traces) close the loop: the Warden queries collision rates back out, picks an action (leases, banning duplicates, capping concurrency), writes .swarmscope/control.json, and agents pick it up on the next task. Each healing action is a swarm.remediation span, so trace and log line up in the Logs tab.

Noz is Cloud-only, so we could not use it against a self-hosted install; fair limitation.

The link at the top jumps to the swarm root; swarm.resource_key on tool spans is what the detector groups by.

Left: contention over time by kind. Right: the "collision rate high" threshold alert firing on a chaos run.

Before and after, same workload

88 to 6 collisions, roughly 25% less simulated cost, same workload. Leases kill write_write and read_write on the hot files. Identical (tool, args_hash) pairs across agents are short-circuited.

Reproduce this

git clone <this repo> swarmscope && cd swarmscope
foundryctl apply -f deploy/casting.yaml            # SigNoz + MCP up
uv sync
uv run swarmscope provision                         # dashboards, alerts, views via MCP
uv run swarmscope demo --chaos   --agents 6 --tasks 6
uv run swarmscope demo --guarded --agents 6 --tasks 6
uv run swarmscope analyze --run-id <chaos_run_id>

Open http://localhost:8080 and filter any panel by the printed swarm.run_id.

Takeaway

Concurrent agents are a distributed system; span links plus one shared resource_key attribute turn them back into one you can reason about. The SigNoz feature that carried the most weight was the MCP server's write tools: agents provision their own observability and read it back to act in the same loop. Shortest path from "agents are opaque" to "agents heal themselves".

Repo: swarmscope/. Docs: docs/ARCHITECTURE.md, docs/DEMO_SCRIPT.md. Span links reference: https://opentelemetry.io/docs/concepts/signals/traces/#span-links.