Architecture

How audio flows from a phone call through AI agents and back.

The system is a real-time voice pipeline that connects callers to AI agents via Azure Communication Services. Two orchestration modes handle the audio differently, but share the same agent framework, tool registry, and session state.

Show original detailed PNG reference →

Phone Call → ACS → WebSocket → [Orchestrator] → AI Agent → Voice Response
                                     ↓
                              [STT → LLM → TTS]

Call Lifecycle

Every call follows this sequence regardless of orchestration mode:

Key points: Session state lives in Redis for fast access during the call. Transcripts are archived to Cosmos DB when the call ends. The backend is stateless — any container instance can handle any call via Redis session affinity.

Orchestration Modes

Both modes use the same agents, tools, and handoff logic. The difference is how audio gets processed — and how much of the turn loop you control versus hand to Azure.

Pick the mode per deployment with one env var. Everything below the audio surface — agent definitions, the tool registry, HandoffService, and session-state sync into MemoManager — is byte-for-byte identical across modes, so an agent built for Cascade runs unchanged on VoiceLive.

Audio In → Azure Speech STT → Transcript → Azure OpenAI → Text → Azure Speech TTS → Audio Out

Audio In → Azure VoiceLive SDK (STT + LLM + TTS managed) → Audio Out

# Sets the DEFAULT orchestration mode for ACS telephony calls
export ACS_STREAMING_MODE=MEDIA       # SpeechCascade
export ACS_STREAMING_MODE=VOICE_LIVE  # VoiceLive

ACS_STREAMING_MODE only sets the default pipeline that inbound ACS phone calls use; the agent framework is identical across both modes.

Control & tradeoffs at a glance

The two modes are not feature-equivalent. Cascade hands you every component to tune; VoiceLive hands the whole audio loop to Azure for the lowest latency. Pick per workload.

How each mode manages turn context & state

Both modes read and write the same MemoManager via sync_state_from_memo() / sync_state_to_memo() — so active agent, visited agents, profile, and handoff context survive across turns and container restarts. They differ in how the model sees that context on each turn:

• SpeechCascade assembles the prompt itself — current agent's per-agent thread plus de-duplicated cross-agent context (_get_conversation_history), then ships it to Azure OpenAI. You decide exactly what the model sees, so this is where the planned compaction levers (rolling window, summarizer) plug in.
• VoiceLive keeps conversation state inside the realtime connection. The orchestrator re-injects a throttled session recap (recent user messages, slots, last assistant turn) via session.update(instructions=…) before responses, and history is per-connection only — visited_agents is intentionally not replayed to avoid stale context on a fresh call.
• Shared truth. Slots, tool outputs, and the user profile always land in MemoManager regardless of mode, so a handoff or a Cosmos archive looks identical either way.

VoiceLive: native vs cascaded common confusion

"VoiceLive" isn't a single architecture — the model you pick decides how audio is handled. This trips people up constantly, especially around what the transcript actually represents. Explore it below — toggle modes and the Inside VoiceLive options to see how the audio flows.

Agent Framework

Agents are defined in YAML, not Python classes. Scenarios wire them together. Tools give them abilities.

custom Reminder — this is our own lightweight framework, not Microsoft Agent Framework. See the "Why this complexity?" section below for the positioning note and reasoning.

Three concepts

Separation of concerns

• Same agent behaves differently in different scenarios (banking vs. insurance)
• Handoff routing is declarative — change the YAML, not the code
• Tools are shared across all agents via a central registry

Minimal agent definition

# agentstore/my_agent/agent.yaml
name: MyAgent
description: Does something useful
handoff:
  trigger: handoff_my_agent
greeting: "Hi, I'm the specialist you need!"
tools:
  - my_tool
  - handoff_concierge
prompts:
  path: prompt.jinja

Browse all agents → · Create your own →

Storage Tiers

Three tiers, each optimized for its access pattern:

MemoManager handles warm-tier persistence. Never store conversation state in-memory globals — it breaks container scaling.

Session Runtime

How the runtime keeps many concurrent calls isolated, how session memory moves through MemoManager / Redis / Cosmos, and what hooks fire on every turn. Everything below is grounded in the current implementation — the names map 1:1 to functions and events in apps/artagent/backend/voice/.

Concurrent session handling and speech-channel isolation

Each call is isolated by session_id + call_connection_id, with per-session handlers, per-session turn state, and per-session outbound envelopes. This prevents cross-talk when many conversations run concurrently on the same container.

Session memory lifecycle (what persists, when)

The hot path stays in MemoManager. Redis is the cross-container source of truth. Cosmos is cold archive only — never read on the hot path. Reads happen once on session start; writes happen on tool/turn boundaries and on cleanup.

1. Session starts: orchestrator reads prior state (active agent, visited agents, profile, handoff context) from MemoManager/Redis.
2. Each turn: orchestrator updates active agent and session vars, then writes back to MemoManager as turn state evolves.
3. Turn/tool boundaries: effective turn IDs are advanced for post-tool segments to keep message history coherent.
4. Session stop/cleanup: handler flushes MemoManager to Redis asynchronously; cold archival to Cosmos remains post-call oriented.

Per-turn memory flow and long-session strategies

Each turn fires the same fixed sequence of ten calls against MemoManager. The hot path is entirely in-process; Redis is touched once per turn via a fire-and-forget task; Cosmos is only touched after the call ends. Where the system stays honest is also where it has work left to do — there is no automatic history compaction today, so longer sessions grow the per-agent thread monotonically.

What actually happens on every turn (verified against voice/speech_cascade/orchestrator.py)

1. on_final(text) (Cascade) or TRANSCRIPTION_COMPLETED (Voice Live) hands the finalized utterance to the orchestrator.
2. sync_state_from_memo(cm, available_agents=…) hydrates active_agent, visited_agents, session_profile, and any pending_handoff.
3. cm.append_to_history(active_agent, "user", text) writes the user turn into the active agent’s thread and stamps last_activity.
4. cm.get_history(active_agent) reads that agent’s full thread — no windowing, no token cap.
5. _get_conversation_history(cm) merges in substantive user messages from other agents (dedup by lowercased content, filters <10 chars / “welcome…”) so handoffs don’t drop context.
6. _build_session_context(cm) pulls session_profile, caller_name, customer_intelligence, and institution_name out of CoreMemory for prompt-template variables.
7. agent.run(…) calls the LLM with [system, …cross_agent_context, …agent_history, user_turn]; tool calls advance turn_sequence via advance_turn_for_tool().
8. cm.append_to_history(active_agent, "assistant", response) appends the assistant turn into the same per-agent thread.
9. sync_state_to_memo(cm, active_agent=…, visited_agents=…) persists orchestration state back into MemoManager.
10. asyncio.create_task(cm.persist_to_redis_async(redis_mgr)) mirrors MemoManager to Redis without blocking the next turn. Cosmos is untouched until build_and_flush(cm, cosmos) after the call ends.

Why a long session can degrade today

• Per-agent thread is append-only. ChatHistory.append() never trims. A 30-minute call with one agent and ~2 turns/minute lands ~60 user+assistant pairs into a single thread that ships in full on every prompt build.
• Cross-agent merge can stack. _get_conversation_history prepends substantive user messages from every other visited agent. Repeated handoffs grow the prefix even when the active agent’s own thread is short.
• No token budgeting. The Voice Live context_window field (default 4000) only configures the realtime session — it doesn’t cap what we ship to the model. Foundry/AOAI calls in Cascade currently send the whole assembled list.
• What partially protects us today. Per-agent partitioning + handoff-as-natural-reset means short tasks usually finish inside one agent before the thread gets large. That’s a workload assumption, not an architectural guarantee.

Strategies to keep performance flat as context fills up

The four levers below are the planned compaction surface. Each one slots in between step ⑧ (append assistant) and step ⑨ (sync state) so it runs once per turn, off the user-visible critical path.

Where to start: the rolling window is the cheapest defense — a single change in _get_conversation_history bounds prompt growth without any new background work. The summarizer and fact-extractor become valuable once we see real-call traces with >40-turn agents. RAG over Cosmos is the right answer once we have a corpus of past calls worth retrieving from.

Turn-by-turn orchestration hooks: Speech Cascade vs Voice Live

Same logical turn, two different surfaces. Speech Cascade reacts to local on_partial / on_final callbacks coming off the Speech SDK thread; Voice Live reacts to ServerEventType.* messages coming off the realtime model websocket. Both converge on the same session-state contract.

Practical implication: Speech Cascade gives explicit local control over callback and queue boundaries; Voice Live gives explicit realtime event hooks from the model connection. Both converge on the same session-state contract via MemoManager + Redis.

Production Hardening

This section is the honest read on the security posture: what the production reference targets, what the threat surface is, and what the current terraform actually ships with the defaults vs. what is variable-gated or still planned. The repurposed “Azure Services” inventory and App Configuration mapping live here too, since both are part of the hardening story.

Production reference architecture

Show original detailed PNG reference →

Threat surface and mitigations

Eight surfaces to defend, mapped to where the mitigation lives in the codebase.

Hardening status — in place / variable-gated / planned

What the default azd up actually ships today vs. what is one variable flip away vs. what is still reference-only. Use this to size remaining work for a given environment.

Service inventory

The same services the previous “Azure Services” section listed, kept here as the hardening surface they actually live on.

Azure App Configuration mapping (high level)

Runtime settings are loaded from Azure App Configuration using hierarchical keys, then synchronized to environment variables used by the service. Local .env.local values can still override for developer workflows.

Full key coverage is defined in apps/artagent/backend/config/appconfig_provider.py in APPCONFIG_KEY_MAP and FEATURE_FLAG_MAP.

Cross-cloud integration

The accelerator supports hybrid deployments. AWS Connect traffic can be routed to Azure via a Certified Session Border Controller — Azure handles the AI workload while the carrier path and call routing stay on the partner side.

Performance Targets

This is a real-time voice system. Latency kills user experience.

Why this complexity?

If you've skimmed the codebase and wondered why there's a custom orchestrator, per-stage streaming, a pool layer, a VAD layer, and a barge-in cancellation path — this section is the answer. Real-time voice has constraints that don't show up in chat or batch-LLM systems, and those constraints drove every structural choice.

1 · The framework choice — own the loop, not the abstraction

Most agent frameworks (Microsoft Agent Framework, LangChain, etc.) hand you a single agent.run() call that returns when the whole turn is done. That's perfect for chat and async work. For voice, it's the wrong shape — you lose the seams where latency and interruption live.

Opinionated note: if you're starting greenfield and don't need per-turn audio control, Agent Framework is worth evaluating first. If you need a realtime voice loop with barge-in and fine-grained telemetry, keep the custom orchestrator and integrate at the workflow boundary.

Plugging in Agent Framework, Foundry agents, and Voice Live

You don't have to choose "all custom" or "all managed." The realtime loop above has clean seams where Microsoft's managed building blocks could slot in — each with its own latency profile to weigh against the ~1-second turn budget. Today the reason + tool stage runs ART's own custom agent framework; the items below are the supported plug-in points, some shipped and some potential.

• Agent Framework at the reason stage potential · not current state. MAF is two things — agents (LLM + tools/MCP) and graph workflows (type-safe routing, checkpointing, human-in-the-loop) — built on model clients, an agent session for state, middleware, and MCP clients. It maps cleanly onto our reasoning + tool-dispatch stage, so it's a natural future drop-in: keep the audio transport here, and let MAF own the agent graph, calling the Responses API on your Foundry project endpoint. This accelerator does not use MAF today — it ships its own custom, latency-tuned agent framework around the realtime loop; MAF is shown here as where it would plug in.
• Voice Live as a managed audio channel. The Voice Live API is exposed as a Foundry Agent Service tool that collapses speech-in and speech-out into one managed WebSocket hop (Realtime-API compatible), with 600+ Azure neural voices, custom voice, and models like gpt-realtime. Because the orchestrator supports both pipelines, Voice Live can take over the entire audio channel — VAD, STT, and TTS — while MAF still sits in the middle to customize the agentic layer. Use it when you want Azure to own the whole audio surface; use the custom Speech cascade when you need per-component control.

Hosted agents vs. prompt agents — where every millisecond counts

Foundry Agent Service offers two agent types, and the difference matters for voice. A prompt agent is configuration only (instructions + model + tools) — fully managed, nothing to scale, but you call it over the Responses API, so every turn pays a network hop and you can't reshape the orchestration. A hosted agent is your container (MAF, LangGraph, or custom) run by Foundry with a managed endpoint, autoscale, and a dedicated identity — great performance and full control once it's running.

2 · The 1-second turn budget

Conversational research is clear: anything over ~1 second between the caller finishing their sentence and hearing a reply feels like lag and breaks turn-taking. That cap drives everything downstream.

3 · Barge-in — the user can talk over the agent

The instant the caller starts speaking, the agent has to stop, drain its audio buffer, and start listening again — all within roughly 150 ms or it feels rude. This is the single biggest reason for the custom orchestrator: we have to be able to cancel in-flight TTS at any moment, which means we have to own the handle.

Key Design Decisions