The architecture problem nobody talks about
Always-on AI agents like Ambi's background listening apps introduce a fundamentally different system design challenge than chat-triggered assistants. Instead of request-response cycles, you're building a continuous streaming pipeline that must process audio in real-time, maintain context across device handoffs, and enforce privacy boundaries without user intervention at every step. The TechCrunch coverage of Ambi's $26M raise reveals the product ambitions—iPhone, Mac, Apple Watch apps with cross-device continuity, proactive meeting suggestions, and planned hardware—but the architectural decisions underneath determine whether the system survives contact with real users.
Persistent audio streaming: the pipeline that never sleeps
Traditional voice assistants wake on keyword, stream to cloud, return result, terminate. Always-on agents invert this: the stream is default-on, and the architecture must decide continuously what deserves attention. This creates three hard constraints:
- Battery and thermal budget: Continuous microphone capture and on-device preprocessing (VAD, keyword spotting, speaker diarization) must run within mobile power envelopes. Ambi's iPhone and Apple Watch apps imply aggressive duty-cycling—likely running lightweight models on the neural engine while keeping the main CPU asleep.
- Network resilience: The stream cannot assume consistent connectivity. You need a local buffer that survives tunnel transitions (Wi-Fi to cellular, subway dead zones) and a reconciliation protocol when the cloud path restores. This isn't "offline-first"—it's "stream-first with eventual consistency."
- Backpressure and prioritization: Not all audio is equal. Meeting segments need full transcription and LLM reasoning; background chatter might only need embedding updates. The pipeline needs a programmable router that classifies audio segments in real-time and routes them to appropriate compute tiers.
Ambi's claim of "real-time intelligence" during meetings suggests they're running at least partial LLM inference on-device or via edge endpoints with sub-second latency targets. The architecture likely uses a cascaded model: tiny VAD → keyword/intent classifier → selective heavy model invocation.
On-device vs cloud: the privacy-compute trade-off
Ambi states they "don't store any recordings" and "encrypt the data" but don't specify the encryption scheme. This omission matters technically. Three architectural patterns emerge:
| Pattern | Compute location | Data leaves device | Trust model |
|---|---|---|---|
| Full local | Device NPU/GPU | Never (or encrypted blobs for sync) | User trusts device OS |
| Hybrid cascade | Light models local, heavy models cloud | Selective: embeddings, intents, not raw audio | User trusts classification logic |
| Confidential cloud | TEE/secure enclave in cloud | Encrypted audio to attested environment | User trusts attestation chain |
Ambi's cross-device continuity ("start a task on one device and continue on another") implies some form of state sync. If they're not storing recordings, they're likely syncing vector embeddings, action items, and context summaries—derived artifacts, not raw audio. This requires a conflict-free replicated data type (CRDT) or operational transform layer for the context graph, especially with 50+ staff across SF, Beijing, Shanghai, and Shenzhen presumably building this.
The desktop app that "looks at the work users do and stores important parts in memory" adds screen-capture context to the audio stream. This multiplies the privacy surface: now you're correlating audio transcripts with visual activity. The architecture needs a unified context store with per-source retention policies and cryptographic separation so a screen-capture compromise doesn't expose audio history.
Privacy-first architecture: enforcement, not promises
Mo's admission that "the onus of declaring ambient capture is still on the users" and "we, as an industry, haven't solved this problem [of disclosure]" reveals the gap between product intent and architectural enforcement. Technical controls that shift burden from user behavior to system guarantees:
- Hardware attestation of capture state: The microphone LED indicator on Macs is OS-controlled. An always-on agent should expose a signed attestation (via Apple's DeviceCheck or Android's Key Attestation) that the capture pipeline is active, consumable by nearby devices via Bluetooth LE advertisement. This lets third parties detect recording without trusting the user's disclosure.
- Consent-gated stream routing: Before audio leaves the local VAD stage, the router checks a consent manifest: "Is speaker X enrolled? Is this a classified-meeting context? Does jurisdiction Y require two-party consent?" The manifest lives in secure enclave, updated only via user-explicit action.
- Ephemeral processing with cryptographic proof: Process audio in a TEE (AWS Nitro, Azure Confidential Compute, or on-device Secure Enclave), emit only derived artifacts (transcripts, embeddings, action items), and produce a zero-knowledge proof that no raw audio persisted. The proof verifies the code hash matches the open-sourced processing logic.
- Data minimization by default: The "off by default" claim for ambient capture should be enforced at the microphone driver level—a hardware mute that requires explicit user gesture to unmute, not a software toggle.
None of these are implemented by "encrypting the data." They require deliberate architecture from the audio HAL up.
Cross-device continuity: state, not sync
Ambi's "start on one device, continue on another" is deceptively hard. It's not file sync—it's migrating a live agent session with:
- Active audio stream context (speaker embeddings, topic tracking, open action items)
- Partial LLM reasoning state (chain-of-thought intermediates, tool call history)
- Device-specific sensor fusion (watch heart rate + phone audio + desktop screen context)
The architecture needs a session migration protocol: serialize the agent's working memory to a portable format, transfer via end-to-end encrypted channel (likely Apple's AirDrop equivalent or custom WireGuard mesh), and resume on the target device's compute tier. The target device may have different model capabilities (watch vs Mac), so the agent must gracefully degrade or offload heavier reasoning to cloud.
This is where edge orchestration tooling pays off: a unified device fleet API that abstracts model availability, thermal state, and network path so the agent logic stays device-agnostic.
Integration with automation workflows: the agent as event source
The content gap notes few guides address integration with n8n/Make for always-on agents. The pattern: treat the agent as a continuous event emitter, not a request-response service.
Audio stream → VAD → Classifier → [Meeting detected]
→ Emit: meeting.started { participants, calendar_event_id }
→ Stream: transcript.chunk { speaker, text, embedding }
→ Emit: action_item.detected { assignee, due_date, confidence }
→ Emit: meeting.ended { summary, decisions, followups }
Each event is a webhook payload to n8n/Make/Zapier. The automation platform handles idempotency, retry, and human-in-the-loop approval gates. The agent doesn't call APIs directly—it emits typed events with JSON Schema contracts. This decouples agent evolution from downstream workflow changes.
Critical detail: the event stream must include provenance metadata (device ID, model versions, confidence scores, consent status) so downstream systems can make routing decisions. An action item from a low-confidence watch transcript routes to human review; high-confidence Mac desktop capture routes straight to Jira.
Disclosure architecture: making the invisible visible
The industry-wide disclosure problem Mo describes isn't solvable by privacy policies. It requires architectural primitives:
- Proximity beacon: When ambient capture is active, the device broadcasts a rotating BLE beacon with a public key. Nearby devices (with user consent) can detect and display "Ambi active: [User Name]." No central registry—pure peer-to-peer.
- Consent receipts: When a meeting participant explicitly consents (via companion app or voice), the agent issues a signed receipt stored in both parties' local stores. Dispute resolution becomes cryptographic verification, not he-said-she-said.
- Jurisdiction-aware routing: The classifier detects language/region from audio metadata and applies regional consent rules automatically. Two-party consent states (California, Illinois) get stricter gating than one-party states.
These aren't features you bolt on after launch. They shape the audio pipeline's first stage—if the router doesn't know the consent state before the first chunk leaves the device, you've already leaked.
What this means for your build
If you're founding or CTO-ing an always-on agent company, the Ambi case study suggests your technical moat isn't the LLM—it's the plumbing:
- Can you run continuous VAD + speaker ID + intent classification on-device at <5% battery/hour?
- Does your session migration survive a user walking from Mac to iPhone to Watch mid-meeting?
- Can you prove to a regulator (or a lawsuit) that no raw audio left the enclave without consent?
- Does your event schema let n8n workflows treat your agent as a reliable, typed event source?
The integration layer between persistent audio pipelines and business automation is where most teams underinvest. They ship the "magic" demo—proactive suggestions during a meeting—then realize the action items can't reliably reach Asana, the transcripts have speaker attribution drift, and the privacy audit fails because the debug logs contained raw audio.
Build the plumbing first. The magic depends on it.
Frequently Asked Questions
What's the minimum viable architecture for an always-on audio agent?
At minimum: on-device VAD and speaker diarization, encrypted local buffer with 30-minute rolling window, selective cloud routing for classified segments (meeting, command, named entity), and a signed event bus for downstream integrations. Skip the proactive LLM reasoning until the pipeline is bulletproof—bad suggestions delivered reliably are better than great suggestions delivered intermittently.
How do you handle cross-device context without storing raw audio?
Sync derived artifacts only: speaker embeddings (not voiceprints), topic vectors, action item structs, and calendar correlations. Use a CRDT-based context graph so concurrent edits on watch and laptop merge deterministically. The raw audio buffer stays local to each device and expires on a fixed TTL.
Can you technically enforce two-party consent without user action?
Partially. The system can detect jurisdiction via IP/timezone/language, require explicit opt-in for each new speaker voiceprint, and emit proximity beacons so nearby devices can alert their users. But you cannot cryptographically prevent a user from hiding the device. The architecture raises the cost of covert recording; it doesn't eliminate the human factor.
