hkjarral/AVA-AI-Voice-Agent-for-Asterisk

An open-source AI Voice Agent that integrates with Asterisk/FreePBX using Audiosocket/RTP technology - hkjarral/AVA-AI-Voice-Agent-for-Asterisk

1.1k stars · 246 forks · Python

README

The most powerful, flexible open-source AI voice agent for Asterisk/FreePBX. Featuring a modular pipeline architecture that lets you mix and match STT, LLM, and TTS providers, plus 6 production-ready golden baselines validated for enterprise deployment. Quick Start • Features • Roadmap • Demo • Docs • Community

📖 Table of Contents

🚀 Quick Start 🎉 What's New 🌟 Why Asterisk AI Voice Agent? ✨ Features 🎥 Demo 🛠️ AI-Powered Actions 🩺 Agent CLI Tools ⚙️ Configuration 🏗️ Project Architecture 📊 Requirements 🗺️ Documentation 🤝 Contributing 💬 Community 📝 License

🚀 Quick Start Get the Admin UI running in 2 minutes. For a complete first successful call walkthrough (dialplan + transport selection + verification), see:

Installation Guide Transport Compatibility

1. Run Pre-flight Check (Required)

Clone repository

Run preflight with auto-fix (creates .env, generates JWT_SECRET)

Important: Preflight creates your .env file and generates a secure JWT_SECRET. Always run this first!

2. Start the Admin UI

Start the Admin UI container

Local: http://localhost:3003 Remote server: http://<server-ip>:3003

Default Login: On first start, a one-time admin password is printed to the container logs. Retrieve it with: docker compose -p asterisk-ai-voice-agent logs admin_ui | grep -i password You must change it at first login. Restrict port 3003 via firewall, VPN, or reverse proxy for production use. Follow the Setup Wizard to configure your providers and make a test call.

⚠️ Security: The Admin UI is accessible on the network. Restrict port 3003 via firewall, VPN, or reverse proxy for production use.

4. Verify Installation

GPU users: If you have an NVIDIA GPU for local AI inference, see docs/LOCAL_ONLY_SETUP.md for the GPU compose overlay (docker-compose.gpu.yml) before building.

Start ai_engine (required for health checks)

Check ai_engine health

Expected: {"status":"healthy"} ("degraded" is also possible if a subsystem is unhealthy)

View logs for any errors

docs/Transport-Mode-Compatibility.md

🔧 Advanced Setup (CLI) For users who prefer the command line or need headless setup. Option A: Interactive CLI ./install.sh agent setup

Note: Legacy commands agent init, agent quickstart, agent doctor, agent troubleshoot, and agent demo remain as hidden compatibility aliases. New workflows should use the visible commands documented in docs/CLI_TOOLS_GUIDE.md.

Option B: Manual Setup

Configure environment

Edit .env with your API keys

Start services

Notes:

Use AI_AGENT to select an operator-managed agent. Its configured target is authoritative unless AI_PROVIDER is intentionally set as a per-call override. Generate a current snippet with agent dialplan --agent <slug>. See docs/FreePBX-Integration-Guide.md for channel variable precedence and examples.

Test Your Agent Health check: agent check View logs: docker compose -p asterisk-ai-voice-agent logs -f ai_engine

🎉 What's New

v7.3.3 — Local AI stabilization 🧠 v7.3.3 is a Local-AI-only stabilization release. It adds no providers and keeps the cloud-provider call paths unchanged.

Calls are isolated by session — agent prompts and conversation state no longer mutate shared Local AI Server configuration or leak across reused WebSocket connections. AI Engine and Local AI Server should be upgraded together; the legacy unscoped switch remains temporarily compatible. Barge-in abandons interrupted output — late LLM/TTS work is quarantined, the interrupted exchange is removed from weak-model history, and the replacement turn stays focused on what the caller just said. Farewells finish exactly once — Local hangup_call speaks the selected Kokoro/Piper/etc. farewell without a second LLM rewrite, drains partial AudioSocket or RTP tails, records agent_hangup, and then disconnects. CPU/GPU deployment is safer — dependency pins, CUDA/cuDNN validation, optional llama.cpp architecture targeting, and idempotent preflight checks reduce first-build and rerun failures. Community GPU evidence — Tesla V100S testing passed Faster-Whisper CUDA float16, Llama 3.1 8B Q4_K_M, Kokoro, AudioSocket, ExternalMedia, barge-in, terminal hangup, concurrent session isolation, and restart recovery.

See the Local AI community test matrix and the Unreleased changelog for the complete scope.

v7.3.2 — stabilization release 🛡️ v7.3.2 is a stabilization-only patch release built from the supervised AudioSocket and ExternalMedia validation cycle.

No new providers — scope is limited to reliability, deployment safety, documentation, and contributor-facing CI. Grok ExternalMedia repaired — clean barge-in, cancelled-output quarantine, named-instance runtime inheritance, complete replacement turns, and exact inactivity announcements through xAI force_message. AudioSocket and modular pipelines hardened — terminal playback, pipeline producer ownership, talk-detect echo, and inactivity-grace regressions are covered by focused tests and supervised calls. Updater and provider-failure recovery hardened — safer ownership, rollback/stash handling, readiness validation, and an opt-in dialplan redirect. PR quality gates expanded — Admin backend/frontend checks and CLI cross-compilation now run before merge.

Release evidence and remaining gates are tracked in the v7.3.2 validation matrix.

v7.3.1 — Silence watchdog & safe call endings ☎️ AVA now protects silent calls and finishes every terminal message before disconnecting.

30-second inbound inactivity protection by default — AVA asks “Are you still there?”, waits 15 seconds for a reply, then speaks a configurable final warning and ends the call. Outbound agents remain opt-in. The agent keeps its configured voice — check-ins and final warnings are synthesized by the active Google Live, OpenAI Realtime, Grok, Deepgram, ElevenLabs, local full-agent, or pipeline voice. Transport-safe hangup — watchdog and hangup_call farewells drain AudioSocket or ExternalMedia/RTP streaming buffers and ARI file playback before ARI disconnects the caller. Fixed sleeps no longer clip long final sentences. Deepgram and ElevenLabs lifecycle fixes — Deepgram control frames no longer split greetings, and ElevenLabs response-completion plus hosted-silence handling keeps AVA's watchdog authoritative. Global and per-agent controls — configure defaults under Advanced Settings → Voice Activity Detection → Caller Inactivity, then optionally override them per agent. Call History labels watchdog endings as No input timeout.

See Caller inactivity configuration, ElevenLabs setup, and the full v7.3.1 changelog.

v7.3.0 — Per-agent voices 🎙️ Voice now belongs to agents. Configure one provider, create multiple agents that share it — each with its own voice.

Provider-aware voice picker in the Agent form: a dropdown of OpenAI's 10 GA voices, suggestions + custom clone IDs for Grok, Google Live's 30 prebuilt voices, Deepgram's Aura models — the control adapts to the agent's selected AI Engine. Safe by default — the provider-level voice becomes the default voice; agents without one behave exactly as before. Unrecognized values (OpenAI/Google/Deepgram catalogs are validated) log a warning and fall back — a bad voice value never fails a call. Observable — every call logs the resolved voice and its source, and Call History shows "Voice: marin (from agent)" per call. Agent voice changes apply instantly — no engine restart.

Thanks @foytech for seeding this feature (#497). Full guide: docs/VOICE_SELECTION.md.

v7.2.0 — Live-status dashboard 📡 Real-time system status for the Admin UI — pushed, not polled.

Live-status hub — a single /api/live-status snapshot endpoint plus an SSE stream (/api/live-status/stream) aggregates AI Engine health, Local AI connectivity, active sessions, audio directories, platform checks, and Asterisk ARI into one normalized status feed. Push-first — ai_engine and local_ai_server push their own readiness to the Admin UI (POST /api/live-status/publish, authenticated with LIVE_STATUS_PUSH_TOKEN), so the dashboard converges in sub-second time after a restart instead of waiting on staggered polls. Legacy /api/system/* probes remain as fallback/enrichment. Configurable — LIVE_STATUS_POLL_INTERVAL_SECONDS (default 30 s, min 2 s) and LIVE_STATUS_INITIAL_PROBE_TIMEOUT_SECONDS (default 2 s), read live from .env.

Full notes in CHANGELOG.md.

v7.1.1 — Dashboard reliability & Admin UI polish 🛠️ A focused quality release across the Admin UI — no call-path changes.

Dashboard reliability — the Asterisk status pill no longer flaps on a transient ARI blip: it reads the engine's authoritative, reconnect-supervised ARI state and applies hysteresis. The system endpoints the Dashboard polls every 5s no longer block the admin event loop, the heaviest is TTL-cached, polling backs off on errors, failed polls surface in the error banner, and a single bad poll no longer flashes cards to "Loading…". No more "Loading configuration…" flash — ~11 config pages now seed from a shared stale-while-revalidate cache of the config document, so revisiting a settings page is instant. Accessibility (WCAG AA) — form labels programmatically associated with inputs, a focus-trapping modal, a navigation landmark + "skip to content" link, accessible names on icon-only buttons, non-colour status cues on the topology, a visible dark-mode toggle on-state, and light-mode contrast fixes. Debug console.logs (including one that leaked the auth token to the browser console) were removed. Prompt editor — configured tool names are colour-coded by their in-call status (enabled / global / not-enabled) as you type. Fix (#436) — a canonical google_live: { type: full } provider can be edited and saved again.

Full notes in CHANGELOG.md.

v7.0.0 — the Agents release 🎯 The biggest release yet: manage your AI agents from the Admin UI, not a config file.

🤖 Agents tab — create, edit, and manage agents in the UI. Start from a template (receptionist, after-hours, appointment booker, and more), set the prompt and provider, and copy a ready-to-paste dialplan snippet. (Voice is configured on the provider, not per agent.) 📊 Multi-agent dashboard — live KPIs (active agents, active calls, calls routed, transfers), per-agent stats, and routing breakdowns at a glance. ☎️ New AI_AGENT dialplan variable — route a call to an agent by name. Your existing AI_CONTEXT dialplans keep working unchanged. 🔄 Automatic migration — your existing contexts move into a local agents database on first start. Nothing to do, and rollback is one command. 🔒 Security hardening — no more admin/admin: a one-time admin password is generated and must be changed at first login. Config exports no longer bundle your .env by default.

⚠️ Major release — please read the Upgrade Notes before upgrading from 6.x.

v6.5.4 (2026-05-25) — OpenAI Realtime GA cleanup across every code path Follow-up to the v6.5.3 hotfix. v6.5.3 only flipped config/ai-agent.yaml; v6.5.4 brings the rest of the codebase in line:

Pydantic defaults in src/config.py now default to api_version: ga + model: gpt-realtime (so fresh wizard installs are correct). Admin UI "Add Provider" template for OpenAI Realtime no longer seeds the sunset preview model. Model dropdown removes the 5 sunset preview options and adds 3 new GA models — gpt-realtime-1.5 (best audio-in/audio-out quality), gpt-realtime-2 (reasoning voice model, GPT-5-class), and gpt-realtime-mini (cost-optimized) — alongside the existing gpt-realtime. Legacy preview values in operator YAML now render in a "Custom (legacy — will not connect)" optgroup with a yellow warning banner above the form so the broken state is visible without silently swapping the operator's config. Engine emits a one-shot warning when api_version: beta is detected in config (exactly once per provider lifetime, not per reconnect attempt). Docs: full rewrite of docs/Provider-OpenAI-Setup.md model section + fix to docs/TROUBLESHOOTING_GUIDE.md.

v6.5.3 hotfix (2026-05-25) — OpenAI Realtime restored OpenAI sunset the Realtime Beta API on 2026-05-12 and removed the gpt-4o-realtime-preview-2024-12-17 model on 2026-05-07. Shipped config/ai-agent.yaml still pinned api_version: beta + that preview model, so every operator using OpenAI Realtime hit error.code: beta_api_shape_disabled and the WebSocket closed immediately. Two-line config flip — no code change required. The provider's GA wire-protocol path has shipped since v6.0.0; v6.5.3 just makes it the default everyone gets:

api_version: ga (was beta) model: gpt-realtime (was gpt-4o-realtime-preview-2024-12-17)

If you have an ai-agent.local.yaml that explicitly pins api_version: beta, remove the override or change it to ga. Refs: OpenAI deprecations, gpt-realtime.

v6.5.2 (2026-05-24) — xAI Grok + multi-instance full-agent providers 🆕 xAI Grok Voice Agent realtime provider (NEW, v6.5.2)

Fifth full-agent realtime provider — structurally parallel to OpenAI Realtime and Google Live, built on a multi-instance foundation from day one μ-law @ 8 kHz caller input with no input resampling; observed xAI output is PCM16 @ 24 kHz and AAVA converts it to the configured Asterisk transport format Five named voices (eve, ara, rex, sal, leo) plus custom voice ID free-text for cloned voices Custom function-tools identical to OpenAI Realtime; xAI-native tools (web_search, x_search, file_search, mcp) accepted via YAML extra_tools escape hatch Conservative long-session warning at 28 minutes for compatibility with older xAI limits; xAI's current Voice Agent model page lists a 120-minute maximum session Setup guide: docs/Provider-Grok-Setup.md

🏢 Multi-instance full-agent providers (NEW, v6.5.2)

Run multiple instances of the same full-agent provider type with isolated credentials (e.g. acme_google_live + globex_google_live both using type: google_live) Per-instance credential files at /app/project/secrets/providers/<provider_key>/{api-key,agent-id,vertex-json} — the new per-provider Vertex upload path does NOT mutate .env Route via AI_PROVIDER channel var, contexts.<name>.provider: YAML, or DID-based dispatch with Asterisk Gosub Setup guide: docs/Multi-Instance-Full-Agent-Providers.md Breaking for multi-instance setups: short aliases AI_PROVIDER=openai, AI_PROVIDER=google, provider: deepgram_agent now fail validation — use exact provider instance keys instead. Single-instance setups using the canonical block names are unaffected.

🎛 Admin UI polish (v6.5.2)

Uniform per-instance credentials paste-style uploader across all full-agent provider forms (Grok, OpenAI Realtime, Deepgram, Google Live, ElevenLabs Agent) EnvPage adds a new "Per-Instance Provider Credentials" status section so operators can audit credential file presence without SSH Dashboard System Topology rebuilt: tri-state per-component health with 2-strike debounce (transient probe blips no longer flip dots red), responsive provider grid, multi-instance sub-rows grouped by provider type, Asterisk + AI Engine cards stretched to match Providers height Backend probe timeouts bumped (ai_engine 1.5s → 5s; local_ai_server 2.5s → 5s) to stop legitimate localhost probes timing out under load ~260 inline help tooltips backfilled across provider forms, Setup Wizard, and System pages — new HelpTooltip is viewport-aware (flips placement to keep popovers visible in scrolled modals)

📞 Call recordings (v6.5.2)

Browser playback for compact .ulaw recordings (Asterisk's 8 kHz μ-law output, ~10× smaller than PCM WAV) via server-side audioop.ulaw2lin WAV wrapping — no transcode dependency Uppercase .WAV, compressed WAV, and .gsm recordings transcode via sox; AAVA_RECORDING_TRANSCODE_TIMEOUT_SEC env var (default 120s) governs the timeout

Previously in v6.5.1

💻 CPU-demo profile end-to-end — Faster-Whisper tiny.en + Piper + Qwen 0.5B wired through the Admin UI; runtime Device/Compute selectors with CPU/float16 gating; Filler Audio and LLM/TTS Overlap runtime toggles 🛡️ Local provider hot-path hardening — send_audio() no longer blocks on per-frame reconnect; asyncio.Lock serializes _reconnect() against _send_loop's on-ConnectionClosed path 🎨 Faster-Whisper verify path tolerates the runtime CUDA→CPU fallback so working CPU/int8 configurations no longer get rolled back as "verification failed"

Previously in v6.5.0

🔧 Local LLM tool-gated response (#368) — new WS protocol message types tool_context / tool_result v2; per-WebSocket fail-closed sync prevents cross-call ACL/policy/prompt leakage on reused connections ☁️ Gemini 3.1 Flash Live verified compatible (no engine changes); Vertex AI mode is the production answer for #351 barge-in 🎤 Deepgram Flux v2 + nova-3 default flip; Admin UI surfaces "Flux Turn-Detection Tuning" panel for flux-* models 🩺 Admin UI HTTP-tool-test guard now reads .env first so Environment-page edits to AAVA_HTTP_TOOL_TEST_* take effect without a container restart (#370)

For older releases, expand Previous Versions below. Full release notes in CHANGELOG.md.

Previous Versions v6.4.2 - Microsoft Calendar V1 + Google Calendar overhaul

🗓️ Microsoft Calendar — Outlook / Microsoft 365 integration via device-code OAuth, Graph free/busy, per-context account binding, Tools UI Connect/Verify/Disconnect 📅 Google Calendar — multi-account / per-context binding (#338), JSON upload + auto-discover, Domain-Wide Delegation, native free/busy mode 🎯 Reschedule reliability — server-side event_id resolution + 400/404 fallback eliminates LLM-id-hallucination duplicate bookings 🔧 Date/time prompt placeholders ({today}, {current_date}, etc.) so models stop reasoning with stale years OpenAI Realtime duplicate-events fix (per-response_id async-event gating); per-context tool_overrides now actually take effect on OpenAI Realtime / Deepgram / Google Live; Google Live 30-voice catalog (#349)

v6.4.1 - CPU Latency Optimization

⚡ Streaming LLM→TTS overlap — sentence-boundary token streaming, sub-2s perceived latency on pipelines Pipeline filler audio (instant "One moment please" acknowledgment) configurable via Admin UI Qwen 2.5-1.5B Instruct recommended for CPU; ~15-30 tok/s vs Phi-3's ~0.8 tok/s Direct PCM→µ-law conversion in all 5 TTS backends (10-50ms saved per response) Preflight hardening — Buildx detection, RAM/disk/network checks, GPU install gated behind --apply-fixes

v6.4.0 - Attended Transfer & Russian Speech

📞 Attended transfer with three screening modes: basic_tts, ai_briefing, caller_recording ExternalMedia RTP streaming delivery; provider-agnostic transfer-target tool guidance 🗣️ Russian speech backends: Sherpa Offline STT (VAD-gated), T-one STT, Silero TTS (multi-language) 🎧 Admin UI: fullscreen dashboard panels, per-message conversation timestamps, JSONPath [*] HTTP-tool wildcards

v6.3.2 - Azure Speech & MiniMax LLM

Microsoft Azure Speech Service STT & TTS pipeline adapters (REST batch, WebSocket streaming, SSML) MiniMax LLM M2.7 via

source