How to build an AI voice assistant UI in React

A voice assistant needs three things: something that listens, something that answers, and an interface that shows which of the two is happening. This tutorial builds all three in React with no API keys: the browser's built-in Web Speech API does the listening and speaking, and ReactFrame's free AI Voice 01 component is the interface.

By the end you'll have a voice pill that turns on when you tap it, shows a live audio visualizer while you talk, answers out loud, and goes back to idle when it's done. Swapping the canned answer for a real model is one function.

1. Install the component

AI Voice 01 is free and installs with the shadcn CLI:

npx shadcn@latest add @reactframe/ai-voice-01

It drops the full source into components/ai-voice-01.tsx and installs its two small dependencies, clsx and tailwind-merge.

2. How the component thinks

The component has three states, idle, listening and speaking, and two ways to change them:

• Your code sets the status prop.
• The user taps the pill, which toggles between idle and listening and fires onStart (leaving idle) or onStop (back to idle).

Out of the box it also runs a demo flow that walks through the states on a timer. For a real assistant, turn that off with autoFlow={false} so only you and the user move it.

Two more props matter here: input="microphone" makes the visualizer follow the real microphone level while listening, and transcript={false} hides the demo sentences it types by default, so you can show what was actually heard.

3. Wire it to the Web Speech API

"use client";

import * as React from "react";
import { AiVoice01 } from "@/components/ai-voice-01";

type Status = "idle" | "listening" | "speaking";

// Replace this with a call to your model (an API route that calls OpenAI, Anthropic, and so on).
async function getReply(text: string): Promise<string> {
  return `You said: ${text}`;
}

export function VoiceAssistant() {
  const [status, setStatus] = React.useState<Status>("idle");
  const [heard, setHeard] = React.useState("");
  const recognition = React.useRef<any>(null);

  const listen = () => {
    const Recognition = (window as any).SpeechRecognition ?? (window as any).webkitSpeechRecognition;
    if (!Recognition) {
      setHeard("Speech recognition isn't supported in this browser.");
      return;
    }
    const r = new Recognition();
    r.lang = "en-US";
    r.onresult = async (event: any) => {
      if (recognition.current !== r) return; // stopped before the result arrived
      const text = event.results[0][0].transcript;
      setHeard(text);
      const reply = await getReply(text);
      if (recognition.current !== r) return;
      setStatus("speaking");
      const utterance = new SpeechSynthesisUtterance(reply);
      utterance.onend = () => setStatus("idle");
      speechSynthesis.speak(utterance);
    };
    r.onerror = () => setStatus("idle");
    recognition.current = r;
    r.start();
  };

  const stop = () => {
    recognition.current?.abort();
    recognition.current = null;
    speechSynthesis.cancel();
    setStatus("idle");
  };

  return (
    <div className="flex w-full max-w-[380px] flex-col items-center gap-4">
      <div className="h-24 w-full">
        <AiVoice01 status={status} autoFlow={false} input="microphone" transcript={false} onStart={listen} onStop={stop} />
      </div>
      {heard && <p className="text-sm text-neutral-500">{heard}</p>}
    </div>
  );
}

What happens when you tap the pill:

1. The pill moves to listening and calls onStart, which starts speech recognition. The visualizer follows your voice.
2. When recognition returns a sentence, we show it, get a reply, and set status to speaking.
3. The browser reads the reply out loud; when it finishes, status goes back to idle.
4. Tapping again at any point calls onStop, which cancels everything; the recognition.current !== r checks make sure a late result from a cancelled session is ignored.

4. Connect a real model

Only getReply needs to change. Point it at an API route so your key stays on the server:

async function getReply(text: string): Promise<string> {
  const res = await fetch("/api/assistant", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text }),
  });
  const data = await res.json();
  return data.reply;
}

For lower latency and more natural voices, replace the Web Speech API with a realtime speech API (OpenAI Realtime, ElevenLabs, Deepgram and others). The component doesn't change: keep setting status from whatever your audio pipeline is doing.

Browser support and tips

• Speech recognition works in Chrome, Edge and Safari; Firefox doesn't support it yet, so show a fallback (a text input) there.
• The microphone needs a secure context: localhost or HTTPS.
• Keep one voice control per screen, and give it a clear label (the idleText prop) so people know it's tappable.

Other voice styles

AI Voice 01 draws flowing curves. The same API powers the rest of the family, so you can swap the look without changing your code: a waveform, a dot matrix, a 3D particle orb, or a full-size portrait card. See them all in React AI Voice Assistant Components.