Why open-source the core?

My Job Odyssey is a live production platform — real users, real data, real career memories. Open-sourcing the entire codebase would expose auth configurations, API routing, and user data structures that have no business being public.

But the voice engine — the Gemini Live integration that powers every conversation with Alex — is something I think other developers should be able to learn from, fork, and build on. So I extracted the core voice pipeline into a public repository that contains everything you need to build your own Gemini Live agent, without exposing the production platform.

This post walks through how it works under the hood.

The architecture in one diagram

Browser (React)
  |
  useGeminiLive hook
  |
  +-- 1. Calls getVoiceContext (Firebase Cloud Function)
  |       +-- Authenticates user via Firebase Auth
  |       +-- Queries career memory from Firestore
  |       +-- Builds enriched system prompt
  |       +-- Issues ephemeral Gemini token (v1alpha authTokens.create)
  |       +-- Returns { systemPrompt, ephemeralToken, model }
  |
  +-- 2. Opens WebSocket to Gemini Live API
  |       Model: gemini-2.5-flash-native-audio-preview
  |
  +-- 3. AudioWorklet captures mic -> PCM 16kHz mono -> base64 -> Gemini
  |
  +-- 4. Receives PCM 24kHz audio + transcripts <- barge-in supported
  |
  +-- 5. Post-session: transcript -> experienceExtractor -> Firestore

The entire voice loop runs on Google Cloud. No third-party services in the critical audio path.

The audio pipeline: AudioWorklet is non-negotiable

Real-time voice in the browser means dealing with the Web Audio API, and specifically AudioWorklet. There is no shortcut here. Older approaches like ScriptProcessorNode introduce latency and run on the main thread. For a voice agent where barge-in matters — where the user can interrupt the AI mid-sentence and get an immediate response — you need the AudioWorklet's dedicated audio thread.

Here's the challenge: Firebase Hosting doesn't serve worker files from arbitrary paths. The standard pattern of loading a separate worklet-processor.js file breaks. The solution is to inline the AudioWorklet processor as a Blob URL. This is a pattern you'll need in any Firebase-hosted app that uses AudioWorklet. It's not elegant, but it's reliable across browsers.

The capture side runs at 16kHz mono PCM. The playback side receives 24kHz PCM from Gemini and schedules AudioBufferSourceNode instances with gapless timing to avoid pops and clicks between chunks. Getting this right took significant iteration — buffer underruns cause audible artifacts, and buffer overruns cause latency drift.

The ephemeral token pattern

This is probably the most important architectural decision in the whole system.

The naive approach is to put your Gemini API key in the frontend. This works for demos. It's also a security disaster for production — anyone can open DevTools and steal your key.

The ephemeral token pattern solves this cleanly:

1. The browser calls a Firebase Cloud Function (getVoiceContext)
2. The function authenticates the user via Firebase Auth
3. The function uses the server-side Gemini API key to call authTokens.create on the v1alpha endpoint
4. This returns a short-lived ephemeral token scoped to a single session
5. The function sends this token back to the browser along with the enriched system prompt
6. The browser uses the ephemeral token to connect directly to Gemini Live

The real API key never leaves the server. The ephemeral token expires. The user gets a direct WebSocket connection to Gemini with server-side access control.

Context injection: making the agent remember

A voice agent without memory is a parlor trick. The difference between "tell me about your week" from a generic chatbot and the same question from an agent that knows you shipped a migration last Tuesday, struggled with a stakeholder on Wednesday, and mentioned wanting to learn Kubernetes three weeks ago — that difference is everything.

Before every voice session, the getVoiceContext Cloud Function queries the user's career memory from Firestore — recent conversation summaries, extracted STAR-format career experiences, current resume content, and critique findings. All of this is compressed into a system prompt that gets sent to Gemini Live at connection time. The agent doesn't just know your name — it knows your career.

Barge-in: the feature that made it real

Early prototypes used a strict turn-taking model. The user talks, then the agent talks. It worked, but it felt robotic. Real conversations don't work that way.

Gemini Live supports barge-in natively. When the user starts speaking while the agent is still talking, the API signals an interruption. The client stops playback immediately, clears the schedule, and begins streaming new input. The agent adjusts its response.

The moment barge-in worked properly was the moment the product clicked. Testers stopped treating Alex like a chatbot and started treating it like a conversation partner.

Function calling in voice: the resume editor

The Voice Resume Editor is where the system gets genuinely agentic. It's not just a conversation — it's a conversation where the AI is simultaneously editing a document.

The Gemini Live session is configured with seven function declarations: updateResumeSummary, addExperienceBullet, removeBullet, addCertification, addProject, flagSectionForReview, and updateSkills. During the conversation, Gemini calls these tools based on the dialogue — while continuing to speak. The function call arrives as a WebSocket event, React state updates, and the resume re-renders with color-coded highlights. All while Alex is still talking.

The system prompt is prescriptive about grounding: only add, modify, or remove content the user explicitly stated or confirmed. A resume editor that invents plausible career details is worse than useless.

Two approaches: Direct SDK vs ADK

The public repository includes two implementations:

Approach A is the production implementation: direct TypeScript using the @google/genai SDK, AudioWorklet for audio, React for UI. This gives full control over the PCM pipeline and the lowest possible latency.

Approach B is a Python implementation using the Google Agent Development Kit (ADK). Same career coach, same tools — but built with Agent, FunctionTool, and AgentRunner abstractions.

We ship Approach A because browser audio demands raw control. But Approach B is cleaner to read and faster to prototype with. Both are documented with setup instructions in the repo.

Try it yourself

git clone https://github.com/flegare/myjobodyssey-agent
cd myjobodyssey-agent/frontend
cp .env.example .env
# Set REACT_APP_DEMO_MODE=true and REACT_APP_GEMINI_API_KEY=your_key
npm install && npm start

Open Chrome, allow mic access, and start talking. Try interrupting the agent. Try saying something you'd never type in a text box. That's the moment you'll understand why voice-first matters.

---

The open-source voice engine is at github.com/flegare/myjobodyssey-agent. The full platform is at myjobodyssey.com. Built for the Gemini Live Agent Challenge 2026.