AboutResearchProjectsStackWritingLet's Talk →
London India
Article•5 min read

Building a Real-Time RAG AI Assistant with Supabase Vector Search and Microsoft TTS

Introduction

Over the past few months, I wanted to explore how modern AI applications could feel more natural, responsive, and context-aware instead of behaving like traditional chatbots with slow responses and no memory.

To experiment with this, I built a real-time RAG (Retrieval-Augmented Generation) AI assistant that combines:

  • OpenAI APIs
  • Supabase Vector Database
  • Microsoft Text-to-Speech
  • Streaming AI responses
  • Real-time communication architecture

The goal was simple:

Build an AI assistant capable of retrieving contextual information quickly while supporting natural voice interaction and low-latency responses.

This project helped me better understand:

  • vector search pipelines
  • retrieval systems
  • streaming architectures
  • AI response optimization
  • real-time backend engineering

What Is RAG?

RAG (Retrieval-Augmented Generation) improves AI responses by retrieving relevant context from a knowledge base before generating an answer.

Instead of relying only on the model’s training data, the system:

  1. Searches relevant documents
  2. Retrieves contextual chunks
  3. Injects them into the prompt
  4. Generates a more accurate response

This approach is especially useful for:

  • internal knowledge bases
  • documentation assistants
  • customer support systems
  • AI search platforms

High-Level Architecture

The system follows this flow:

User Input
   ↓
Embedding Generation
   ↓
Supabase Vector Search
   ↓
Relevant Context Retrieval
   ↓
OpenAI Response Generation
   ↓
Streaming Response
   ↓
Microsoft TTS Audio Output

Tech Stack

Frontend

  • Next.js
  • React
  • Tailwind CSS

Backend

  • Node.js
  • Express.js
  • WebSockets
  • OpenAI APIs
  • Supabase Vector Database
  • Embedding-based Retrieval

Audio

  • Microsoft Text-to-Speech

Infrastructure

  • Docker
  • Redis
  • AWS Deployment

Why I Chose Supabase Vector Search

I wanted a solution that was:

  • simple to manage
  • developer friendly
  • production capable
  • fast enough for semantic retrieval

Supabase provided:

  • PostgreSQL-based vector support
  • SQL familiarity
  • good developer experience
  • easy deployment and management

The vector database stores embeddings generated from uploaded content and allows semantic similarity search during user queries.

Retrieval Pipeline

One of the most important parts of the system was the retrieval pipeline.

Step 1 — Chunking Documents

Documents are split into smaller chunks before embeddings are generated.

This improves:

  • retrieval accuracy
  • semantic matching
  • prompt efficiency

Large chunks reduced retrieval quality, while very small chunks lost contextual meaning.

Finding the right chunk size required experimentation.

Step 2 — Generating Embeddings

Each chunk is converted into vector embeddings using OpenAI embedding models.

These embeddings are stored inside Supabase vector tables.

When a user sends a query:

  1. The query is converted into embeddings
  2. Supabase performs similarity search
  3. Relevant chunks are retrieved
  4. Context is injected into the AI prompt

This allows the assistant to respond using relevant contextual information instead of generic answers.

Streaming Responses

One of the biggest UX improvements came from streaming responses.

Instead of waiting for the full AI response:

  • tokens are streamed progressively
  • users see responses immediately
  • perceived latency becomes much lower

Without streaming:

  • responses felt slow
  • voice interaction felt unnatural

Streaming made the application feel significantly more interactive and responsive.

Real-Time Communication

I used WebSockets for:

  • real-time response delivery
  • live AI streaming
  • synchronized voice playback

This reduced communication overhead compared to repeated HTTP polling.

The realtime layer became especially important during:

  • voice conversations
  • long AI responses
  • continuous streaming sessions

Microsoft Text-to-Speech Integration

I integrated Microsoft TTS to make interactions more natural.

The workflow:

  1. AI generates response text
  2. Response is streamed
  3. Text is converted into speech
  4. Audio playback begins progressively

One challenge was balancing:

  • speech quality
  • response speed
  • audio buffering

Long responses initially caused noticeable delays during playback generation.

To improve this:

  • responses were streamed incrementally
  • audio generation started earlier
  • backend processing was optimized asynchronously

Performance Challenges

1. High Latency

Early versions had response times close to 10–12 seconds.

This happened because:

  • retrieval
  • generation
  • TTS processing
  • audio preparation

were all happening sequentially.

2. Blocking Operations

Some backend operations were blocking the response pipeline unnecessarily.

This affected:

  • concurrency
  • realtime responsiveness
  • streaming smoothness

Optimizations

To improve performance, I implemented:

Redis Caching

Frequently accessed data and retrieval results were cached to reduce repeated processing.

Streaming Pipelines

Instead of waiting for full completion:

  • responses streamed progressively
  • TTS processing started earlier
  • frontend rendering became realtime

Asynchronous Processing

Several heavy tasks were moved into async workflows to avoid blocking the main request lifecycle.

What I Learned

This project taught me that building AI applications is not only about calling APIs.

The difficult parts are:

  • latency management
  • retrieval quality
  • realtime communication
  • prompt engineering
  • streaming pipelines
  • infrastructure design

I also learned that user experience matters heavily in AI systems.

Even if the model is intelligent, slow interactions make the system feel weak.

Reducing perceived latency dramatically improved the overall experience.

Future Improvements

Some improvements I plan to explore:

  • hybrid search (semantic + keyword search)
  • response reranking
  • conversation memory optimization
  • distributed websocket scaling
  • advanced caching strategies
  • queue-based audio processing
  • multi-model orchestration

Conclusion

Building this real-time RAG AI assistant helped me better understand:

  • AI system architecture
  • vector databases
  • streaming responses
  • realtime backend systems
  • voice AI workflows

The project evolved far beyond a simple chatbot and became a practical exploration of how modern AI applications can feel more interactive, contextual, and responsive.

It also reinforced how important backend engineering and system design are when building production-grade AI experiences.

Links

Live Demo

https://app.amvinodjaspa.tech/

Tech Stack

  • Next.js
  • Node.js
  • OpenAI
  • Supabase
  • Microsoft TTS
  • Redis
  • WebSockets
  • Docker