The Traditional AI Approaches

// Expensive, inflexible, requires retraining for updates
const fineTunedModel = await openai.fineTuning.jobs.create({
  training_file: "portfolio_data.jsonl",
  model: "gpt-3.5-turbo",
  // Cost: $100+ for training, $0.02 per 1K tokens
});

// Limited context, prone to hallucination
const response = await openai.chat.completions.create({
  model: "gpt-4",
  messages: [
    {
      role: "system",
      content:
        "You are David Kwon. Here's a summary of his work: [static summary]",
    },
  ],
  // Problem: Static knowledge, no access to detailed content
});

// Complex setup, requires vector database
const embedding = await openai.embeddings.create({
  model: "text-embedding-ada-002",
  input: portfolioContent,
});
// Problem: Overkill for structured content, expensive

The RAG Solution

RAG Pipeline Flow

Here's the complete sequence of how your question flows through the RAG system:

📚 Document index built: {
  totalChunks: 62,
  totalWords: 125000,
  sources: { blogs: 45, resume: 17, projects: 0 }
}

🔍 RAG Retrieval: {
  query: "What AWS experience do you have?...",
  chunksRetrieved: 5,
  avgScore: 15.4,
  sources: ["AWS Lambda Experience", "Serverless ETL", "Cloud Architecture"]
}

📊 RAG Retrieval Details: {
  query: "What AWS experience do you have?...",
  chunksRetrieved: 5,
  avgScore: 15.4,
  sources: ["AWS Lambda Experience", "Serverless ETL", "Cloud Architecture"]
}

Architecture Overview

System Components

graph TB
    A[User Question] --> B[Document Indexer]
    B --> C[RAG Service]
    C --> D[Relevance Scoring]
    D --> E[Context Retrieval]
    E --> F[OpenAI GPT-4o-mini]
    F --> G[Response as David]

    H[Blog Posts] --> B
    I[Resume Data] --> B
    J[Project Data] --> B

    B --> K[Document Chunks]
    K --> L[In-Memory Cache]
    L --> C

Detailed Pipeline Flow

sequenceDiagram
    participant U as User
    participant F as Frontend
    participant API as API Route
    participant RAG as RAG Service
    participant IDX as Document Index
    participant AI as OpenAI GPT-4o-mini

    U->>F: "What AWS experience do you have?"
    F->>API: POST /api/chat
    API->>RAG: getRelevantContext(query)
    RAG->>IDX: Search 62 chunks
    IDX-->>RAG: Return all chunks
    RAG->>RAG: Calculate relevance scores
    RAG->>RAG: Sort by score (avg: 15.4)
    RAG->>RAG: Select top 5 chunks
    RAG-->>API: Return context + metadata
    API->>AI: Send prompt with context
    AI-->>API: Generate response
    API-->>F: Return response
    F-->>U: Display answer as David

Core Technologies

Implementation Deep Dive

1. Document Indexing System

The foundation of our RAG system is intelligent document chunking that preserves context while creating searchable segments.

// lib/services/document-indexer.ts
export interface DocumentChunk {
  id: string;
  content: string;
  metadata: {
    source: string;
    sourceType: "blog" | "resume" | "project";
    title: string;
    category?: string;
    tags?: string[];
    wordCount: number;
    dateAdded?: string;
  };
}

function chunkText(
  text: string,
  maxChunkSize: number = 1000,
  overlap: number = 200
): string[] {
  const words = text.split(/\s+/);
  const chunks: string[] = [];

  // Create overlapping chunks for context preservation
  for (let i = 0; i < words.length; i += maxChunkSize - overlap) {
    const chunk = words.slice(i, i + maxChunkSize).join(" ");
    if (chunk.trim()) {
      chunks.push(chunk);
    }
  }

  return chunks.length > 0 ? chunks : [text];
}

Key Design Decisions:

2. Relevance Scoring Algorithm

Our custom scoring system combines multiple signals to find the most relevant content:

export function calculateRelevanceScore(
  query: string,
  chunk: DocumentChunk
): number {
  const queryWords = query.toLowerCase().split(/\s+/);
  const contentLower = chunk.content.toLowerCase();
  const titleLower = chunk.metadata.title.toLowerCase();
  const tags = chunk.metadata.tags || [];

  let score = 0;

  // Exact phrase match (highest weight)
  if (contentLower.includes(query.toLowerCase())) {
    score += 10;
  }

  // Title matches (high weight)
  queryWords.forEach((word) => {
    if (titleLower.includes(word)) {
      score += 5;
    }
  });

  // Tag matches (medium weight)
  queryWords.forEach((word) => {
    if (tags.some((tag) => tag.includes(word) || word.includes(tag))) {
      score += 3;
    }
  });

  // Content word matches (base weight)
  queryWords.forEach((word) => {
    if (word.length > 3 && contentLower.includes(word)) {
      score += 1;
    }
  });

  // Contextual boosting based on query intent
  const queryLower = query.toLowerCase();

  if (queryLower.includes("experience") || queryLower.includes("work")) {
    if (chunk.metadata.category === "experience") {
      score += 5; // Boost experience chunks for work queries
    }
  }

  if (queryLower.includes("skill") || queryLower.includes("technology")) {
    if (chunk.metadata.category === "skills") {
      score += 3; // Boost skills chunks for tech queries
    }
  }

  return score;
}

Scoring Strategy:

3. RAG Service Implementation

The RAG service orchestrates retrieval with intelligent caching and detailed metadata:

// lib/services/rag-service.ts
export interface RetrievalResult {
  chunks: DocumentChunk[];
  scores: number[];
  context: string;
  metadata: {
    totalChunksSearched: number;
    chunksRetrieved: number;
    maxScore: number;
    minScore: number;
    avgScore: number;
  };
}

// Cache the index to avoid rebuilding on every request
let cachedIndex: DocumentIndex | null = null;
let lastIndexTime: number = 0;
const INDEX_CACHE_TTL = 5 * 60 * 1000; // 5 minutes

export async function getRelevantContext(
  query: string,
  topK: number = 5
): Promise<RetrievalResult> {
  const index = await getDocumentIndex();

  // Calculate relevance scores for all chunks
  const scoredChunks = index.chunks
    .map((chunk) => ({
      chunk,
      score: calculateRelevanceScore(query, chunk),
    }))
    .filter((item) => item.score > 0) // Only include relevant chunks
    .sort((a, b) => b.score - a.score) // Sort by highest score first
    .slice(0, topK); // Take top K results

  const chunks = scoredChunks.map((item) => item.chunk);
  const scores = scoredChunks.map((item) => item.score);

  // Build context string with metadata
  const context = chunks
    .map((chunk, index) => {
      return `[Source: ${chunk.metadata.title}]
[Type: ${chunk.metadata.sourceType}]
[Relevance Score: ${scores[index]}]
Content:
${chunk.content}`;
    })
    .join("\n\n---\n\n");

  return {
    chunks,
    scores,
    context,
    metadata: {
      totalChunksSearched: index.metadata.totalChunks,
      chunksRetrieved: chunks.length,
      maxScore: scores.length > 0 ? Math.max(...scores) : 0,
      minScore: scores.length > 0 ? Math.min(...scores) : 0,
      avgScore:
        scores.length > 0
          ? scores.reduce((sum, s) => sum + s, 0) / scores.length
          : 0,
    },
  };
}

Performance Optimizations:

4. OpenAI Integration

The API route handles the complete RAG pipeline with detailed logging:

// app/api/chat/route.ts
export async function POST(request: NextRequest) {
  try {
    const { messages } = await request.json();
    const lastUserMessage = messages
      .filter((m: { role: string }) => m.role === "user")
      .pop();

    // Get relevant context using RAG
    const retrievalResult = await getRelevantContext(
      lastUserMessage.content,
      5
    );
    const relevantContext = retrievalResult.context;

    // Log retrieval details for debugging
    console.log("📊 RAG Retrieval Details:", {
      query: lastUserMessage.content.substring(0, 50) + "...",
      chunksRetrieved: retrievalResult.metadata.chunksRetrieved,
      avgScore: retrievalResult.metadata.avgScore.toFixed(2),
      sources: retrievalResult.chunks.map((c) => c.metadata.title),
    });

    // Create system prompt with context
    const systemPrompt = `You are David Kwon, a Full-Stack Engineer & Cloud Architect. You are answering questions about yourself in first person.

Your expertise includes:
- Full-stack development with React, TypeScript, Next.js, Go
- Cloud architecture with AWS (Lambda, DynamoDB, CloudFront, Step Functions, etc.)
- Building scalable systems, GraphQL APIs, serverless architectures
- System design, performance optimization, and developer experience

IMPORTANT INSTRUCTIONS:
- Only use information from the context below - do NOT make up facts
- If the context doesn't contain enough information, say "I don't have that information in my knowledge base"
- Be professional but conversational
- Reference specific projects, blog posts, or experiences when relevant
- Keep responses concise but informative (2-3 paragraphs max)
- Use technical terms appropriately but explain complex concepts clearly

RELEVANT CONTEXT FROM PORTFOLIO (with relevance scores):
---
${relevantContext}
---

Answer in first person as David Kwon based on the context above.`;

    // Create chat completion
    const completion = await openai.chat.completions.create({
      model: "gpt-4o-mini", // Most cost-effective: $0.15/1M input, $0.60/1M output tokens
      messages: [
        { role: "system", content: systemPrompt },
        ...messages.slice(-5), // Only include last 5 messages for context
      ],
      temperature: 0.7,
      max_tokens: 500,
    });

    return NextResponse.json({
      message: completion.choices[0]?.message?.content,
    });
  } catch (error) {
    console.error("Chat API error:", error);
    return NextResponse.json(
      { error: "Failed to process chat request" },
      { status: 500 }
    );
  }
}

Key Features:

Frontend Implementation

Chat Interface Components

The frontend provides a responsive chat interface with suggested questions and real-time updates:

// components/home/chatbot-section.tsx
function useChatbot() {
  const [messages, setMessages] = useState<Message[]>([INITIAL_MESSAGE]);
  const [input, setInput] = useState("");
  const [isLoading, setIsLoading] = useState(false);
  const [suggestedQuestions, setSuggestedQuestions] = useState<
    SuggestedQuestion[]
  >([]);

  const handleSubmit = async (e: React.FormEvent) => {
    e.preventDefault();
    if (!input.trim() || isLoading) return;

    const userMessage: Message = {
      role: "user",
      content: input.trim(),
      timestamp: new Date(),
    };

    setMessages((prev) => [...prev, userMessage]);
    setInput("");
    setIsLoading(true);

    try {
      const response = await fetch("/api/chat", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          messages: [...messages, userMessage].map((m) => ({
            role: m.role,
            content: m.content,
          })),
        }),
      });

      const data = await response.json();
      const assistantMessage: Message = {
        role: "assistant",
        content: data.message,
        timestamp: new Date(),
      };

      setMessages((prev) => [...prev, assistantMessage]);
    } catch (error) {
      console.error("Chat error:", error);
      // Handle error gracefully
    } finally {
      setIsLoading(false);
    }
  };

  return {
    messages,
    input,
    setInput,
    isLoading,
    suggestedQuestions,
    handleSubmit,
  };
}

Suggested Questions System

Dynamic question generation based on conversation context:

// lib/utils/suggested-questions.ts
export function generateSuggestedQuestions(
  messages: Message[]
): SuggestedQuestion[] {
  const lastMessage = messages[messages.length - 1];
  const hasExperienceQuestion = messages.some(
    (m) =>
      m.content.toLowerCase().includes("experience") ||
      m.content.toLowerCase().includes("work")
  );

  if (messages.length === 1) {
    // Initial suggestions
    return [
      { text: "What's your experience with AWS?", category: "technical" },
      { text: "Tell me about your recent projects", category: "projects" },
      { text: "What technologies do you work with?", category: "technical" },
      { text: "How did you build this portfolio?", category: "meta" },
    ];
  }

  if (hasExperienceQuestion) {
    return [
      { text: "What was your role at [Company]?", category: "experience" },
      { text: "What technologies did you use there?", category: "technical" },
      { text: "What were your biggest achievements?", category: "experience" },
    ];
  }

  // Context-aware suggestions based on conversation
  return [
    { text: "Can you elaborate on that?", category: "follow-up" },
    { text: "What challenges did you face?", category: "experience" },
    { text: "How did you solve that problem?", category: "technical" },
  ];
}

Performance & Cost Analysis

Cost Breakdown

Model Selection Analysis:

Real-world Usage:

Performance Metrics

Response Times:

Scalability:

Optimization Strategies

Production Deployment

Environment Setup

# .env.local
OPENAI_API_KEY=sk-proj-your-actual-api-key-here

Vercel Deployment

// vercel.json
{
  "functions": {
    "app/api/chat/route.ts": {
      "maxDuration": 30
    }
  },
  "env": {
    "OPENAI_API_KEY": "@openai-api-key"
  }
}

Security Considerations

1. API Key Protection: Server-side only, never exposed to client
2. Rate Limiting: Consider implementing for production use
3. Input Validation: Sanitize user inputs before processing
4. Error Handling: Graceful fallbacks for API failures

Monitoring & Debugging

Console Logging

The system provides detailed logging for debugging:

// Example console output
📚 Document index built: {
  totalChunks: 45,
  totalWords: 125000,
  sources: { blogs: 35, resume: 10, projects: 0 }
}

🔍 RAG Retrieval: {
  query: "What AWS experience do you have?...",
  chunksRetrieved: 5,
  avgScore: 8.2,
  sources: ["AWS Lambda Experience", "Serverless ETL", "Cloud Architecture"]
}

📊 RAG Retrieval Details: {
  query: "What AWS experience do you have?...",
  chunksRetrieved: 5,
  avgScore: 8.2,
  sources: ["AWS Lambda Experience", "Serverless ETL", "Cloud Architecture"]
}

Visual Inspection

The /rag-index page provides a visual interface to inspect indexed content:

• View all document chunks
• Search and filter functionality
• Relevance scores and metadata
• Source attribution

Future Enhancements

1. Semantic Search with Embeddings

// Future: Vector-based similarity search
const embedding = await openai.embeddings.create({
  model: "text-embedding-ada-002",
  input: query,
});

// Calculate cosine similarity with chunk embeddings
const similarity = cosineSimilarity(queryEmbedding, chunkEmbedding);

2. Hybrid Search Strategy

// Combine keyword and semantic scoring
const keywordScore = calculateRelevanceScore(query, chunk);
const semanticScore = calculateSemanticSimilarity(query, chunk);
const finalScore = keywordScore * 0.7 + semanticScore * 0.3;

3. Streaming Responses

// Real-time response streaming
const stream = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [...],
  stream: true
});

for await (const chunk of stream) {
  // Stream response chunks to client
}

4. User Feedback Loop

// Collect user feedback to improve scoring
interface UserFeedback {
  query: string;
  response: string;
  rating: 1 | 2 | 3 | 4 | 5;
  chunksUsed: string[];
}

// Use feedback to tune relevance scoring weights

Lessons Learned

What Worked Well

1. Custom relevance scoring - More effective than simple keyword matching
2. Contextual boosting - Query intent recognition improves accuracy
3. Cost optimization - GPT-4o-mini provides excellent quality at low cost
4. Caching strategy - 5-minute cache balances performance and freshness
5. Detailed logging - Essential for debugging and optimization

What Could Be Improved

1. Semantic search - Embeddings would improve relevance for complex queries
2. Response streaming - Better UX for longer responses
3. User feedback - Need data to tune scoring algorithms
4. Content updates - Manual cache invalidation could be automated
5. Rate limiting - Production deployment needs usage controls

Key Insights

1. RAG beats fine-tuning - More flexible, cheaper, easier to update
2. Relevance scoring is critical - Quality of retrieval determines response quality
3. Context injection works - AI responds accurately when given proper context
4. Cost optimization matters - Model selection has massive cost implications
5. Debugging is essential - Detailed logging enables continuous improvement

Conclusion

Building a production RAG chatbot taught me that the magic isn't in the AI model—it's in the retrieval system. By focusing on intelligent document chunking, sophisticated relevance scoring, and context injection, we created a chatbot that:

• Costs less than $0.001 per conversation
• Responds accurately using only verified content
• Scales effortlessly with content updates
• Provides detailed debugging visibility

The RAG approach proved superior to fine-tuning because it's more flexible, cost-effective, and maintainable. The key was building a robust retrieval system that finds the right content and injects it into the AI prompt with proper context.

Next steps: Implement semantic search with embeddings, add user feedback loops, and explore streaming responses for even better user experience.

The code is open-source and available in my portfolio repository. Feel free to adapt it for your own use case—the patterns work for any domain where you need an AI assistant with access to specific knowledge.

Resources

• Repository: GitHub - Kwonfolio
• Live Demo: Portfolio Chatbot
• RAG Index: View Indexed Content
• OpenAI Documentation: Chat Completions API
• Next.js API Routes: API Routes Guide

This article is part of my technical blog series. Check out my other posts on DynamoDB patterns, workflow editors, and system architecture.