EstateWise: AI‑Powered Real Estate Assistant for Chapel Hill, NC - AI/ML/Infrastructure Technical Documentation

EstateWise is an AI‑powered real estate assistant focused on Chapel Hill, NC and surrounding areas. This document provides a deep dive into every component—from raw data ingestion to AI orchestration, delivery via a React/Next.js front‑end, and end‑to‑end deployment.

Below, we outline the architecture, key components, and challenges faced during development. This is intended for developers and data scientists interested in understanding the inner workings of EstateWise. We also provide a flowchart and various diagrams to visualize the architecture and data flow throughout the system.

Important

For a CLI version of the chatbot, as well as the EDA (Exploratory Data Analysis) of the properties data and interactive geospatial visualizations, check out the Jupyter notebooks in the root directory: EDA-CLI-Chatbot.ipynb. Alternatively, feel free to go to this Colaboratory Notebook to directly view and run the code in this notebook & see the results in real time.

0. Production Readiness Updates (2026-04)
1. Data Ingestion & Validation
2. Embedding Generation & Vector Store
3. Retrieval‑Augmented Generation (RAG)
4. RAG‑Decision Orchestration Layer
5. Clustering & Similarity Analysis
6. Mixture‑of‑Experts (MoE) Ensemble
7. Chain-of-Thought Reasoning
8. Backend API & Data Layer
9. Frontend Integration
10. Infrastructure & Deployment
11. Challenges
12. Logging & Monitoring
13. GitHub Actions CI/CD Pipeline
14. Context Engineering System
15. Appendices

0. Production Readiness Updates (2026-04)

Recent implementation hardening completed across backend, MCP, and agentic runtime:

Backend typecheck gate fixed: forum post sorting now uses strict literal sort-order typing, and backend build enforces tsc (no silent pass-through).
tRPC example compile stability: backend now includes @trpc/client so src/trpc/client.example.ts remains type-safe and build-clean.
Live data reliability: MCP live snapshot search accepts both normalized (listings) and raw (records) snapshot shapes.
Runtime contract enforcement: LangGraph MCP wrappers include live.zillow.search, with automated tests preventing future required-tool drift.
Policy resilience: malformed ranking policy JSON no longer crashes orchestration; runtime falls back to default policy with warning.

flowchart LR
  BuildGate[backend tsc hard gate] --> SaferRelease[Safer production releases]
  ContractGuard[LangGraph/MCP contract test] --> ToolConsistency[Tool contract consistency]
  PolicyFallback[Policy parse fallback] --> RuntimeContinuity[Runtime continuity]
  SnapshotCompat[records + listings support] --> FreshDataReliability[Fresh data retrieval reliability]

1. Data Ingestion & Validation

What it is:
The pipeline stage that reads raw Zillow JSON files, selects only the relevant fields, cleans and sanitizes each record, and enforces data quality rules.

Why we use it:
To ensure we only store accurate, complete, and well‑formed property data—avoiding garbage in, garbage out—so that downstream embeddings, similarity searches, and AI reasoning work reliably.

1.1 Streaming Parser Setup

We avoid loading multi‑GB JSON files into memory by using stream-json and its array streamer:

import { parser } from "stream-json";
import { streamArray } from "stream-json/streamers/StreamArray";
import { createReadStream } from "fs";

function processFileStreaming(filePath: string) {
  const jsonStream = createReadStream(filePath, { encoding: "utf8" })
    .pipe(parser())
    .pipe(streamArray());

  jsonStream.on("data", async ({ value }) => {
    jsonStream.pause();
    try {
      await handleRecord(value);
    } finally {
      jsonStream.resume();
    }
  });

  jsonStream.on("end", () => {
    console.log(`Finished processing ${filePath}`);
  });
}

Back‑pressure: We call pause() and resume() around each async record to keep the event loop responsive.
Streaming: Each record is handled in isolation.

1.2 Schema & Field Selection

We explicitly keep only the fields necessary for chat and search on Pinecone:

Category	Kept Fields
Identifiers & Location	`zpid`, `address.streetAddress`, `city`, `state`, `zipcode`
Physical Attributes	`bedrooms`, `bathrooms`, `livingArea`, `yearBuilt`, `homeType`
Price & Status	`price`, `homeStatus`, `listingDataSource`
Geocoordinates	`latitude`, `longitude`
Description	`description` (free‑text)

Dropped: incomplete records missing critical fields, unrealistic yearBuilt (<1800 or >2100), non‑numeric/invalid ranges, and dozens of extraneous Zillow-only flags.

1.3 Cleaning & Sanitization Utilities

function safeStr(val: any, fallback = "Unknown"): string {
  /* … */
}
function safeNum(val: any, fallback = 0, min?: number, max?: number): number {
  /* … */
}

interface CleanedProperty {
  zpid: number;
  address: {
    streetAddress: string;
    city: string;
    state: string;
    zipcode: string;
  };
  bedrooms: number;
  bathrooms: number;
  livingArea: number;
  yearBuilt: number;
  latitude: number;
  longitude: number;
  homeType: string;
  price: number;
  homeStatus: string;
  listingDataSource: string;
  description: string;
}

function cleanDocument(raw: any): CleanedProperty {
  // apply safeStr, safeNum, range checks, defaults…
}

safeStr: trims, rejects empty, returns "Unknown"
safeNum: enforces numeric ranges (e.g. 0 ≤ bedrooms ≤ 20, 0 < price ≤ 10 000 000)
Addresses are flattened into a single JSON field for metadata.

1.4 Error Handling & Logging

Every step logs warnings and errors:

try {
  const clean = cleanDocument(raw);
  if (shouldSkip(clean)) {
    console.warn(`Skipping invalid: zpid=${clean.zpid}`);
    return;
  }
  // …
} catch (e) {
  console.error("Record processing error:", e, raw);
}

Malformed JSON: Caught at stream level
Validation Failures: Logged and skipped
Upsert Errors: Caught & retried in next batch

2. Embedding Generation & Vector Store

What it is:
Transforms each cleaned property record into a fixed‑length numeric vector (via Google’s text‑embedding model) and upserts those vectors into a Pinecone index in batches.

Why we use it:
Embeddings capture semantic meaning of property descriptions and metadata; storing them in a vector database enables lightning‑fast kNN lookups for retrieval‑augmented generation (RAG).

2.1 Prompt Engineering for Embeddings

Each record is transformed into a concise single‑sentence prompt:

function buildPrompt(prop: CleanedProperty): string {
  return (
    `Property at ${prop.address.streetAddress}, ${prop.address.city}, ${prop.address.state} ${prop.address.zipcode}. ` +
    `Price: $${prop.price}. Beds: ${prop.bedrooms}, Baths: ${prop.bathrooms}, ` +
    `Area: ${prop.livingArea} sqft, Built in ${prop.yearBuilt}. ${prop.description}`
  );
}

Uniform structure ensures embedding consistency.

2.2 Google Embedding API Integration

import { GoogleGenerativeAI } from "@google/generative-ai";

const genAI = new GoogleGenerativeAI(process.env.GOOGLE_AI_API_KEY);
const embedModel = genAI.getGenerativeModel({
  model: "models/gemini-embedding-001",
});

async function generateEmbedding(text: string): Promise<number[]> {
  const resp = await embedModel.embedContent(text);
  if (!resp.embedding?.values) throw new Error("Invalid embedding");
  return resp.embedding.values;
}

1,536‑dimensional vectors
Retry on 5xx error codes with exponential backoff

2.3 Pinecone Batch Upsert Strategy

import { index } from "../pineconeClient";
const BATCH_SIZE = 50;
let batch: PineconeVector[] = [];

async function maybeUpsert() {
  if (batch.length < BATCH_SIZE) return;
  const toSend = batch.splice(0, BATCH_SIZE);
  await index.upsert({ vectors: toSend });
  console.log(`Upserted ${toSend.length} vectors`);
}

Atomic batches of 50 vectors to avoid Pinecone’s payload limit
Final flush on stream end

2.4 Monitoring & Retry Logic

Metrics: track embedding latency, upsert success/failure rates (via Prometheus + Grafana)
Retries: 3× with jitter on network/API errors

3. Retrieval‑Augmented Generation (RAG)

What it is:
The process of taking a user’s query, converting it into an embedding, fetching the top‑K most similar property vectors from Pinecone, and assembling their details into context for the LLM.

Why we use it:
RAG grounds the AI’s responses in real data—ensuring recommendations are factually based on actual listings rather than hallucinations.

3.1 KNN Query Implementation

async function queryProperties(
  query: string,
  topK = 10,
): Promise<RawQueryResult[]> {
  const qEmb = await generateEmbedding(query);
  const resp = await index.query({
    vector: qEmb,
    topK,
    includeMetadata: true,
  });
  return resp.matches.map((m) => ({
    id: m.id,
    score: m.score,
    metadata: m.metadata,
  }));
}

Pinecone returns top‑K similar vectors by cosine similarity and kNN.

3.2 Assembling Retrieval Context

function propertiesToContext(results: RawQueryResult[]): string {
  return results
    .map(
      (r) =>
        `• ${r.metadata.address} — $${r.metadata.price}, ${r.metadata.bedrooms}bd/${r.metadata.bathrooms}ba`,
    )
    .join("\n");
}

This string is inserted into the LLM system prompt.

3.3 Caching Frequent Queries

In‑memory LRU cache (node-lru-cache) for embeddings & responses
TTL of 30 minutes to balance freshness

3.4 Hybrid RAG (Vector + Graph)

What it is:
A retrieval pipeline that combines semantic vector search (Pinecone) with structural graph enrichment (Neo4j), then merges the results into a single context for the LLM.

Why we use it:

Vector search captures semantic intent from free-form user prompts.
Graph traversal adds explainable relationships (same neighborhood, same ZIP, similar-to edges).
The merged context improves recall and makes recommendations easier to justify.

High-level flow:

Embed the user query and fetch Top-K candidates from Pinecone.
Enrich top results with Neo4j neighbors and relationship reasons.
Merge, dedupe, and cluster results before building the final prompt context.

flowchart LR
  Q[User Query] --> E[Embed Query]
  E --> V[Pinecone Vector Search]
  V --> K[Top-K Results]
  K --> G[Neo4j Graph Enrichment]
  G --> M[Merge + Dedupe]
  M --> C[Context for LLM]

For full diagrams, sample queries, and evaluation notes, see RAG_SYSTEM.md.

4. RAG‑Decision Orchestration Layer

What it is:
A lightweight “decision” layer that runs a single LLM call to determine whether property data (RAG from Pinecone) is needed. It emits a simple JSON flag, and—based on that—either fetches the data or skips straight to the Mixture‑of‑Experts pipeline.

Why we use it:

To minimize latency and complexity. Instead of full agentic tool‑calling loops, we do one quick check: “Do I need external property data to answer this?”
If we can bypass RAG for simpler queries (e.g. greetings or basic questions), the chatbot saves significant time and compute by skipping the database lookup entirely.
When RAG is needed, it still allows us to pull in rich context on demand.

4.1 JSON‑Only Decision Protocol

The model must respond only with:

{ "usePropertyData": true }

{ "usePropertyData": false }

No extra text, just the exact JSON object in a single code block.

4.2 Simplified Control Flow

User → Send prompt to the decision agent
Agent → Emits JSON {"usePropertyData": …}
Backend →
- If true, fetch queryPropertiesAsString + queryProperties from Pinecone
- If false, skip data retrieval (fast path)
Backend → Append fetched context (if any) to userContext
Backend → Invoke the full Mixture‑of‑Experts pipeline with or without RAG context
Experts → Produce specialized outputs → Merger → Final answer

Note: Skipping RAG for simple requests (greetings, clarifications, etc.) lets the bot respond almost instantly, saving both time and resources.

4.3 Implementation Highlights

Single LLM call: No repeated tool loops needed—just one decision step.
Deterministic output: Use temperature = 0, topP = 1, topK = 1 to force strict JSON responses.
Context injection: When usePropertyData is true, we prepend the text summary of property listings into the system/user context for the experts.
Fast‑path fallback: If JSON parsing fails or the model misbehaves, default to usePropertyData: false to avoid blocking the pipeline.
Efficiency gains: By bypassing RAG for queries that don’t need it, we cut out database latency and reduce overall compute, making the chatbot far more responsive.

5. Clustering & Similarity Analysis

What it is:
Extracts numeric features (price, beds, baths, area, year built), normalizes them, and runs k‑Means to group similar properties into clusters. Those cluster labels augment the AI context.

Why we use it:
Clustering surfaces market segments (e.g. luxury vs starter homes) and helps the AI explain patterns or suggest diverse options, enhancing recommendation depth.

5.1 Numeric Feature Extraction & Normalization

function extractFeatures(results: RawQueryResult[]): number[][] {
  return results.map((r) => [
    r.metadata.price,
    r.metadata.bedrooms,
    r.metadata.bathrooms,
    r.metadata.livingArea,
    r.metadata.yearBuilt,
  ]);
}

function normalize(vectors: number[][]): number[][] {
  /* min-max normalize each dimension */
}

5.2 Plain‑JS K‑Means Implementation

function kmeans(data: number[][], k: number, maxIter = 50): number[] {
  const n = data.length,
    dims = data[0].length;
  let centroids = initializeCentroids(data, k);
  let assignments = new Array(n).fill(0);
  for (let iter = 0; iter < maxIter; iter++) {
    let changed = assignClusters(data, centroids, assignments);
    if (!changed) break;
    updateCentroids(data, assignments, centroids, k, dims);
  }
  return assignments;
}

5.3 Cluster Context Injection

We append cluster labels to the RAG context:

Cluster assignments:
- ID 12345: cluster 2
- ID 67890: cluster 0

6. Mixture‑of‑Experts (MoE) Ensemble

What it is:
Five specialized “expert” models (Data Analyst, Lifestyle Concierge, Financial Advisor, Neighborhood Expert, Cluster Analyst) each generate their own perspective in parallel; a master model then merges them into one cohesive answer.

Why we use it:
To combine multiple domain‑focused viewpoints—statistics, lifestyle, finance, community, cluster insights—into a richer, more trustworthy recommendation than a single monolithic AI call.

6.1 Expert Definitions & System Prompts

const experts = [
  {
    name: "Data Analyst",
    instructions: "Extract distributions, avg/median prices…",
  },
  {
    name: "Lifestyle Concierge",
    instructions: "Focus on schools, parks, commute…",
  },
  // Financial Advisor, Neighborhood Expert, Cluster Analyst are added similarly
];

6.2 Parallel Expert Invocations

const expertResults = await Promise.all(
  experts.map(async (e) => {
    const model = genAI.getGenerativeModel({
      systemInstruction: base + e.instructions,
    });
    const chat = model.startChat({ history, generationConfig, safetySettings });
    const res = await chat.sendMessage(userMessage);
    return { name: e.name, text: res.response.text() };
  }),
);

6.3 Weight Normalization & Feedback Loop

Initial weights default to 1 each → normalized to sum = 1
On thumbs‑down, multiply selected experts by 0.8, others by 1.2, renormalize → persist
We cap weights (no expert drops below 0.1) to maintain diversity

6.4 Master Merger Model

const mergerInstruction = `
You are the Master Merger. Combine these expert views (with weights):
${expertResults
  .map(
    (r) => `**${r.name}** (${weights[r.name]}):
${r.text}`,
  )
  .join("\n\n")}
Now synthesize a single concise recommendation…
`;

6.5 Prompt Engineering

All expert prompts are carefully crafted to elicit the most relevant information:

Data Analyst: “Analyze the price distribution, average/median prices, and any outliers.”
Lifestyle Concierge: “Focus on schools, parks, commute times, and local amenities.”
Financial Advisor: “Evaluate the investment potential, ROI, and financing options.”
Neighborhood Expert: “Provide insights on the neighborhood’s safety, culture, and community.”
Cluster Analyst: “Analyze the cluster of similar properties and their market trends.”

The master merger model also receives a prompt that combines all expert outputs and their respective weights:

Master Merger: “You are the Master Merger. Combine these expert views (with weights): [expert outputs]. Now synthesize a single concise recommendation…”

Also, all models, including the master model, share a common system prompt, carefully crafted so that they work together seamlessly:

System Prompt: “You are an AI model that specializes in real estate analysis. Your task is to provide insights based on the data and context provided.”

Additionally, the AI Agent also receives a carefully-crafted system prompt to ensure it understands its role in the process:

Agentic AI: “You are an AI agent that orchestrates the entire process. You will decide whether to use property data, invoke experts, and merge their responses into a final recommendation.”

Takeaways: All prompts, especially system prompts, are designed to be clear, concise, and focused on the specific task at hand. This ensures that the AI can generate the most relevant and accurate responses. They are ultra-specific and tailored to the task at hand, ensuring that the AI can generate the most relevant and accurate responses.

6.6 Agentic AI Runtime

EstateWise includes a production agentic-ai/ package that extends the core MoE pipeline with runtime-selectable, tool-centric workflows.

Runtime modes:

default: round-based orchestrator with specialist agents and blackboard state.
langgraph: ReAct tool-calling runtime over MCP with optional Pinecone and Neo4j augmentation.
crewai: Python CrewAI runtime bridged from Node for crew-style execution.

Entry surfaces:

CLI: npm run dev -- "<goal>" (--langgraph, --crewai, or AGENT_RUNTIME).
HTTP: POST /run, GET /run/stream, GET /config.
A2A: POST /a2a (JSON-RPC, optional task requestId), GET /a2a/tasks/{taskId}/events (SSE).

flowchart LR
  Input[CLI / HTTP / A2A] --> Selector{runtime}
  Selector --> O[default orchestrator]
  Selector --> L[langgraph react]
  Selector --> C[crewai]
  O --> MCP[MCP Tools]
  L --> MCP
  L --> Pinecone
  L --> Neo4j
  C --> MCP

Observability updates:

LangGraph run output includes normalized toolExecutions (status, duration, output/error).
Cost telemetry is returned for LangGraph and CrewAI runs.
Optional LangSmith tracing is supported via environment configuration with strict-mode validation.

7. Chain-of-Thought Reasoning

What it is: A mechanism that allows the AI to break down complex tasks into smaller, manageable steps, providing a more structured and logical approach to problem-solving.

Why we use it: To enhance the AI's reasoning capabilities, allowing it to tackle intricate queries and provide more coherent and comprehensive responses.

7.1 Implementation

const chainOfThoughtPrompt = `<chain-of-thought-instructions>`; // such as: "Think step-by-step and provide a detailed breakdown of your reasoning process before you attempt to answer the question."
const fullSystemPrompt = `${baseSystemPrompt} ${chainOfThoughtPrompt}`;
const model = genAI.getGenerativeModel({
  systemInstruction: fullSystemPrompt,
});

7.2 Example

User Query: "Can you analyze the price trends of properties in Chapel Hill over the last year?"

AI Response:

Data Collection: Gather property data from the last year.
Data Analysis: Analyze the price trends, including average, median, and outliers.
Conclusion: Summarize the findings and provide insights on the overall market trends.
Recommendation: Suggest potential investment opportunities based on the analysis.

Note: Each expert also receives a chain-of-thought prompt to ensure they follow the same structured approach. The master merger model also receives a chain-of-thought prompt to ensure it can effectively combine the expert outputs into a coherent final recommendation.

7.3 Benefits

Improved Clarity: By breaking down complex tasks, the AI can provide clearer and more structured responses.
Enhanced Reasoning: The AI can tackle intricate queries more effectively, leading to better insights and recommendations.
User Engagement: Users can follow the AI's thought process, making the interaction more transparent and engaging.
Consistency: All experts and the master model follow the same structured approach, ensuring coherent and comprehensive responses.
Flexibility: The chain-of-thought reasoning can be adapted to various tasks, making it a versatile tool in the AI's toolkit.

7.4 Agentic AI Orchestration

What it is:
A runtime router that executes the same business intent through one of three modes (default, langgraph, crewai) and standardizes output envelopes across CLI/HTTP/A2A surfaces.

Current orchestration contract:

HTTP batch: POST /run with goal, optional runtime, rounds, threadId, requestId, and deterministic.
HTTP stream: GET /run/stream with the same runtime controls plus SSE progress events and optional deterministic=true|false.
Runtime config discovery: GET /config reports supported runtimes, MCP tool requirements, and LangSmith tracing state.
Request correlation: requestId (or x-request-id) is carried into LangGraph trace metadata.

flowchart LR
  Goal[User Goal] --> Runtime[Runtime Selector]
  Runtime --> Default[Orchestrator]
  Runtime --> LangGraph[LangGraph ReAct]
  Runtime --> CrewAI[CrewAI Runner]
  Default --> Tools[MCP Tool Layer]
  LangGraph --> Tools
  CrewAI --> Tools
  Tools --> Reporter[Unified Result Envelope]

7.5 MCP Server (Model Context Protocol)

What it is:
A stdio-based MCP server that exposes tools (search, graph, analytics, monitoring) to agentic runtimes and other MCP-compatible clients.

Why we use it:

Standardizes tool access for agents across runtimes.
Centralizes logging, caching, and observability for tool calls.
Enables a clean separation between tools and orchestration logic.

flowchart TB
  Client[MCP Client] --> Server[MCP Server]
  Server --> Tools[Tool Registry]
  Tools --> API[Backend APIs]
  Tools --> Graph[Neo4j Graph]
  Tools --> Vector[Pinecone]

Docs & entry points:

MCP server lives in mcp/
Full tool catalog and setup: mcp/README.md

7.6 Agent-to-Agent Communication

What it is: A JSON-RPC task protocol exposed by agentic-ai for asynchronous agent-to-agent execution.

Why we use it:

Standardizes inter-agent task lifecycle (queued, running, succeeded, failed, canceled).
Supports async execution with polling (tasks.get, tasks.wait) or live streams (/a2a/tasks/{taskId}/events).
Enables runtime selection per task without changing caller integration.

sequenceDiagram
  participant Client as A2A Client
  participant API as Agentic /a2a
  participant Runtime as Selected Runtime
  Client->>API: tasks.create(goal, runtime, rounds)
  API->>Runtime: execute task
  Runtime-->>API: task updates + output/error
  Client->>API: tasks.wait(taskId) or tasks.get(taskId)
  API-->>Client: terminal task state + result
  Client->>API: GET /a2a/tasks/{taskId}/events (optional SSE)

Note: The chain-of-thought reasoning and agentic AI orchestration are designed to work together seamlessly, allowing for complex workflows that can adapt to a wide range of user queries and scenarios.

8. Backend API & Data Layer

What it is:
An Express.js + TypeScript server with REST endpoints for chat, rating, conversation CRUD, authentication (JWT), and integrated Swagger/OpenAPI docs; MongoDB persists users, conversations, and expert weights.

Why we use it:
Provides a secure, scalable interface for the frontend to interact with the AI pipeline, store history, and capture feedback for continuous improvement.

8.1 Express.js Routes & Controllers

// routes/chat.ts
router.post("/chat", chatController);
router.post("/chat/rate", ratingController);
router.get("/live-data/status", liveDataStatusController);
router.get("/live-data/search", liveDataSearchController);
// More endpoints follow similarly…

Production guardrails added in this layer:

npm run check:no-agentic prevents backend imports/coupling to agentic-ai internals.
npm run build now enforces strict TypeScript compilation (tsc) as a hard quality gate.
Live-data endpoints consume local snapshot artifacts and do not trigger inline web scraping during chat requests.

8.2 MongoDB Models & Conversations

import mongoose from "mongoose";

const ConversationSchema = new mongoose.Schema(
  {
    userId: String,
    title: String,
    messages: [{ role: String, text: String, expertViews: Object }],
    expertWeights: Object,
  },
  { timestamps: true },
);

export const Conversation = mongoose.model("Conversation", ConversationSchema);

8.3 Authentication & JWT Workflow

Two sign-in paths converge on the same JWT contract:

Password flow — POST /api/auth/signup and POST /api/auth/login validate credentials with bcryptjs and issue a JWT signed with JWT_SECRET.
Passkey (WebAuthn) flow — POST /api/auth/webauthn/{register,login}/{options,verify} use @simplewebauthn/server for ceremony orchestration. Successful assertions issue the same JWT shape, so downstream code is path-agnostic.
Authorization — authMiddleware decodes Authorization: Bearer <token> (or the token cookie); requireAuth hard-fails routes that must be authenticated.
Storage
- Passkey credentials are embedded on the User document as a credentials[] subdoc array (credentialID, publicKey, counter, transports, deviceType, backedUp, nickname).
- In-flight WebAuthn challenges live in a separate webauthnchallenges collection with a TTL index (5 min) and are deleted on consumption — never stored on the user.
RP configuration — WEBAUTHN_RP_ID (frontend host), WEBAUTHN_RP_NAME, WEBAUTHN_ORIGINS (comma-separated allowed browser origins).

8.4 OpenAPI / Swagger Integration

openapi: 3.0.1
paths:
  /api/chat:
    post:
      summary: Send chat message
      requestBody: { /* … */ }
      responses:
        "200":
          {
            content:
              {
                "application/json":
                  { schema: { $ref: "#/components/schemas/ChatResponse" } },
              },
          }

8.5 tRPC API

What it is:
A TypeScript-first RPC layer that shares types end-to-end between the backend and frontend without code generation.

Why we use it:

Keeps API contracts and client calls fully type-safe.
Pairs well with React Query for caching and request batching.
Runs alongside REST at /trpc/*, so it can be adopted incrementally.

Implementation notes:

Backend router lives in backend/src/trpc/ with domain-specific routers.
Input validation uses Zod schemas.
Frontend clients consume typed procedures directly.

flowchart TB
  UI[Next.js Client] --> TC[tRPC Client]
  TC --> AR[App Router]
  AR --> PR[Properties Router]
  AR --> AN[Analytics Router]
  PR --> DB[(MongoDB/Pinecone)]
  AN --> AI[Gemini/Neo4j]

Example tRPC calls:

import { createTRPCProxyClient, httpBatchLink } from "@trpc/client";
import type { AppRouter } from "../../backend/src/trpc/routers";

// Adjust the AppRouter import to your frontend setup.
const client = createTRPCProxyClient<AppRouter>({
  links: [httpBatchLink({ url: "http://localhost:5000/trpc" })],
});

const listings = await client.properties.list.query({
  limit: 5,
  bedrooms: 3,
});

const trends = await client.analytics.marketTrends.query({
  location: "Chapel Hill, NC",
  period: "month",
});

For detailed router layouts, middleware, and usage patterns, see GRPC_TRPC.md.

8.6 gRPC Services

What it is:
A high-performance RPC layer using Protocol Buffers and HTTP/2 for cross-language and streaming workloads.

Why we use it:

Efficient binary transport and streaming support.
Strongly-typed contracts for services outside the TypeScript stack.
Ideal for microservice and analytics pipelines.

Implementation notes:

Service definitions live in grpc/proto/ (for example, MarketPulseService).
Server implementation lives in grpc/src/ and defaults to 0.0.0.0:50051.
gRPC configuration is controlled via GRPC_HOST and GRPC_PORT.

sequenceDiagram
  participant C as gRPC Client
  participant S as gRPC Server
  participant MP as MarketPulseService
  participant DS as Data Sources
  C->>S: GetSnapshot(query)
  S->>MP: Handle request
  MP->>DS: Load metrics + trends
  DS-->>MP: Snapshot data
  MP-->>C: MarketSnapshotResponse

Example gRPC calls:

# List available markets
grpcurl -plaintext localhost:50051 \
  estatewise.marketpulse.MarketPulseService.ListMarkets

# Request a market snapshot
grpcurl -plaintext -d '{"query":"Chapel Hill, NC"}' localhost:50051 \
  estatewise.marketpulse.MarketPulseService.GetSnapshot

For service diagrams, protocols, and performance guidance, see GRPC_TRPC.md.

9. Frontend Integration

What it is:
A Next.js + React + TailwindCSS app using Shadcn UI components, Framer Motion animations, React‑Markdown custom components, localStorage for guest mode, and more.

Why we use it:
Delivers a smooth, responsive chat experience—complete with inline charts, expert‑view toggles, theme switching, and conversation management—so users can focus on real estate insights.

Also, using Next.js's SSR (Server-Side Rendering) capabilities, we can pre-render pages for better performance and SEO and improve the overall user experience.

Live Frontend URL:
https://estatewise-backend.vercel.app/

10. Infrastructure & Deployment

EstateWise features production-ready, enterprise-grade infrastructure with multi-cloud deployment options, advanced deployment strategies, comprehensive monitoring, and automated operations.

[!TIP] > 📘 Complete Documentation:

DEVOPS.md – Comprehensive DevOps guide with deployment strategies, monitoring, and operations

DEPLOYMENTS.md – Platform-specific deployment guides (AWS, Azure, GCP, Kubernetes)

PRODUCTION-READINESS.md – Complete production checklist and metrics

10.1 Deployment Strategies

EstateWise supports three zero-downtime deployment strategies:

Blue-Green Deployment

# Automated script for blue-green deployments
./kubernetes/scripts/blue-green-deploy.sh backend \
  ghcr.io/your-org/estatewise-app-backend:v1.2.3

# Features:
# - Instant traffic switch (< 1 second)
# - Full environment testing before switch
# - Immediate rollback capability
# - Ideal for major releases

Canary Deployment

# Progressive rollout with traffic shifting
./kubernetes/scripts/canary-deploy.sh backend \
  ghcr.io/your-org/estatewise-app-backend:v1.2.3

# Features:
# - Gradual traffic: 10% → 25% → 50% → 75% → 100%
# - Automated health monitoring
# - Automatic rollback on failures
# - Manual approval gates
# - Ideal for new features

Rolling Update

# Kubernetes-native gradual rollout
kubectl set image deployment/estatewise-backend \
  backend=ghcr.io/your-org/estatewise-app-backend:v1.2.3

# Features:
# - Zero-downtime pod replacement
# - Resource-efficient
# - Ideal for regular updates

10.2 Production Kubernetes Manifests

EstateWise includes comprehensive Kubernetes resources for production deployments:

# High Availability
kubernetes/base/
├── backend-hpa.yaml              # Autoscaling: 2-10 replicas
├── frontend-hpa.yaml             # Autoscaling: 2-8 replicas
├── backend-pdb.yaml              # Pod disruption budgets
├── frontend-pdb.yaml             # Ensures availability during updates

# Security
├── network-policy.yaml           # Network segmentation
├── rbac.yaml                     # Role-based access control
├── resource-quota.yaml           # Resource limits

# Monitoring
kubernetes/monitoring/
├── prometheus-servicemonitor.yaml # Metrics collection
├── prometheus-rules.yaml         # 16 alert rules
└── grafana-dashboard.json        # Visualization

# Operations
kubernetes/jobs/
├── mongodb-backup-cronjob.yaml   # Daily automated backups
└── db-migration-job.yaml         # Database migrations

# Chaos Engineering
kubernetes/chaos/
├── chaos-tests.yaml              # Automated resilience tests
└── manual-chaos-tests.sh         # Manual test suite

Helm packaging (for CI/CD and multi-cloud clusters):

Chart lives in helm/estatewise/ and mirrors the base manifests with configurable values.
Includes templates for Deployments, Services, Ingress, HPA, PDB, NetworkPolicy, RBAC, and ServiceMonitor.
Cloud-specific annotations can be injected via global.cloud.* for EKS/AKS/GKE/OCI.
See helm/estatewise/README.md for install and override examples.

10.3 Multi-Cloud Deployment Options

EstateWise is cloud-agnostic and deployable to:

AWS ECS Fargate – CloudFormation + CodePipeline
Azure Container Apps – Bicep + Azure DevOps
GCP Cloud Run – Deployment Manager + Cloud Build
Oracle Cloud (OCI) – Terraform + OCIR + Compute/LB
Kubernetes – Kustomize/Helm on any cluster (EKS, AKS, GKE, self-managed)
Vercel – Frontend + optional backend edge functions

10.4 CI/CD Pipeline (Enhanced Jenkins)

// jenkins/workflow.Jenkinsfile
pipeline {
  stages {
    // Quality & Security (5 new stages)
    stage('Security Audit') { /* npm audit */ }
    stage('SAST Scan') { /* Semgrep + Trufflehog */ }
    stage('Code Coverage') { /* Test coverage */ }
    stage('Container Security') { /* Trivy + Dockle */ }
    stage('Integration Tests') { /* Full integration suite */ }

    // Build & Push
    stage('Build Images') { /* Docker build + push */ }

    // Advanced Deployments (2 new stages)
    stage('Blue-Green Deployment') {
      when { env.DEPLOY_BLUE_GREEN == '1' }
      /* Automated blue-green with health checks */
    }
    stage('Canary Deployment') {
      when { env.DEPLOY_CANARY == '1' }
      /* Progressive rollout with monitoring */
    }

    // Multi-Cloud Deploy
    stage('Multi-Cloud Deploy') {
      /* AWS/Azure/GCP/OCI/Kubernetes parallel deployment */
    }
  }
}

Security Scanning Layers:

npm audit – Dependency vulnerabilities
Semgrep – Static application security testing (SAST)
Trufflehog – Secret scanning
Trivy – Container image vulnerabilities
Dockle – Container best practices
SonarQube – Continuous code quality analysis (bugs, code smells, technical debt, coverage)
Snyk SCA – Open-source dependency vulnerability scanning
Snyk Code – Static application security testing (SAST)
Snyk Container – Docker image OS + application layer CVE scanning
Snyk IaC – Infrastructure-as-code misconfiguration detection (Terraform, K8s, Helm, Docker Compose)

SonarQube Code Quality

SonarQube is configured as a multi-module project via sonar-project.properties, scanning all 7 services:

Module	Sources	Tests
`backend`	`src/`	`tests/`
`frontend`	`pages/`, `components/`, `lib/`, `server/`	`__tests__/`
`grpc`	`src/`	`src/*/.test.ts`
`mcp`	`src/`	–
`agentic-ai`	`src/`	`src/*/.test.ts`
`deployment-control`	`src/`	–
`context-engineering`	`src/`	–

Quality gate enforcement breaks CI on new code violations. A local SonarQube server is available:

docker compose -f docker/compose.sonarqube.yml up -d   # SonarQube 10 + Postgres 16
make sonar                                               # Run analysis + wait for gate

Snyk Vulnerability Scanning

Snyk provides four scanning layers with configurable severity threshold (--severity-threshold=high):

flowchart LR
  subgraph Snyk["Snyk Scanning Layers"]
    SCA["SCA<br/>Dependencies"]
    Code["Code SAST<br/>Source Patterns"]
    Container["Container<br/>Docker Images"]
    IaC["IaC<br/>Terraform · K8s · Helm"]
  end

  PkgJSON["package.json"] --> SCA
  Source["TypeScript Source"] --> Code
  Images["Docker Images"] --> Container
  Infra["IaC Files"] --> IaC

  SCA --> Report["Security Reports"]
  Code --> Report
  Container --> Report
  IaC --> Report

Per-service Snyk policies live in .snyk (root) and .snyk.d/ (per-service overrides). Both tools are integrated into Jenkins and AWS CodeBuild pipelines.

10.5 Monitoring & Observability

Prometheus Metrics

# 16 Alert Rules across 5 categories:
- Availability: ServiceDown, HighErrorRate, CriticalErrorRate
- Performance: HighResponseTime, VeryHighResponseTime
- Resources: HighMemoryUsage, HighCPUUsage, PodCrashLooping
- Scaling: HPAMaxedOut, HPAScalingDisabled
- Deployment: ReplicasMismatch, GenerationMismatch

Grafana Dashboards

// Real-time visualization:
- Request rate by service
- Error rate by service
- Response time (P50, P95, P99)
- Pod count and health
- CPU/Memory usage
- Resource utilization trends

Centralized Logging

Kubernetes logs – kubectl logs with structured JSON
CloudWatch (AWS) / Cloud Logging (GCP) / Log Analytics (Azure)
Prometheus exporters – Custom metrics for business KPIs
Datadog Log Management – Centralized logs with automatic trace correlation via DD_LOGS_INJECTION=true

Datadog Observability

Datadog provides full-stack APM, centralized log management, production monitors, SLOs, dashboards, and synthetic health checks. The integration spans all deployment targets:

flowchart TB
  subgraph App["Application Services"]
    BE["Backend<br/>DD_SERVICE: estatewise-backend"]
    FE["Frontend<br/>DD_SERVICE: estatewise-frontend"]
    GRPC["gRPC<br/>DD_SERVICE: estatewise-grpc"]
    MCP["MCP<br/>DD_SERVICE: estatewise-mcp"]
    AI["Agentic AI<br/>DD_SERVICE: estatewise-agentic-ai"]
  end

  subgraph Agent["Datadog Agent (DaemonSet / Sidecar)"]
    APM["APM Collector<br/>TCP/8126"]
    DSD["DogStatsD<br/>UDP/8125"]
    LogCol["Log Collector"]
  end

  subgraph Cloud["Datadog Cloud"]
    Monitors["17 Monitors"]
    Dashboard["Production Dashboard"]
    SLOs["Availability + Latency SLOs"]
    Synthetics["Synthetic Checks<br/>(3 AWS regions)"]
    Traces["APM Service Map"]
    Logs["Centralized Logs"]
  end

  App -->|"traces"| APM
  App -->|"custom metrics"| DSD
  App -->|"stdout/stderr"| LogCol
  Agent -->|"HTTPS/443"| Cloud

Configuration by deployment target:

Target	Config Location	What It Manages
Terraform	`terraform/datadog.tf`	Monitors, dashboard, SLOs, synthetic checks, downtime schedules
Helm	`helm/estatewise/templates/datadog-*.yaml`	Agent DaemonSet, Cluster Agent, monitors ConfigMap, NetworkPolicies
Docker Compose	`docker/compose.prod.yml`	`datadog-agent` service with APM + logs + DogStatsD
Deployment Control	`deployment-control/src/datadog.ts`	Deploy events + DogStatsD custom metrics

Quick enable:

# Docker Compose
DD_API_KEY=your-key docker compose -f docker/compose.prod.yml --profile monitoring up -d

# Helm
helm upgrade --install estatewise ./helm/estatewise \
  --set datadog.enabled=true --set datadog.monitors.enabled=true

# Terraform
terraform apply -var='enable_datadog=true' -var='datadog_api_key=KEY' -var='datadog_app_key=KEY'

For full details, see 📘 docs/datadog-integration.md.

SRE Dashboard & SLO Monitoring

A dedicated SRE dashboard (sre-dashboard/) provides real-time visualization of:

SLO compliance — circular gauges for availability (99.9%) and P95 latency (<500ms), error budget burn-down
Burn-rate alerting — multi-window burn rate (1h, 6h, 3d) with Google SRE pattern thresholds
Deployment health — blue/green active color, canary rollout progress, recent deploy feed
DORA metrics — deployment frequency, lead time, MTTR, change failure rate

Error budget: 43.2 minutes per 30-day window. When exhausted, non-critical deployments are frozen.

Canonical SLO definitions: docs/SLO.md | Full SRE guide: SRE.md

10.6 High Availability Setup

# Horizontal Pod Autoscaling
Backend HPA:
  minReplicas: 2
  maxReplicas: 10
  CPU target: 70%
  Memory target: 80%

# Pod Disruption Budgets
minAvailable: 1  # Ensures at least 1 pod during disruptions

# Resource Management
Resource Quotas:
  CPU: 20 cores (requests), 40 cores (limits)
  Memory: 40Gi (requests), 80Gi (limits)

# Network Policies
- Backend: Allow from frontend, ingress, MongoDB, external APIs
- Frontend: Allow from ingress, to backend
- Default: Deny all (explicit allow required)

10.7 Disaster Recovery

Automated Backups

# Daily MongoDB backups
CronJob: mongodb-backup
  Schedule: "0 2 * * *"  # Daily at 2 AM
  Storage: S3 + local (7-day retention)
  Compression: gzip
  Features:
    - Automatic S3 upload
    - Backup size reporting
    - Parallel collection dumps

Recovery Procedures

# Database restore
kubectl apply -f kubernetes/jobs/db-migration-job.yaml

# Application rollback
kubectl rollout undo deployment/estatewise-backend

# Blue-Green instant switch
kubectl patch service estatewise-backend \
  -p '{"spec":{"selector":{"version":"blue"}}}'

Recovery Metrics:

RTO (Recovery Time Objective): < 4 hours
RPO (Recovery Point Objective): < 24 hours
Rollback Time: < 1 second (Blue-Green)

10.8 DevOps Metrics

EstateWise achieves industry-leading DevOps metrics:

Metric	Industry Standard	EstateWise Achievement
Deployment Frequency	Multiple per week	✅ Multiple per day (automated)
Lead Time	< 1 day	✅ < 30 minutes
MTTR	< 1 hour	✅ < 5 minutes (instant rollback)
Change Failure Rate	< 15%	✅ < 5% (automated tests + canary)
Availability	99.9% (3 nines)	✅ 99.95%+ (HA setup)

10.9 Operational Runbooks

Comprehensive operational documentation:

docs/runbooks/INCIDENT-RESPONSE.md – Incident handling procedures for common issues
docs/runbooks/PRODUCTION-OPS.md – Daily operations, deployments, scaling, security operations

10.10 Chaos Engineering

Resilience testing suite to validate production readiness:

# Automated tests with Chaos Mesh
kubectl apply -f kubernetes/chaos/chaos-tests.yaml

# Manual test suite
./kubernetes/chaos/manual-chaos-tests.sh

# Test scenarios:
1. Pod Deletion - Verify recovery
2. Resource Stress - Verify autoscaling
3. Database Failure - Verify resilience
4. Network Partition - Verify connectivity
5. Rolling Restart - Verify zero-downtime
6. High Traffic - Verify scaling

11. Challenges

Throughout the development of EstateWise, we encountered several technical and operational hurdles. Below is a breakdown of the most significant challenges and how we addressed them:

11.1 Large‑Scale Data Ingestion

Memory Constraints: Ingesting four multi‑gigabyte JSON files without overwhelming Node.js required a streaming parser and explicit back‑pressure (pause()/resume()).
Data Quality & Completeness: Many raw records had missing or malformed fields (e.g. yearBuilt outside realistic ranges, non‑numeric beds/baths). We iterated on our cleanDocument() logic, adding strict range checks and defaulting to "Unknown" where appropriate.

11.2 Embedding Generation & Rate Limits

API Throughput: Generating 30,000+ embeddings against Google’s gemini-embedding-001 model risked hitting per‑minute or per‑day quotas. We implemented exponential backoff, jittered retries, and an in‑flight counter to throttle concurrent requests.
Latency Variability: Embedding calls occasionally spiked to hundreds of milliseconds each; batching and parallelizing up to a safe concurrency limit (5–10 simultaneous requests) was critical.

11.3 Vector Store Scaling

Pinecone Payload Limits: Pinecone imposes size caps on each upsert call. We experimented to find a reliable batch size (50 vectors) that balanced throughput and success rate.
Index Cold Starts: Query latency spiked when the index was idle. To mitigate, we schedule a small “ping” query every 5 minutes to keep the index warm.

11.4 RAG & Context Assembly

Context Window Management: Feeding too many retrieved records into Gemini could exceed its context limits. We settled on a top‑50 retrieval, then truncated or summarized lower‑relevance entries to stay within token budgets.
Cache Invalidation: Simple in‑memory caches risked serving stale data after new upserts. We scoped our caches with short TTLs (30 minutes) and clear entries on data‑loading events.

11.5 Mixture‑of‑Experts Complexity

Latency Accumulation: Invoking six separate Gemini calls (5 experts + merger) often pushed total response time close to Vercel’s 60 second limit. We fine‑tuned temperatures, maxOutputTokens, and reduced expert instructions’ verbosity to shave off precious seconds.
Weight Convergence: Naïve feedback adjustments could drive one expert’s weight to near zero, starving the system of diversity. We capped weight updates (no expert drops below 0.1) and renormalize to maintain a healthy ensemble.
Also, implementing a MoE pipeline also means we need 2 separate calls to the LLM: 5 parallel calls to the experts and 1 call to the master model. This means that we need to ensure that the LLM can handle multiple concurrent requests without crashing or slowing down.

11.6 Deployment & Infrastructure

Vercel Timeouts: The free‑tier 60 second function limit forced us to split some heavy operations (e.g. clustering) into background tasks or pre‑compute nightly.
- However, many heavy queries can still cause timeouts. This is unavoidable, and we have to ensure that the user is informed about the timeout and that they should retry the query.
Secret Management: Ensuring that Google AI keys, Pinecone credentials, and JWT secrets remained secure yet available to both frontend (only public‑safe tokens) and backend required careful .env scoping and GitHub Actions secrets configuration.
Payload Size Limits: Vercel enforces a 1024 * 100 KB payload limit on API responses. We had to ensure that our responses were concise and efficient, especially when dealing with large datasets or multiple expert outputs.
- This can be problematic for guest users since we have to transmit the entire conversation history to the backend. We had to implement a mechanism to limit the size of the conversation history sent to guest users, ensuring that it fits within the payload size limits while still providing meaningful context.

12. Logging & Monitoring

What it is: A comprehensive logging and monitoring system that tracks the performance, errors, and usage of the EstateWise application.

Why we use it: To ensure the application runs smoothly, identify issues quickly, and gather insights into user behavior and system performance.

12.1 Logging

We use Winston for logging in the backend. The logging system is configured to log messages at different levels (info, warn, error) and to output logs in JSON format for easy parsing and analysis:

Logs are written to both the console and a file.
The log file is rotated daily to prevent it from growing too large.
We log important events such as:
- Incoming requests
- Responses sent to clients
- Errors encountered during processing
- Embedding generation times
- Database query times
- Expert model invocations
- User ratings for AI responses
- and more...
Each log entry includes:
- Timestamp
- Log level (info, warn, error)
- Message
- Additional metadata (e.g., request ID, user ID, etc.)

12.2 Monitoring

We use Prometheus to collect metrics from the application, including:
- Request counts and latencies
- Error rates
- Embedding generation times
- Database query times
- And more…
Prometheus is configured to scrape metrics from the application at regular intervals.
The app exposes a /metrics endpoint that Prometheus can scrape.
- This endpoint provides a summary of the application’s performance metrics in a format that Prometheus can understand.
- It includes metrics such as:
  - http_requests_total: Total number of HTTP requests received
  - http_request_duration_seconds: Duration of HTTP requests in seconds
  - embedding_generation_duration_seconds: Duration of embedding generation in seconds
  - database_query_duration_seconds: Duration of database queries in seconds
  - And more…

12.3 Visualization

We use express-status-monitor to expose a /status endpoint to visualize the application’s health and performance metrics.
We also use Grafana to create dashboards for monitoring key metrics over time.
Grafana is configured to pull data from Prometheus and display it in a user-friendly format.
Dashboards include:
- Request rates and latencies
- Error rates
- Embedding generation times
- Database query times
- And more…

12.4 Datadog APM & Monitors

Datadog augments the Prometheus + Grafana stack with full-stack APM, centralized log management, and proactive monitoring:

APM Distributed Tracing – Automatic trace propagation across all services (backend, gRPC, MCP, agentic-ai) via dd-trace
Log Management – Centralized logs with automatic trace ↔ log correlation (DD_LOGS_INJECTION=true)
17 Production Monitors – Error rate, P95/P99 latency, pod crash loops, memory/CPU thresholds, ALB 5xx, ECS task failures, deploy frequency/duration
SLOs – 30-day availability (99.9%) and latency (P95 < 500ms) objectives with burn-rate alerts
Synthetic Checks – Multi-location HTTP health probes against /api/health every 5 minutes
DogStatsD Custom Metrics – Deploy events, counters, and histograms from deployment-control

See 📘 docs/datadog-integration.md for architecture, setup, and runbooks.

12.5 SonarQube & Snyk Integration

SonarQube provides continuous inspection of code quality across all 7 monorepo modules:

Multi-module config in sonar-project.properties with per-module source, test, and exclusion paths
Quality gate enforcement — CI builds break on new bugs, code smells, or coverage regressions
Local development server via docker compose -f docker/compose.sonarqube.yml up -d

Snyk provides four layers of security scanning:

flowchart LR
  SCA["SCA<br/>Dependency Vulns"] --> Gate{Severity Gate}
  SAST["Code SAST<br/>Source Patterns"] --> Gate
  Container["Container Scan<br/>Docker Image CVEs"] --> Gate
  IaC["IaC Scan<br/>Terraform · K8s · Helm"] --> Gate
  Gate -->|Pass| Deploy["Deploy"]
  Gate -->|Fail| Block["Block Pipeline"]

Scan Type	Tool	Scope
Dependency SCA	`snyk test`	All `package.json` dependency trees
Code SAST	`snyk code test`	TypeScript/JavaScript source patterns
Container	`snyk container test`	Docker image OS + app layer CVEs
IaC	`snyk iac test`	Terraform, Kubernetes, Helm, Docker Compose

Per-service policies in .snyk (root) and .snyk.d/ (overrides) control ignore/patch rules. Both tools gate Jenkins and AWS CodeBuild pipelines.

12.6 SRE Practices

EstateWise follows Google SRE principles with formal error budgets, multi-window burn-rate alerting, and progressive delivery. Key artifacts:

docs/SLO.md — canonical SLO definitions and error budget calculations
SRE.md — comprehensive SRE reference (alerting, incident response, DORA metrics)
sre-dashboard/ — real-time observability dashboard (port 4200)
kubernetes/monitoring/prometheus-config.yaml — Prometheus recording rules and burn-rate alerts

13. GitHub Actions CI/CD Pipeline

What it is: A continuous integration and deployment (CI/CD) pipeline that automates the testing, building, and deployment of the EstateWise application.

Why we use it: To ensure that every change to the codebase is tested, built, and deployed automatically, reducing the risk of introducing bugs and ensuring a smooth deployment process.

13.1 Workflow Configuration

The GitHub Actions workflow is defined in .github/workflows/ci.yml and includes the following steps:

Checkout Code Uses the actions/checkout@v4 action to pull the repository code to the runner.
Set Up Node.js Uses actions/setup-node@v4 to configure Node.js (version 18), along with caching of npm dependencies for faster runs.
Preflight Setup Performs environment verification, printing Node.js and npm versions, and showing the workspace structure.
Cache Dependencies Caches root, backend, and frontend node_modules using the actions/cache@v3 action.
Database Connectivity Check Runs a preflight database connectivity check using a decoded shell script, verifying DB credentials and availability.
Lint & Format Runs Prettier and ESLint to enforce code style and lint the entire workspace.
CodeQL Security Scan Initializes CodeQL for JavaScript and TypeScript, performs an autobuild, and then runs a static analysis scan to identify vulnerabilities.
Security & License Scan Performs npm audit, license checking, and static analysis with ESLint and Semgrep to catch vulnerabilities and license issues.
Backend Tests Executes backend unit tests using Jest, with support for matrix testing across MongoDB versions.
Frontend Tests Runs frontend unit tests with Jest, and executes E2E tests via Cypress and Selenium across multiple browsers.
Test Coverage Reports Generates and uploads code coverage reports for both backend and frontend components.
Build Stage Builds both the backend and frontend artifacts, storing them for later deployment steps.
Lighthouse Performance Audit Runs Lighthouse against the deployed preview to capture a performance report.
Documentation Generation Generates JSDoc and TypeDoc documentation for both backend and frontend codebases, and uploads them as artifacts.
Docker Publishing Builds and pushes Docker images for the backend and frontend to GitHub Container Registry (GHCR).
Image Vulnerability Scan Runs Trivy vulnerability scans on built Docker images before deployment.
Performance Benchmark Executes quick load tests using Artillery to validate application health endpoints.
Deployment Deploys to AWS infrastructure and Vercel, handling both static and dynamic deployments.
Pipeline Done Marks the CI/CD pipeline as complete with a final confirmation step.

13.2 Configuring Secrets

To ensure the CI/CD pipeline runs smoothly, you need to configure the following secrets in your GitHub repository:

Name	Purpose
`MONGO_URI`	MongoDB connection string
`GOOGLE_AI_API_KEY`	Google Gemini & Embedding API key
`PINECONE_API_KEY`	Pinecone service key
`PINECONE_ENVIRONMENT`	Pinecone environment identifier
`PINECONE_INDEX`	Name of your Pinecone vector index
`JWT_SECRET`	Secret key for signing JWTs
`WEBAUTHN_RP_ID` / `WEBAUTHN_RP_NAME` / `WEBAUTHN_ORIGINS`	Passkey RP configuration (must be set per-environment)
additional secrets for Docker, AWS, and Vercel deployment as needed	...

13.3 Monitoring and Troubleshooting

What it is: This section describes how to monitor, debug, and troubleshoot the GitHub Actions CI/CD pipeline for the EstateWise application.

Monitoring the pipeline:

GitHub Actions Dashboard Navigate to the Actions tab in your GitHub repository to view real-time build and deployment progress, logs, and job status.
Code Scanning Alerts Security issues identified by CodeQL or Trivy will appear under the repository’s Security > Code scanning alerts section.
Artifacts Coverage reports, documentation, and build artifacts are uploaded at various pipeline stages and are accessible from the workflow run summary in the Actions dashboard.

Troubleshooting tips:

Workflow Failures
- Inspect the job logs in the Actions tab for stack traces or error messages.
- Ensure environment variables and required secrets are correctly set up under Settings > Secrets and variables.
- Check that the correct Node.js version and package dependencies are defined in package.json and package-lock.json.
Database or connectivity issues
- Confirm that the DB_* secrets are valid and that the database is reachable from the GitHub runner.
- Validate that firewalls or network rules allow connections from GitHub-hosted runners.
Container Image and Deployment
- If Docker/Podman pushes fail, verify that GHCR credentials are correct and that the user has permission to push packages.
- For AWS or Vercel deploy steps, confirm that deployment secrets are valid and permissions are correct.

Common troubleshooting commands:

Re-run failed jobs from the GitHub Actions dashboard with the Re-run jobs button
Use npm ci locally to replicate dependency issues
Use npm test, npm run lint, or npx typedoc locally to reproduce test or documentation build failures

13.4 Deployment Rollback

What it is: A strategy to revert the application to a previous stable version if a deployment introduces critical bugs or downtime.

Why it matters: Rolling back quickly ensures minimal disruption for users and reduces operational risk.

How to roll back:

Vercel Rollback
- Visit the Vercel dashboard for the estatewise project.
- Navigate to Deployments, identify a previous successful build, and promote it to production by clicking Restore.
AWS Infrastructure Rollback
- Use your infrastructure-as-code (IaC) tools (e.g., CloudFormation, Terraform) to redeploy the previous working stack definition.
- If using Docker images, redeploy the prior known working tag (e.g., ghcr.io/owner/estatewise-app-frontend:previous).
Container Images Rollback
- If the latest pushed image is broken, redeploy an earlier image tagged as latest or use a SHA-tagged release:
```
docker pull ghcr.io/your-org/estatewise-app-frontend:<previous-sha>
docker run ...
# Or with Podman:
# podman pull ghcr.io/your-org/estatewise-app-frontend:<previous-sha>
# podman run ...
```
- For Kubernetes or similar orchestrators, update the deployment to use the previous container image tag.

Checklist before rollback:

✅ Confirm the production issue is critical enough to warrant a rollback
✅ Notify the team and stakeholders about the rollback
✅ Review logs, alerts, and monitoring dashboards to confirm the nature of the failure
✅ Validate that the prior deployment is healthy before promoting it

Post-rollback:

Perform a root cause analysis (RCA)
Document lessons learned in your incident reports
Create issues or tasks to prevent similar failures in future releases

13.5 Other CI/CD Enhancements (Travis, GitLab CI, Jenkins)

The EstateWise repo also includes alternative CI/CD configurations for Travis CI, GitLab CI, and Jenkins. These configurations provide similar functionality to the GitHub Actions pipeline, ensuring flexibility for different development teams and environments.

These alternative configurations can be found in the following files:

.travis.yml – Travis CI configuration
.gitlab-ci.yml – GitLab CI configuration
Jenkinsfile – Jenkins pipeline configuration

Feel free to adapt and extend these configurations based on your specific requirements and infrastructure.

For more details, refer to the DEVOPS.md and DEPLOYMENTS.md documentation files.

14. Context Engineering System

The context-engineering/ package is an enterprise-grade subsystem that provides AI agents with structured domain knowledge through a knowledge graph, knowledge base, and intelligent context window management. It runs as a standalone TypeScript service on port 4200, exposing a REST API, WebSocket events, and a D3.js visualization dashboard.

14.1 Knowledge Graph Architecture

The knowledge graph is an event-driven, in-memory graph engine built on typed adjacency lists with optional Neo4j persistence.

Data Model:

erDiagram
  GraphNode {
    string id PK
    NodeType type
    string label
    json properties
    float[] embedding
    NodeMetadata metadata
  }
  GraphEdge {
    string id PK
    string source FK
    string target FK
    EdgeType type
    float weight
    json properties
    EdgeMetadata metadata
  }
  GraphNode ||--o{ GraphEdge : "source"
  GraphNode ||--o{ GraphEdge : "target"

Node Type Taxonomy (12 types):

Type	Description	Seed Count
Property	Real estate listings with zpid, price, address	0 (populated via ingestion)
Concept	Domain concepts: valuation, mortgage, ROI, etc.	14
Entity	Named entities extracted from text	0
Topic	High-level categories: residential, commercial, trends	6
Document	Knowledge base document references	0
Conversation	Chat session nodes	0
Agent	AI agent nodes: Planner, GraphAnalyst, etc.	11
Tool	MCP tool nodes: graph.similar, finance.mortgage, etc.	7
Workflow	Multi-step process definitions	4
Neighborhood	Geographic neighborhood nodes	0
ZipCode	ZIP code area nodes	0
MarketSegment	Market classification segments	0

Edge Type Taxonomy (14 types):

Type	Semantics	Typical Source -> Target
SIMILAR_TO	Similarity relationship	Property -> Property
RELATED_TO	General association	Concept -> Concept
BELONGS_TO	Membership	Entity -> Topic
MENTIONS	Reference in text	Document -> Entity
DERIVED_FROM	Derivation chain	Concept -> Concept
DEPENDS_ON	Dependency	Workflow -> Tool
LINKS_TO	Hyperlink/reference	Document -> Document
PART_OF	Containment	Concept -> Topic
USES	Tool usage	Agent -> Tool
PRODUCES	Output creation	Tool -> Concept
IN_NEIGHBORHOOD	Geographic membership	Property -> Neighborhood
IN_ZIP	ZIP code membership	Property -> ZipCode
HAS_CAPABILITY	Agent capability	Agent -> Concept
PRECEDES	Temporal ordering	Workflow -> Workflow

Traversal Algorithms:

The graph engine implements 8 traversal algorithms, each respecting configurable TraversalOptions (max depth, edge type filters, node type filters, minimum weight, direction):

flowchart LR
  subgraph "Search"
    BFS[BFS<br/>breadth-first]
    DFS[DFS<br/>depth-first]
  end
  subgraph "Pathfinding"
    DIJ[Dijkstra<br/>shortest path]
    ALL[All Paths<br/>enumeration]
  end
  subgraph "Analysis"
    PR[PageRank<br/>importance]
    CD[Community Detection<br/>label propagation]
    CC[Connected Components<br/>component discovery]
    BC[Betweenness Centrality<br/>bridge identification]
  end

Fluent Query Builder:

import { query, NodeType, EdgeType } from "@estatewise/context-engineering";

const result = query(graph)
  .match(NodeType.Concept)
  .where({ "metadata.importance": { $gte: 0.7 } })
  .traverse(EdgeType.RELATED_TO, { maxDepth: 2 })
  .orderBy("metadata.importance", "desc")
  .limit(10)
  .execute();

Supported operators: $gt, $lt, $gte, $lte, $eq, $ne, $contains, $in. Supports dot-path property access (e.g., properties.price, metadata.tags).

Neo4j Synchronization:

The Neo4jSyncManager provides bidirectional sync between the in-memory graph and Neo4j:

pushToNeo4j(): MERGE semantics, batched in 500-node chunks
pullFromNeo4j(): Merges into in-memory graph without overwriting newer local nodes
Graceful degradation: Neo4j unavailability never crashes the application
Configuration via NEO4J_URI, NEO4J_USERNAME, NEO4J_PASSWORD, NEO4J_ENABLE

14.2 Knowledge Base & Document Retrieval

The knowledge base stores documents with automatic chunking and TF-IDF embedding generation.

Document Processing Pipeline:

flowchart LR
  RAW[Raw Document] --> CHUNK[Auto-Chunker<br/>~500 tokens<br/>50 token overlap]
  CHUNK --> EMB[TF-IDF Embedder<br/>128-dimensional<br/>hash projection]
  EMB --> STORE[Document Store<br/>Map&lsaquo;id, KBDocument&rsaquo;]
  STORE --> IDX[Chunk Index<br/>embedding vectors]

Retrieval Strategies:

Strategy	Method	Score Composition	Best For
Semantic	Cosine similarity on embeddings	Vector distance (0-1)	Conceptual matches, paraphrases
Keyword	TF-IDF token frequency	Term overlap ratio	Exact term matches, technical queries
Hybrid	Weighted blend	65% semantic + 35% keyword	General-purpose retrieval

All strategies support:

SearchFilter predicates (eq, ne, gt, lt, contains, in)
Recency boost (exponential decay weighting for newer documents)
Frequency boost (prioritize frequently accessed documents)

Seed Documents (10): The knowledge base comes pre-loaded with ~4,000 tokens of domain reference material covering platform overview, property search, market analysis, financial tools, graph system, agent capabilities, MCP tools, neighborhood analysis, commute analysis, and compliance regulations.

14.3 Context Assembly Engine

The context engine orchestrates context assembly from multiple providers with ranking and priority-based token allocation.

Assembly Pipeline:

sequenceDiagram
  participant A as Agent/Tool
  participant E as ContextEngine
  participant GP as GraphProvider
  participant DP as DocumentProvider
  participant CP as ConversationProvider
  participant TP as ToolResultProvider
  participant R as Ranker
  participant W as ContextWindow

  A->>E: assembleContext(query)
  par Parallel collection
    E->>GP: getContext()
    E->>DP: getContext()
    E->>CP: getContext()
    E->>TP: getContext()
  end
  GP-->>E: graph items
  DP-->>E: document items
  CP-->>E: conversation items
  TP-->>E: tool result items
  E->>R: rank(items, "combined")
  R-->>E: scored items
  E->>W: allocate(items)
  W-->>E: budget-fitted items
  E-->>A: AssembledContext

Priority Tiers:

Priority	Level	Weight	Examples
Critical (4)	Highest	Always included	System prompts, safety rules
High (3)	High	Included first	Direct graph/KB matches
Medium (2)	Medium	Space permitting	Related context, recent conversation
Low (1)	Low	If budget allows	Background knowledge
Background (0)	Lowest	First evicted	Oldest/least relevant items

Ranking Strategies:

Strategy	Formula	Use Case
Recency	Exponential decay (30-min half-life)	Prioritize recent information
Relevance	70% provider score + 30% keyword overlap	Match quality
Importance	Priority tier + metadata importance	Domain significance
Combined	40% relevance + 30% recency + 30% importance	Default strategy

Per-Agent Token Budgets:

Agent Role	Budget	Reserved
Orchestrator/Coordinator	12,000	3,000
PropertyAnalyst	10,000	2,500
FinanceAnalyst	8,000	2,000
GraphAnalyst	6,000	1,500
Default	8,000	2,000

14.4 Ingestion Pipeline

flowchart TB
  subgraph "Ingestion Sources"
    S1["{ type: 'property', data: {...} }"]
    S2["{ type: 'conversation', data: [...] }"]
    S3["{ type: 'document', data: '...' }"]
  end

  subgraph "Router"
    ING[Ingester]
  end

  subgraph "Parsers"
    PP[PropertyParser<br/>creates Property, Neighborhood,<br/>ZipCode nodes + IN_NEIGHBORHOOD,<br/>IN_ZIP edges]
    CP[ConversationParser<br/>creates Conversation node,<br/>MENTIONS edges to ZPIDs,<br/>RELATED_TO edges to concepts]
    DP[DocumentParser<br/>creates Document, Concept<br/>nodes + LINKS_TO, MENTIONS edges]
  end

  subgraph "Output"
    KG[Knowledge Graph<br/>nodes + edges]
    KB[Knowledge Base<br/>documents + chunks]
  end

  S1 --> ING --> PP
  S2 --> ING --> CP
  S3 --> ING --> DP
  PP & CP & DP --> KG & KB

Each parser returns a ParsedData object containing nodes, edges, and documents. The Ingester resolves node labels to IDs (creating new nodes if needed) and persists everything to the graph and knowledge base. Batch ingestion collects errors without aborting the entire batch.

14.5 MCP Tool Integration

10 context engineering tools are registered in the MCP server for agent access:

Tool	Args	Description
`context.search`	query, strategy?, limit?	Hybrid search across knowledge base
`context.assembleForAgent`	agentRole, query, maxTokens?	Assemble full context window
`context.graphTraverse`	startNodeId, edgeTypes?, maxDepth?, direction?	BFS graph traversal
`context.graphQuery`	nodeType?, filters?, limit?	Query graph with filters
`context.ingest`	type, title, content, tags?	Ingest new data
`context.getStats`	--	System metrics snapshot
`context.getGraphOverview`	limit?	Full graph for visualization
`context.findRelated`	concept, maxDepth?, limit?	Find related knowledge nodes
`context.getNodeDetail`	nodeId, includeNeighbors?	Detailed node information
`context.getTimeline`	limit?	Recent activity timeline

4 MCP resources are also registered:

context://graph/stats -- Current graph statistics
context://kb/stats -- Knowledge base statistics
context://metrics/snapshot -- Metrics snapshot
context://graph/nodes/{type} -- Nodes filtered by type

14.6 D3 Visualization Dashboard

The context engineering system includes a professional dark-themed D3.js visualization dashboard:

Technology: D3.js v7 (CDN), vanilla JavaScript, CSS Grid layout

Features:

Force-directed graph with D3 force simulation (link, charge, center, collide, clustering forces)
12 color-coded node types with importance-based sizing (radius 8-24px)
Unicode emoji icons per node type (Property, Agent, Tool, Concept, etc.)
Interactive: zoom/pan (0.1x-5x), drag, click-to-select with highlight, double-click to center
Left sidebar: search, node/edge type filters, statistics, color legend
Right sidebar: node detail panel, knowledge base search, metrics charts
Footer: real-time metric pills (assemblies, avg time, cache hits, ingestions)
WebSocket integration for live graph mutation events

14.7 Monitoring & Observability

The monitoring system tracks all context engineering operations:

Metric	Type	Source
Context assemblies	Counter + timing	ContextEngine
KB searches	Counter + timing	KnowledgeBase
Graph traversals	Counter + timing	Traversal functions
Ingestions	Counter + timing + errors	Ingester
Cache hits/misses	Counter	ToolResultProvider

Metrics are collected in a rolling window (last 1,000 events) and exposed via:

GET /api/context/metrics -- Aggregate snapshot
GET /api/context/metrics/timeseries?metric=context_assembly -- Time-series data
MCP resource context://metrics/snapshot

15. Appendices

Additional resources, diagrams, and references for developers and data scientists who are interested working on EstateWise.

A. Environment Variables Reference

The following environment variables are required for the application to function correctly. Ensure they are set in your .env file.

Name	Purpose
`MONGO_URI`	MongoDB connection string
`JWT_SECRET`	Secret key for signing JWTs
`WEBAUTHN_RP_ID`	Effective domain of the frontend for passkey binding (e.g. `localhost`, `estatewise.vercel.app`)
`WEBAUTHN_RP_NAME`	Human-readable RP name shown in OS passkey picker (default `EstateWise`)
`WEBAUTHN_ORIGINS`	Comma-separated browser origins accepted during WebAuthn ceremonies
`GOOGLE_AI_API_KEY`	Google Gemini API key (chat + embeddings)
`OPENAI_API_KEY`	OpenAI API key (fallback/runtime + CrewAI)
`PINECONE_API_KEY`	Pinecone service key
`PINECONE_INDEX`	Name of your Pinecone vector index
`NEO4J_URI`	Neo4j endpoint for graph workflows
`NEO4J_USERNAME` / `NEO4J_PASSWORD`	Neo4j credentials
`AGENT_RUNTIME`	Default agentic runtime (`default`, `langgraph`, `crewai`)
`THREAD_ID`	Optional LangGraph conversation continuity identifier
`LANGSMITH_ENABLED`	Enable/disable LangSmith tracing explicitly
`LANGSMITH_API_KEY`	LangSmith authentication key
`LANGSMITH_PROJECT`	LangSmith project name
`LANGSMITH_ENDPOINT`	Optional custom LangSmith endpoint
`LANGSMITH_RUN_TAGS`	Comma-separated baseline trace tags
`LANGSMITH_STRICT`	Fail fast on tracing misconfiguration
`CONTEXT_PORT`	Port for context engineering API + UI (default: 4200)
`CONTEXT_API_BASE_URL`	Base URL for context API when used by MCP tools
`DD_API_KEY`	Datadog API key (required when Datadog is enabled)
`DD_APP_KEY`	Datadog application key (Terraform monitors/SLOs)
`DD_AGENT_HOST`	Datadog agent host (auto-detected on K8s via `status.hostIP`)
`DD_DOGSTATSD_PORT`	DogStatsD UDP port (default: `8125`)
`DD_SERVICE`	Service name for unified tagging (e.g. `estatewise-backend`)
`DD_ENV`	Environment tag (e.g. `production`, `staging`)
`DD_VERSION`	Deployed version for trace/log correlation
`DD_LOGS_INJECTION`	Enable automatic log ↔ trace correlation (`true`)
`SONAR_HOST_URL`	SonarQube server URL (default: `http://localhost:9000`)
`SONAR_TOKEN`	SonarQube authentication token for CI analysis
`SONAR_PROJECT_KEY`	SonarQube project key (default: `estatewise`)
`SNYK_TOKEN`	Snyk API token for CLI and CI scanning
`SNYK_ORG`	Snyk organization ID for `snyk monitor` reports
`SNYK_SEVERITY_THRESHOLD`	Minimum severity to fail builds (`low`, `medium`, `high`, `critical`)

B. AI/ML Flow Chart

This flowchart illustrates the AI/ML pipeline, from data ingestion to embedding generation, RAG, and expert model invocation.

C. Overall App's Flow Diagram

Below is a simplified flow diagram of the entire application architecture, from user interaction to backend processing and data storage.

flowchart TB
  User["👤 User Interaction<br/>(Chat, Signup, Login, etc.)"]

  subgraph Frontend["Frontend (Next.js, React)"]
    UI["Responsive UI + Animations"]
    APICalls["API Calls to Backend"]
    Ratings["User Ratings for AI Responses"]
  end

  subgraph Backend["Backend (Express + TypeScript)"]
    Auth["Auth (JWT, Signup/Login)"]
    ConvoAPI["Conversation & Chat APIs"]
    AIProc["AI Processing & RAG"]
    Swagger["Swagger API Docs"]
    Container["Containerized (Docker/Podman)"]
  end

  subgraph DataStores["Data Layer"]
    MongoDB["MongoDB<br/>(User Data, Convo History)"]
    Pinecone["Pinecone Vector DB<br/>(Knowledge Base)"]
  end

  subgraph AIEngine["Response Processing"]
    Gemini["Google Gemini API"]
    RAG["RAG (kNN) Retrieval"]
    KMeans["k-Means Clustering"]
    Agentic["Agentic AI Orchestration"]
    MoE["Mixture of Experts<br/>(Data Analyst, Lifestyle Concierge,<br/>Financial Advisor, Neighborhood Expert,<br/>Cluster Analyst)"]
    Feedback["Feedback Loop & Reinforcement<br/>Learning for Expert Weights"]
  end

  subgraph Display["Frontend Display"]
    ChatResp["Show Chat Response"]
    UpdateUI["Update UI (Conversation)"]
    AuthFlows["User Authentication Flows"]
    ConvoMgmt["Search & Manage Conversations"]
    DataViz["Data Visualizations"]
  end

  subgraph Observability["Observability"]
    Prometheus["Prometheus + Grafana"]
    Datadog["Datadog APM + Monitors + SLOs"]
  end

  User --> Frontend
  Frontend -->|"REST API Calls"| Backend
  Backend <--> DataStores
  Backend --> AIEngine
  AIEngine --> Display
  Backend --> Observability

D. Mermaid Sequence Diagram

This UML sequence diagram illustrates the flow of data and interactions between the user, UI, API, agent, decision model, Pinecone, experts, and Gemini models:

sequenceDiagram
    participant User
    participant UI as Next.js UI
    participant API as /api/chat
    participant Agent as runEstateWiseAgent
    participant DecisionAI as Decision Model
    participant Pinecone
    participant Experts as MoE Pipeline & Chain-of-Thought
    participant Gemini as Gemini Models

    User->>UI: sendMessage()
    UI->>API: POST /api/chat { message }
    API->>Agent: runEstateWiseAgent(prompt)

    Agent->>DecisionAI: {"usePropertyData": ?}
    DecisionAI-->>Agent: {"usePropertyData": true/false}

    alt usePropertyData = true
        Agent->>Pinecone: queryPropertiesAsString(prompt,50)\nqueryProperties(prompt,50)
        Pinecone-->>Agent: propertyContext + rawResults
        Agent->>Experts: chatWithEstateWise(history, prompt, propertyContext)
    else usePropertyData = false
        Agent->>Experts: chatWithEstateWise(history, prompt, userContext)
    end

    Experts->>Gemini: expert & merger invocations
    Gemini-->>Experts: mergedResponse

    Experts-->>API: { finalText, expertViews }
    API->>UI: 200 { response }
    UI->>User: renderResponse()

Diagram not working? Paste the code into a compatible Mermaid editor, like Mermaid Live Editor or Mermaid Preview to visualize it.

E. Vector Schema & Metadata Example

Below is an example of the vector schema and metadata stored in Pinecone:

{
  "id": "123456",
  "values": [0.0123, -0.0456 /* 1536 dimensions */],
  "metadata": {
    "zpid": 123456,
    "address": "{\"streetAddress\":\"123 Main St\",\"city\":\"Chapel Hill\",\"state\":\"NC\",\"zipcode\":\"27514\"}",
    "price": 350000,
    "bedrooms": 3,
    "bathrooms": 2,
    "livingArea": 1450,
    "yearBuilt": 1995,
    "homeType": "SingleFamily",
    "listingDataSource": "Legacy",
    "description": "Charming 3-bed home near UNC campus..."
  }
}

Thank you for reading through the technical documentation of EstateWise! If you have any questions or need further clarification, feel free to reach out to us! 🏠

🔝 Back to Top

🏠 Back to README

FilesExpand file tree

TECH_DOCS.md

Latest commit

History

TECH_DOCS.md

File metadata and controls

EstateWise: AI‑Powered Real Estate Assistant for Chapel Hill, NC - AI/ML/Infrastructure Technical Documentation

Table of Contents

0. Production Readiness Updates (2026-04)

1. Data Ingestion & Validation

1.1 Streaming Parser Setup

1.2 Schema & Field Selection

1.3 Cleaning & Sanitization Utilities

1.4 Error Handling & Logging

2. Embedding Generation & Vector Store

2.1 Prompt Engineering for Embeddings

2.2 Google Embedding API Integration

2.3 Pinecone Batch Upsert Strategy

2.4 Monitoring & Retry Logic

3. Retrieval‑Augmented Generation (RAG)

3.1 KNN Query Implementation

3.2 Assembling Retrieval Context

3.3 Caching Frequent Queries

3.4 Hybrid RAG (Vector + Graph)

4. RAG‑Decision Orchestration Layer

4.1 JSON‑Only Decision Protocol

4.2 Simplified Control Flow

4.3 Implementation Highlights

5. Clustering & Similarity Analysis

5.1 Numeric Feature Extraction & Normalization

5.2 Plain‑JS K‑Means Implementation

5.3 Cluster Context Injection

6. Mixture‑of‑Experts (MoE) Ensemble

6.1 Expert Definitions & System Prompts

6.2 Parallel Expert Invocations

6.3 Weight Normalization & Feedback Loop

6.4 Master Merger Model

6.5 Prompt Engineering

6.6 Agentic AI Runtime

7. Chain-of-Thought Reasoning

7.1 Implementation

7.2 Example

7.3 Benefits

7.4 Agentic AI Orchestration

7.5 MCP Server (Model Context Protocol)

7.6 Agent-to-Agent Communication

8. Backend API & Data Layer

8.1 Express.js Routes & Controllers

8.2 MongoDB Models & Conversations

8.3 Authentication & JWT Workflow

8.4 OpenAPI / Swagger Integration

8.5 tRPC API

8.6 gRPC Services

9. Frontend Integration

10. Infrastructure & Deployment

10.1 Deployment Strategies

Blue-Green Deployment

Canary Deployment

Rolling Update

10.2 Production Kubernetes Manifests

10.3 Multi-Cloud Deployment Options

10.4 CI/CD Pipeline (Enhanced Jenkins)

SonarQube Code Quality

Snyk Vulnerability Scanning

10.5 Monitoring & Observability

Prometheus Metrics

Grafana Dashboards

Centralized Logging

Datadog Observability

SRE Dashboard & SLO Monitoring

10.6 High Availability Setup

10.7 Disaster Recovery

Automated Backups

Recovery Procedures

10.8 DevOps Metrics

10.9 Operational Runbooks

10.10 Chaos Engineering

11. Challenges

11.1 Large‑Scale Data Ingestion

11.2 Embedding Generation & Rate Limits