LLM Task Chat

A Databricks full-stack chat app that shows how to capture conversation meaning, store it in governed Lakebase Postgres, and log eval metadata to MLflow — with human confirmation before anything important is committed.

The design separates fast chat from deeper analysis, and lets a user manually trigger downstream jobs. That makes the pipeline easy to follow and demonstrates how to orchestrate Databricks tools (Apps, Lakebase, Model Serving, bundle jobs, MLflow) without hiding work behind magic automation.

Two LLMs, two jobs

LLM	Role	When it runs
Capture LLM	Replies in chat; may suggest a task title/description	Every message — sync, in the chat app
Judge LLM	Meaning capture — assignability, confidence, sentiment, themes	After the turn is saved — async bundle job

The capture model drives the user experience. The judge model is a second opinion on what the turn meant for leadership and eval — it does not change the reply the user already saw.

How data flows (Lakebase first, MLflow second)

Nothing goes to MLflow until the turn lives in Lakebase. The order is intentional:

1. Chat          User message → capture LLM → reply (+ optional task proposal)
                 Saved to Lakebase: messages + eval queue row

2. Judge         (manual or scheduled job) reads queue → judge LLM
                 Writes meaning to Lakebase: routing, sentiment, themes, flags

3. MLflow        (manual job, after judge) reads Lakebase → logs eval run
                 Metadata only — params, metrics, redacted artifacts in workspace experiment

4. Tasks         User Accept/Reject in UI → only Accept writes app.tasks

Lakebase is the source of truth. MLflow is an observability layer on top of what was already stored and judged — not a parallel write path.

Why manual triggers?

Judge and MLflow jobs are not wired into every chat request. In the deployed dev app, leadership can start them from the Chat tab:

1. Run judge worker — process pending turns and capture meaning in Lakebase
2. Run MLflow sidecar — read judged rows from Lakebase and append runs to the MLflow experiment

This is deliberate:

• Chat stays fast — no judge or MLflow latency on the hot path
• Pipeline is visible — you see queue → judge → MLflow as separate steps
• Tools are maneuverable — bundle jobs, Lakebase OAuth, serving endpoints, and MLflow are composed explicitly, not buried in one monolithic handler

Scheduled jobs exist in the bundle (paused in dev); the UI trigger is for learning and smoke-testing the full path.

Human-in-the-loop

The LLM may propose a task; it never creates one. Only an explicit Accept in the UI calls POST /api/tasks with confirmation. The judge may flag a turn as assignable — that still does not auto-create a task.

What else is in the app

• Leadership surfaces (dev-gated) — Review inbox, eval health, temperature/sentiment, read-only Lakebase and MLflow browsers
• Unity Catalog — analytics sync for governed sharing across teams
• Security defaults — redacted leadership APIs, OAuth Lakebase, no client-side model keys

Stack

Layer	Tech
Frontend	React + Vite
Backend	FastAPI
Operational DB	Lakebase Postgres
Analytics	Unity Catalog
LLMs	Databricks Model Serving (capture + judge endpoints)
Eval logging	MLflow (workspace experiment)
Jobs & deploy	Databricks Asset Bundles + Apps

Prerequisites

Before you deploy, you need in your Databricks workspace:

Resource	Notes
CLI auth	databricks auth login (prefer profile auth over PATs in repo)
Lakebase Postgres	Autoscaling project + branch; app connects via OAuth at runtime
Model Serving	At least one chat endpoint; judge can reuse the same endpoint
MLflow experiment	Name /Shared/llm_task_chat_eval (or your override) — see below
UC catalog (optional)	For analytics sync jobs — default dev_llm_chat

Configure for your workspace

Edit databricks.yml variables to match your environment (defaults are placeholders):

Variable	Purpose
lakebase_project_id	Lakebase project slug
lakebase_branch	Full branch path, e.g. projects/<project>/branches/production
lakebase_database	Database resource path on that branch
llm_serving_endpoint_name	Capture LLM endpoint name
judge_serving_endpoint_name	Judge endpoint (often same as capture)
uc_dev_catalog	UC catalog for sync jobs

Override at deploy time without editing the file:

export BUNDLE_VAR_lakebase_branch="projects/my-project/branches/production"
export BUNDLE_VAR_llm_serving_endpoint_name="databricks-meta-llama-3-3-70b-instruct"
uv run databricks bundle deploy -t dev

Copy .env.example → .env for local CLI and tests. Never commit .env.

Bundle job configuration

Serverless jobs do not receive compute environment variables from the bundle YAML. Instead, job scripts bootstrap from files synced with the bundle:

File	Purpose
config/job_env.defaults.json	Committed — non-secret defaults deployed with the bundle
config/job_env.json	Local override (gitignored) — copy from config/job_env.example.json

Update job_env.defaults.json before deploy if your endpoint names or Lakebase branch differ from the placeholders. For secrets-free local dry-runs, copy the example to config/job_env.json and adjust values there.

MLflow experiment (one-time)

Create or tag the workspace experiment before running the MLflow sidecar:

cp config/job_env.example.json config/job_env.json   # edit branch + endpoints
uv run python scripts/ensure_genai_experiment.py

The sidecar resolves the experiment by name (MLFLOW_EXPERIMENT_NAME) when no ID is set. Optionally set MLFLOW_EXPERIMENT_ID in your local job_env.json after creation.

Grant the app service principal and job run-as identity CAN READ on the experiment.

Lakebase schema

Apply SQL migrations in order (OAuth — set LAKEBASE_BRANCH_ID in .env):

for f in sql/lakebase/00*.sql sql/lakebase/grants_app_role.sql; do
  uv run python scripts/apply_lakebase_sql.py "$f"
done

Apply UC analytics DDL on a SQL warehouse (see docs/uc-sync.md).

Quick start (local)

uv venv && uv sync
cp .env.example .env          # fill workspace values
cp config/job_env.example.json config/job_env.json   # optional for job dry-runs

# Backend
uv run uvicorn main:app --app-dir app --host 127.0.0.1 --port 8000

# Frontend (separate terminal)
npm install && npm run dev

Open the Vite dev server (default http://127.0.0.1:5173); it proxies API calls to the backend.

Deploy (dev)

npm run build
uv run databricks bundle validate -t dev
uv run databricks bundle deploy -t dev
uv run databricks bundle run llm_task_chat -t dev

The CLI prints the app URL when the deploy succeeds.

Post-deploy checklist:

1. Lakebase migrations applied (see above)
2. config/job_env.defaults.json matches your branch and endpoints (re-deploy after edits)
3. MLflow experiment exists and ACLs are set
4. Chat in the app → Run judge worker → Run MLflow sidecar to walk the full pipeline

Smoke-test bundle jobs from CLI:

uv run databricks bundle run turn_judge_worker -t dev
uv run databricks bundle run turn_mlflow_sidecar -t dev

Tests

uv run pytest tests/ --ignore=tests/integration   # unit tests (no live workspace)
npm run test:e2e                                   # Playwright (backend must be running)

Integration tests under tests/integration/ require live Lakebase, LLM, and env vars — run only in your workspace.

Browser session behavior

The chat UI stores your session id in browser local storage and reloads messages from Lakebase on refresh. Task suggestions remain drafts until you explicitly Accept. Leadership tabs remember your last tab and can filter Review/Temperature to the active chat session.

Troubleshooting

Symptom	Likely cause
Judge job fails immediately on startup	Job config missing — check config/job_env.defaults.json is deployed; override locally with config/job_env.json
MLflow sidecar fails on startup	Experiment name not found — run ensure_genai_experiment.py or fix MLFLOW_EXPERIMENT_NAME
Chat works but jobs cannot connect to Lakebase	Branch path wrong in defaults; grants not applied (grants_app_role.sql)
Bundle validate warnings	Run uv run databricks bundle validate -t dev after editing resources/*.yml

Further reading

Doc	Contents
judge-path.md	Judge + MLflow pipeline in plain language
docs/uc-sync.md	Lakebase → Unity Catalog sync
scripts/README.md	Worker scripts and bundle job mapping
SECURITY.md	Security policy and reporting

License

MIT — see LICENSE.