PerfScope

Production-focused website performance checking platform.

PerfScope lets users submit a website URL from a web dashboard, runs real asynchronous checks through a Redis-backed worker, stores results in PostgreSQL, and exposes full operational observability with Prometheus/Grafana/Alertmanager.

Table of Contents

1. What You Get
2. Architecture
3. Tech Stack
4. Repository Structure
5. Quick Start (Local)
6. How It Works
7. Configuration
8. Self-Hosting (Production)
9. Monitoring, Logging, and Alerting
10. Security Model
11. Testing
12. CI/CD
13. Operations (Backups, Upgrades, Troubleshooting)
14. Roadmap

What You Get

• Real user-facing dashboard (Next.js 15)
• Real API (FastAPI) with validation, rate limiting, health/readiness, metrics
• Real background processing (Redis + RQ worker)
• Real persistence (PostgreSQL + Alembic migrations)
• Real website checks (HTTP/HTTPS fetch + redirects + timing + headers + TLS info)
• Real observability stack
  • Prometheus
  • Grafana dashboards (auto-provisioned)
  • Alertmanager with working local webhook receiver and SMTP hooks
  • Node Exporter + cAdvisor + Redis/Postgres exporters
  • Loki + Promtail for centralized logs
• Real deployment assets
  • Dockerfiles for frontend/API/worker
  • Docker Compose for local and production
  • Split Compose for 2-3 VPS topology
  • Terraform and Ansible templates

Architecture

Tech Stack

• Frontend: Next.js 15, TypeScript, Tailwind CSS, shadcn-style components, TanStack Query
• API: Python 3.12, FastAPI, SQLAlchemy, Pydantic, Alembic, Prometheus client
• Worker: Python 3.12, Redis, RQ, httpx
• Data: PostgreSQL, Redis
• Infra: Docker, Docker Compose, Nginx
• Observability: Prometheus, Grafana, Alertmanager, Loki, Promtail, Node Exporter, cAdvisor
• Provisioning/automation (optional): Terraform, Ansible
• Load testing: k6

Repository Structure

.
├── apps/
│   ├── common/          # shared Python package (models, checker, config, queue, jobs)
│   ├── api/             # FastAPI app + Alembic + backend tests
│   ├── worker/          # RQ worker process
│   └── frontend/        # Next.js dashboard
├── infra/
│   ├── nginx/
│   ├── prometheus/
│   ├── grafana/
│   ├── alertmanager/
│   ├── loki/
│   ├── promtail/
│   ├── docker/          # split compose files for multi-node deployment
│   ├── terraform/
│   └── ansible/
├── scripts/
├── docs/
├── tests/
├── docker-compose.yml
└── docker-compose.prod.yml

Quick Start (Local)

Prerequisites

• Docker Engine + Docker Compose plugin
• 4+ CPU cores and 8+ GB RAM recommended (full stack includes observability services)
• No API keys are required for core website checks.

Copy/Paste Bring-Up (Whole Stack)

Use this exactly from the repo root.
If your Docker user is not in the docker group, prepend sudo to each docker compose command.

cd /path/to/PerfScope

# Create env file once (safe if already present)
[ -f .env ] || cp .env.example .env

# Clean old state (containers, volumes, orphaned services)
docker compose down -v --remove-orphans

# Build app images fresh
docker compose build --no-cache api worker frontend

# Start core services first (data + app path)
docker compose up -d postgres redis
docker compose up -d api
docker compose up -d worker
docker compose up -d frontend nginx

# Start observability stack
docker compose up -d prometheus grafana alertmanager loki promtail node-exporter cadvisor redis-exporter postgres-exporter alert-webhook

# Check status
docker compose ps --all

Copy/Paste End-to-End Verification

# 1) Basic health
curl -fsS http://localhost/api/health
curl -fsS http://localhost/api/ready

# 2) Create a real check
CHECK_ID=$(
  curl -sS -X POST http://localhost:8000/api/checks \
    -H 'Content-Type: application/json' \
    -d '{"url":"https://example.com"}' \
  | python3 -c 'import sys, json; print(json.load(sys.stdin)["id"])'
)
echo "CHECK_ID=$CHECK_ID"

# 3) Poll until done/failed
for i in $(seq 1 30); do
  RESP=$(curl -sS "http://localhost:8000/api/checks/$CHECK_ID")
  STATUS=$(printf '%s' "$RESP" | python3 -c 'import sys, json; print(json.load(sys.stdin)["status"])')
  echo "[$i] status=$STATUS"
  if [ "$STATUS" = "done" ] || [ "$STATUS" = "failed" ]; then
    echo "$RESP"
    break
  fi
  sleep 2
done

Copy/Paste If Jobs Stay pending

# See current container states
docker compose ps --all

# Check API + worker logs
docker compose logs --tail=200 api worker

# Queue depth (should go down when worker is healthy)
docker compose exec redis redis-cli LLEN rq:queue:website_checks

# Rebuild and restart job path only
docker compose build --no-cache api worker
docker compose up -d api worker
docker compose logs -f worker

1) Configure env

cp .env.example .env

2) Start the platform

docker compose up -d --build

Or use helper script:

./scripts/run-local.sh

3) Open services

• App: http://localhost
• API health: http://localhost/api/health
• API ready: http://localhost/api/ready
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3001 (default: admin/admin)
• Alertmanager: http://localhost:9093

4) Submit your first check

• Open the dashboard and submit a URL, for example https://example.com
• Watch status transition: pending -> running -> done/failed
• Open check details to see measured values

5) Stop services

docker compose down

To remove volumes too:

docker compose down -v

How It Works

1. User submits URL in UI.
2. API validates and normalizes URL, applies SSRF safety checks, creates a checks row in PostgreSQL.
3. API enqueues job into Redis queue.
4. Worker consumes job, marks check running, performs real HTTP/HTTPS request(s) with timeout and redirect handling.
5. Worker stores result (check_results) and marks check done or failed.
6. Frontend polls API and updates list/detail pages with real data.
7. Metrics/logs are scraped and visualized; alerts fire on configured conditions.

Configuration

All config is environment-driven. Start from .env.example.

Important variables:

• DATABASE_URL, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB
• REDIS_URL, QUEUE_NAME
• CORS_ORIGINS (JSON array string, e.g. ["https://app.example.com"])
• RATE_LIMIT_PER_MINUTE
• REQUEST_TIMEOUT_SECONDS, MAX_REDIRECTS
• ALLOW_PRIVATE_TARGETS (default false for SSRF safety)
• ENABLE_TLS_PROBE
• GRAFANA_ADMIN_USER, GRAFANA_ADMIN_PASSWORD
• ALERT_* for Alertmanager email integration

Self-Hosting (Production)

PerfScope supports two practical modes:

• Single host: everything on one VPS (fastest to deploy)
• 2-3 host split: data/app/monitoring separated (better isolation)

Option A: Single VPS Deployment

1. Provision Linux server (Ubuntu 24.04 recommended).
2. Install Docker + Compose plugin.
3. Clone repository and enter project directory.
4. Configure production .env:
   • set strong Postgres/Grafana credentials
   • set CORS to your public domain
   • set alerting values
5. Configure TLS certificates for nginx at:
   • infra/nginx/certs/fullchain.pem
   • infra/nginx/certs/privkey.pem
6. Start stack:

docker compose -f docker-compose.prod.yml up -d --build

7. Validate:

curl -fsS https://YOUR_DOMAIN/api/health
curl -fsS https://YOUR_DOMAIN/api/ready

Option B: 2-3 VPS Split Deployment

Use split compose files in infra/docker/:

• compose.data.yml: PostgreSQL + Redis
• compose.app.yml: frontend + api + worker + nginx
• compose.monitoring.yml: Prometheus + Grafana + Alertmanager + Loki stack

Deploy each compose file to the corresponding host. Update env values to point app host to data host endpoints.

Terraform / Ansible (Optional)

• Terraform templates: infra/terraform/envs/prod
• Ansible playbooks: infra/ansible/playbooks

Typical flow:

# Provision
cd infra/terraform/envs/prod
cp terraform.tfvars.example terraform.tfvars
terraform init
terraform apply

# Bootstrap and deploy
cd ../../ansible
ansible-playbook playbooks/bootstrap.yml
ansible-playbook playbooks/deploy.yml

Detailed deployment notes are in docs/DEPLOYMENT.md.

Monitoring, Logging, and Alerting

Metrics

Prometheus scrapes:

• API metrics: GET /api/metrics
• Worker metrics endpoint (port 9101)
• Redis exporter
• Postgres exporter
• Node exporter
• cAdvisor

Dashboards (Auto-Provisioned)

Grafana dashboards are auto-loaded from infra/grafana/dashboards/:

• PerfScope API Health
• PerfScope Worker Health
• PerfScope Infrastructure Health
• PerfScope Queue and Database Health

Alerts

Alert rules are in infra/prometheus/rules/alerts.yml, including:

• API down
• Worker down
• High API latency (p95)
• Queue backlog high
• Excessive failed jobs
• Sustained CPU high
• Disk usage high

Alertmanager config is in infra/alertmanager/alertmanager.yml.

• Local webhook receiver works out of the box.
• SMTP receiver is available via env variables.

Logs

• API and worker output structured JSON logs.
• Nginx logs are available in container log streams.
• Loki + Promtail pipeline is included for centralized querying in Grafana.

Security Model

Implemented safeguards:

• URL scheme allowlist (http, https only)
• URL normalization and strict validation
• DNS resolution checks on submitted URL and redirects
• Blocks localhost/private/internal/reserved targets by default
• Configurable override with ALLOW_PRIVATE_TARGETS=true
• Redirect and timeout limits
• Redis-backed rate limiting on POST /api/checks
• Environment-based secrets/config (no hardcoded credentials)
• CORS allowlist configuration
• Readiness/liveness endpoints and container healthchecks

Testing

Backend tests

cd apps/api
PYTHONPATH=../common:. pytest tests

Frontend quality checks

cd apps/frontend
npm install
npm run lint
npm run typecheck
npm run build

End-to-end integration (docker-based)

./scripts/integration-test.sh

This script starts PostgreSQL + Redis + API + worker, submits a real URL check, and waits for completion.

Load testing

k6 run scripts/load-test.js

With custom target:

BASE_URL=https://YOUR_DOMAIN k6 run scripts/load-test.js

CI/CD

Workflows:

• ci.yml
  • backend tests
  • frontend lint/typecheck/build
  • docker-based integration path
• docker-build.yml
  • validates image builds for API, worker, frontend
• deploy.yml
  • optional SSH deployment workflow

Operations (Backups, Upgrades, Troubleshooting)

Backups

Minimum recommended:

• PostgreSQL logical backups (daily):

docker compose exec postgres pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" > backup.sql

• Persist and back up Docker volumes (postgres_data, redis_data, prometheus_data, grafana_data, loki_data).

Upgrade Procedure

1. Pull latest code.
2. Review .env changes and release notes.
3. Rebuild and restart:

docker compose -f docker-compose.prod.yml up -d --build

4. Verify health and dashboards.
5. Roll back to previous git revision if required.

Common Troubleshooting

• API not ready: check Postgres/Redis health and API logs.
• Jobs stuck pending: verify worker is running and queue reachable.
• Checks failing unexpectedly: inspect worker logs for network/SSL/DNS errors.
• No metrics in Grafana: verify Prometheus targets are UP.
• No logs in Loki: verify Promtail is reading Docker container log files.

Operational runbook: docs/OPERATIONS.md

Roadmap

• Multi-tenant authentication and quotas
• Scheduled recurring checks
• OpenTelemetry tracing
• HA data stores and backup automation
• Progressive deployment strategy (canary/blue-green)