Skip to content

Add MiniStack CORS proxy and AWS Glue auto-log-group setup - #51

Merged
dmux merged 4 commits into
mainfrom
develop
Jul 24, 2026
Merged

Add MiniStack CORS proxy and AWS Glue auto-log-group setup#51
dmux merged 4 commits into
mainfrom
develop

Conversation

@dmux

@dmux dmux commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes critical CORS issues when using the hosted app with local MiniStack, and adds automatic log group creation for AWS Glue jobs on MiniStack.

Features

1. CORS Proxy for MiniStack Browser Access

  • nginx-based reverse proxy that adds Access-Control-Allow-Origin headers to all MiniStack responses
  • Enables the hosted app (app.openarchflow.cloud) to access local MiniStack on localhost:4567
  • Handles OPTIONS preflight requests with reflected Access-Control-Request-Headers
  • Preserves AWS SigV4 Host headers for signature verification
  • Streams large payloads (S3 uploads, Lambda .zip) without buffering

2. AWS Glue Log Group Auto-Creation

  • Auto-creates CloudWatch Logs groups before running Glue jobs
  • Creates: /aws-glue/jobs/error, /aws-glue/jobs/output, /aws-glue/jobs/audit
  • Prevents "400: The specified log group does not exist" errors
  • Integrated into glueStartJobRun() function

3. Docker Compose Configurations

  • docker-compose.ministack.yml — Basic MiniStack + CORS proxy (4567)
  • docker-compose.ministack-aws-glue-full.yml — MiniStack full + Glue + persistent S3 + CORS proxy
  • Auto-pulls Glue Docker image via glue-image-puller service
  • Uses pull_policy: always for MiniStack image

4. Documentation

  • Complete setup guide for three use cases:
    1. Local OpenArchFlow + basic MiniStack
    2. Local OpenArchFlow + MiniStack with Glue
    3. Hosted app + MiniStack with CORS proxy (± Glue)
  • Troubleshooting sections for CORS and Glue image issues
  • Clear endpoint configuration instructions

Breaking Changes

None. Fully backward compatible:

  • Local users can continue using localhost:4566 directly
  • Hosted app users opt-in to proxy via localhost:4567
  • DEFAULT_MINISTACK_CONFIG.endpoint unchanged
  • Glue log group creation is transparent

Files Changed

  • docker/ministack-cors-proxy/nginx.conf — CORS proxy configuration
  • docker/ministack-cors-proxy/Dockerfile — nginx Alpine image
  • docker-compose.ministack.yml — Updated with glue-image-puller
  • docker-compose.ministack-aws-glue-full.yml — New Glue-enabled compose
  • src/lib/ministack/glue-actions.ts — Auto log group creation
  • README.md — Updated documentation
  • .devcontainer/docker-compose.yml — Docker socket mount for DinD

Testing

CORS Proxy

docker compose -f docker-compose.ministack.yml up -d

# Verify headers
curl -i -H "Origin: https://app.openarchflow.cloud" http://localhost:4567/_ministack/health
# Should see: Access-Control-Allow-Origin: *

# Test OPTIONS preflight
curl -i -X OPTIONS \
  -H "Origin: https://app.openarchflow.cloud" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: authorization,x-amz-date,x-amz-content-sha256" \
  http://localhost:4567/
# Should return 204 with Allow headers

Glue Log Groups

docker compose -f docker-compose.ministack-aws-glue-full.yml up -d

# Create Glue job and run it in OpenArchFlow UI
# Glue job should now execute without 400 log group errors
# CloudWatch Logs groups will be automatically created

Build & Lint

  • ✅ pnpm lint — Zero errors
  • ✅ pnpm build — Successful TypeScript compilation
  • ✅ 56 packages updated to latest versions
  • ✅ All API routes compile correctly

Related Issues

Fixes CORS error: Access to fetch at 'http://localhost:4566/...' from origin 'https://app.openarchflow.cloud' has been blocked by CORS policy

Fixes Glue error: 400: The specified log group does not exist: /aws-glue/jobs/error

dmux added 3 commits July 24, 2026 20:31
@dmux
chore: update dependencies to latest versions
3ba5b71 
Updates 56 packages across dependencies and devDependencies:

Dependencies (31 updated):
- AWS SDK clients: 3.1085.0 → 3.1094.0 (30 packages)
  - All AWS service clients including S3, Lambda, DynamoDB, etc.
- Radix UI components: Various minor patch bumps (10 packages)
  - React alert dialog, dialog, dropdown menu, label, popover, progress, etc.
- Core: next 16.2.10 → 16.2.11, react/react-dom 19.2.7 → 19.2.8

DevDependencies (8 updated):
- TypeScript ESLint: 8.63.0 → 8.65.0
- Autoprefixer: 10.5.2 → 10.5.4
- PostCSS: 8.5.17 → 8.5.22
- ESLint Next.js config: 16.2.10 → 16.2.11

Verified:
- pnpm lint: ✓ PASSED (no errors)
- pnpm build: ✓ PASSED (TypeScript, static generation successful)
- No breaking changes introduced
- All API routes compile correctly

⚠ 3 deprecated subdependencies detected (glob, inflight, sourcemap-codec)
  but they do not impact functionality or build output.
@dmux
feat: auto-create CloudWatch Logs groups for Glue jobs
3dca8b0 
When starting a Glue job, automatically ensure required log groups exist:
- /aws-glue/jobs/error
- /aws-glue/jobs/output
- /aws-glue/jobs/audit

This fixes the '400: The specified log group does not exist' error
that occurs when running Glue jobs on MiniStack.

The ensureGlueLogGroups() function:
- Checks if log group exists via DescribeLogGroupsCommand
- Creates it if missing via CreateLogGroupCommand
- Is called before every glueStartJobRun() invocation

MiniStack (unlike real AWS Glue) requires log groups to be created
upfront before jobs can write logs to them.
@dmux
fix: remove duplicate CORS headers and add Glue config
96f8d67 
Changes:
1. Remove duplicate Access-Control-Allow-Origin header in nginx proxy
   - Was being added in both OPTIONS block and final response
   - Caused browser error: multiple values '*, *' not allowed
   - Now adds header only once in final response

2. Update docker-compose.ministack.yml for glue-image-puller service
   - Automatically pulls AWS Glue image before MiniStack starts
   - Uses service_completed_successfully dependency condition

3. Add docker-compose.ministack-aws-glue-full.yml
   - Comprehensive Glue setup with persistent S3 and Docker integration
   - Includes glue-image-puller service
   - Includes ministack with full image and CORS proxy

4. Update README documentation
   - Clarify log group auto-creation behavior
   - Simplify setup instructions
   - Remove references to manual setup scripts
   - Add Glue troubleshooting section
@vercel

vercel Bot commented Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openarchflow-app Ready Ready Preview, Comment Jul 24, 2026 11:38pm

@dmux
dmux merged commit 055c941 into main Jul 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dmux