Should we port ctf_setup.sh to Python?

[madebygps]

Our setup script is not too complicated, but getting there. We should review if porting to Python makes sense.

1. Evaluate current complexities in our script.
2. Decide if we want to keep it in exchange for not having to depend on Python (bash being more universally installed)
3. If yes: which version, how would we handle .venv usage, could flag generation obscurity be improved.

[madebygps]

Decision: Port ctf_setup.sh to Python using uv

We went through the full script, researched the unclear parts, and answered the open questions in issue #81. The plan is now locked in.

---

1. Current complexities in ctf_setup.sh

The script handles 11 distinct responsibility areas:

• Flag generation and hashing with HMAC-SHA256 per challenge
• Verification token generation
• System setup including packages, SSH config, sysctl, and DNS
• User management for ctf_user, flag_user, and old_admin
• CTF state storage
• The embedded verify CLI, currently 350+ lines of bash inside a heredoc
• MOTD and welcome message
• Setup readiness check script
• 18 individual challenge environments
• Completion certificate generation
• Idempotency marker

The main pain points are the embedded verify heredoc, string escaping complexity, repeated system setup patterns, and the lack of unit-testable setup code.

---

2. Final direction

Decision: Port to Python. Keep bash only as a thin cloud-init bootstrap.

ctf_setup.sh should remain the cloud-init entry point, but only do bootstrap work:

1. Enable strict shell behavior and log capture.
2. Check an idempotency marker.
3. Install uv to /usr/local/bin.
4. Run the Python setup package.
5. Install the Python verify CLI as /usr/local/bin/verify.
6. Clean the uv cache.
7. Write the completion marker after setup succeeds.

Everything else moves to Python.

---

3. Python and uv decisions

Python version: Pin to >=3.13 in both setup/pyproject.toml and verify/pyproject.toml.

Research showed Python 3.13 is already a stable bugfix and maintenance release. Python 3.13.0 final was released on 2024-10-07, regular bugfix releases are still happening, and security support is expected until October 2029.

Using Python 3.13 also gives cleaner ownership handling because shutil.chown() supports follow_symlinks. Recursive ownership changes still require traversal, but Python 3.13 gives us better control over symlink behavior.

No system Python dependency: Use uv to manage Python. The VM should not rely on the Python version installed by Ubuntu.

---

4. Verify CLI decision

The learner-facing verify command should become an installable Python CLI package.

The CLI should be fully Python-native:

• Use argparse or equivalent Python CLI parsing.
• Use Python file and JSON handling for CTF state.
• Use pyfiglet for large certificate text.
• Use rich for colored terminal output.
• Do not shell out to figlet or lolcat.

This keeps the CLI testable and avoids needing extra visual-output binaries on the learner VM.

---

5. Cloud-init idempotency marker

Use this marker instead of /var/log/setup_complete:

/var/lib/cloud/instance/ctf-setup.done

This is suitable across AWS, Azure, and GCP because it is cloud-init's own per-instance state path on Ubuntu cloud images. The current Terraform files use official Ubuntu cloud images on all three providers, and those images use cloud-init.

The marker should only be written at the very end after setup succeeds. If setup fails, the marker should not be written, so a rerun can try again.

---

6. Ubuntu version decision

The repo currently targets Ubuntu 22.04 LTS on all three providers:

Provider	Current image
AWS	ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*
Azure	0001-com-ubuntu-server-jammy, 22_04-lts-gen2
GCP	ubuntu-os-cloud/ubuntu-2204-lts

For issue #81, keep Ubuntu 22.04 as the target so the Python refactor does not also become an operating system upgrade.

A separate issue now tracks the Ubuntu 24.04 LTS migration: #82

---

7. Challenge 9 DNS decision

Challenge 9 should be redesigned so it does not modify /etc/resolv.conf.

Reason: on AWS, Azure, and GCP Ubuntu cloud images, /etc/resolv.conf is cloud-managed DNS plumbing. It is commonly connected to systemd-resolved, usually through a symlink to /run/systemd/resolve/stub-resolv.conf. Removing that symlink and writing a static resolver file is technically possible, but it fights the platform defaults and could make DNS behavior less reliable across providers.

The Python port should:

1. Leave /etc/resolv.conf untouched.
2. Create a harmless DNS-themed challenge file, such as /etc/systemd/resolved.conf.d/ctf-dns.conf.
3. Place the flag in a comment inside that challenge-owned file.
4. Guide learners toward inspecting modern Ubuntu DNS with commands like ls -l /etc/resolv.conf, cat /etc/resolv.conf, resolvectl status, and ls /etc/systemd/resolved.conf.d/.
5. Keep the lesson focused on DNS discovery without changing the VM's live resolver configuration.

---

8. Responsibility categorization

Every responsibility was researched and categorized.

Category	Responsibilities
Pure Python	Flag generation and hashing, verification tokens, CTF state storage, MOTD, setup readiness check, verify CLI, challenges 1, 2, 3, 5, 7, 8, 9, 13, 15, 16, completion marker
Subprocess only	Package installation, user management for ctf_user, flag_user, and old_admin, challenge 18 filesystem commands
Mixed	OS and SSH config, challenges 4, 6, 10, 11, 12, 14, 17

Important researched details:

• User management should stay subprocess because there is no Python stdlib replacement for useradd, chpasswd, and usermod.
• Challenge 18 should stay subprocess because there is no Python stdlib replacement for mkfs.ext4, mount, and umount.
• Recursive ownership changes need a helper. shutil.chown() changes one path only, so the helper must traverse with os.walk() or similar. On Python 3.13, use follow_symlinks deliberately.
• Systemd service handling should be wrapped in helpers for writing unit files, running daemon-reload, enabling services, starting services, and restarting existing services.

---

9. Planned file structure

linux-ctfs/
├── ctf_setup.sh                          # Thin shell bootstrap only
├── verify_token.py                       # Existing contributor utility, unchanged
├── setup/
│   ├── pyproject.toml                    # requires-python = ">=3.13"
│   ├── main.py                           # Orchestrates setup
│   ├── flags.py                          # Flag generation and hashing
│   ├── system.py                         # Packages, SSH config, users, MOTD, hostname
│   ├── state.py                          # Writes /etc/ctf and /var/ctf state
│   ├── helpers.py                        # run(), recursive_chown(), write_service(), enable_service()
│   └── challenges/
│       ├── __init__.py
│       ├── ch01_hidden_file.py
│       ├── ch02_file_search.py
│       ├── ch03_log_analysis.py
│       ├── ch04_user_investigation.py
│       ├── ch05_permissions.py
│       ├── ch06_service_discovery.py
│       ├── ch07_encoding.py
│       ├── ch08_ssh_secrets.py
│       ├── ch09_dns.py
│       ├── ch10_remote_upload.py
│       ├── ch11_web_config.py
│       ├── ch12_network_traffic.py
│       ├── ch13_cron.py
│       ├── ch14_process_env.py
│       ├── ch15_archive.py
│       ├── ch16_symlinks.py
│       ├── ch17_history.py
│       └── ch18_disk_detective.py
└── verify/
    ├── pyproject.toml                    # requires-python = ">=3.13", deps: pyfiglet, rich
    └── src/
        └── verify/
            ├── __init__.py
            ├── __main__.py               # CLI dispatch
            ├── state.py                  # Reads CTF state
            └── commands.py               # progress, list, hint, time, export, flag checks

---

10. Implementation phases

1. Create a feature branch.
2. Rewrite ctf_setup.sh as a thin bootstrap.
3. Scaffold the setup/ and verify/ Python packages with Python 3.13.
4. Implement shared setup helpers, flag generation, state writing, and system setup.
5. Port challenges in batches.
6. Implement the Python verify CLI.
7. Wire setup orchestration in setup/main.py.
8. Update shellcheck and CTF test scripts for the new layout and idempotency marker.
9. Run the full CTF validation flow.

Next step: start implementation on a feature branch.

[madebygps]

As we were testing the Python setup refactor, the Azure deployment appeared to be stuck for 10 plus minutes during null_resource.wait_for_setup. The Terraform resources had been created, SSH connected, and then Terraform kept waiting for setup to finish.

To investigate, we used these CLI commands:

git log --oneline --decorate --all --grep='local setup' --grep='user-data' --grep='GitHub' --grep='download' --grep='gzip' -n 20
git show -s --format='%H%n%s%n%b' 035f310
git show -s --format='%H%n%s%n%b' 4e9668c
git show -s --format='%H%n%s%n%b' 2865253
gh pr list --state all --search 'local setup OR use_local_setup OR gzip OR user-data OR download setup repo:learntocloud/linux-ctfs' --json number,title,url,mergedAt,headRefName --limit 20
gh api -H 'Accept: application/vnd.github+json' /repos/learntocloud/linux-ctfs/commits/035f310/pulls

What we found:

• The normal Terraform setup path was changed to download ctf_setup.sh from GitHub in PR Add 6 new challenges (13-18), systemd services, hints, timer, and cer… #51: Add 6 new challenges (13-18), systemd services, hints, timer, and cer… #51
• The related commit is 035f3104ad7bcbe0fb31a034a1084787b0f10500: 035f310
• That commit says the GitHub download path was added to fix the AWS user data size limit and remove duplicate provider-specific setup scripts.
• Local setup mode was introduced in commit 4e9668c8cf1e12cd696d9e41188747dfa9419559: 4e9668c
• That commit added use_local_setup to all Terraform configs so contributors could test local setup changes before they were merged.
• AWS local setup later needed gzip compression in commit 2865253f4019811a5af635fea9bbe96cae3826e8: 2865253
• That was needed because the old single-file Bash setup script was too large for AWS EC2 user data when base64 encoded.

The refactor changes the problem. Setup is no longer one large Bash file. It is now a small Bash bootstrap plus Python directories. So the VM needs these files together:

ctf_setup.sh
setup/
verify/

That means downloading only ctf_setup.sh from GitHub is no longer enough. It also means the old AWS user data size workaround is no longer the right mental model.

We also looked at whether Terraform should use remote-exec for normal setup. Terraform's own docs say provisioners are a last resort and recommend purpose-built setup methods instead, such as cloud-init, provider-native features, or prebuilt images. The issue with remote-exec is that Terraform has to SSH into the VM and run commands after the VM is created. That makes the deployment more fragile because it depends on SSH timing, network access, credentials, and the VM being fully ready.

Terraform docs: https://developer.hashicorp.com/terraform/language/provisioners

The solution we settled on:

1. Keep a normal release mode for learners and regular users.
2. Keep a contributor mode for testing local, unmerged changes.
3. In release mode, Terraform should not use remote-exec. It should pass a small cloud-init script to the VM.
4. That cloud-init script should download a published GitHub release asset, verify its SHA-256 checksum, unpack it, and run ctf_setup.sh.
5. In contributor mode, Terraform can still upload local ctf_setup.sh, setup/, and verify/, then use remote-exec to run setup. This is acceptable because contributor mode exists specifically for local testing.

For the release artifact, a GitHub Action seems ideal. The action can create a setup tarball from the required files, calculate the SHA-256 checksum, and upload the tarball to a GitHub Release. Then Terraform can point release mode at that versioned asset instead of moving main.

GitHub's docs support this direction. GitHub release assets expose a download URL and a digest value, and GitHub's source archive docs recommend releases when archive stability matters for security.

References:

• GitHub release assets API: https://docs.github.com/en/rest/releases/assets
• GitHub source archive stability: https://docs.github.com/en/repositories/working-with-files/using-files/downloading-source-code-archives
• cloud-init command behavior: https://docs.cloud-init.io/en/latest/reference/modules.html#runcmd

[madebygps]

Follow-up from the release-mode planning pass, avoiding repeating the earlier findings about remote-exec and the old GitHub main download path.

The key new finding is that the release asset approach should solve the cloud setup-size problem only if Terraform keeps first-boot data small. Terraform should not embed the full setup package. It should pass only a small cloud-init or startup script that downloads the versioned setup package on the VM.

Cloud size limits we confirmed:

Provider	First-boot data limit	What that means for us
AWS	EC2 user data is limited to 16 KB raw before base64 encoding	This is the strictest limit, so embedding the full setup package is not safe.
Azure	VM custom data is limited to 64 KB	More room than AWS, but still not enough to treat custom data as a package transport.
GCP	Startup script metadata value is limited to 256 KB, and total metadata is limited to 512 KB	Larger than AWS and Azure, but still better to keep startup data small and consistent across providers.

References:

• AWS EC2 user data: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
• Azure custom data: https://learn.microsoft.com/en-us/azure/virtual-machines/custom-data
• GCP custom metadata limits: https://cloud.google.com/compute/docs/metadata/setting-custom-metadata
• GCP Linux startup scripts: https://cloud.google.com/compute/docs/instances/startup-scripts/linux

Planning decisions from this pass:

1. Release mode should use a pinned release tag, not main, latest, or another moving target.
2. The release should publish both linux-ctfs-setup-<tag>.tar.gz and linux-ctfs-setup-<tag>.tar.gz.sha256.
3. The VM should download the tarball and checksum file, verify the SHA-256 checksum locally, unpack the tarball, then run ctf_setup.sh from the unpacked directory.
4. Contributor mode should keep use_local_setup=true for testing local unmerged changes. That mode can still upload local files and use remote-exec because that is specifically for contributor testing, not the normal learner path.
5. Release mode cannot become fully usable until the first release asset exists. The first release may need to be created through contributor mode, then Terraform defaults can point to that release tag.
6. Changing the setup release tag should be documented as first-boot setup data. Users should expect to recreate the VM rather than expecting setup to rerun automatically on an existing VM.

This keeps normal deployments small enough for AWS, Azure, and GCP, avoids the moving-branch security concern, and limits Terraform SSH provisioning to the one workflow where it is useful: local contributor testing.

[madebygps]

Manual testing got further along after the release-mode and contributor-mode changes. In the latest Azure contributor-mode test, Terraform successfully created the VM, connected over SSH, uploaded the local setup archive, installed uv, installed Python 3.13, and created the setup virtual environment. The failure now happens later, when the Python setup runs apt-get update.

We researched the apt failure and found two related problems:

1. Terraform remote-exec can start as soon as SSH is available, but SSH being available does not always mean the VM has finished first-boot cloud-init work. This can make contributor mode start our setup while Ubuntu/Azure first-boot package tasks are still settling.
2. The exact apt error is coming from Ubuntu's command-not-found apt hook, /usr/lib/cnf-update-db, which runs after apt-get update. If command metadata is missing or temporarily inconsistent, apt-get update can exit with code 100 even though the main package list download mostly worked.

Based on that research, we are implementing two fixes:

1. In contributor mode only, wait for cloud-init to finish before Terraform runs the uploaded local setup script. This directly handles the remote-exec timing issue.
2. In the shared Python setup code, make apt commands use apt's built-in lock timeout and repository retry options. This applies to both release mode and contributor mode because both modes run the same Python setup.

This keeps release mode safe because the release setup runs inside first boot, so it should not wait for cloud-init to finish from inside cloud-init itself. It also keeps the fix clean because we are not disabling Ubuntu apt hooks, deleting apt lock files, or hiding real apt failures.

[madebygps]

Manual testing update for the setup-to-Python work:

We tested contributor mode on Azure using the local setup package. The lab deployed successfully, SSH worked, the challenge setup completed, and all challenge flags were verified manually. verify export also worked as expected after completion.

Issues found during manual testing and what we changed:

1. Challenge 3, large log

   • Problem: The large log was generated from random binary data. Using tail -1 could return invalid bytes along with the flag, which caused verify to crash with a Unicode encoding error.
   • Fix: Changed the challenge to generate a large plain-text log instead. The flag remains on the final line, so the intended tail workflow is safe for learners.

2. Challenge 4, other user profile

   • Problem: /home/flag_user/.profile was world-readable, but the home directory itself was not traversable by ctf_user, so learners got Permission denied.
   • Fix: Set /home/flag_user to 0755 after setup so learners can reach the readable .profile file.

3. Challenge 17, bash history

   • Problem: This had the same pattern as Challenge 4. Learners need to read another user's .bash_history, so the parent home directory must be traversable.
   • Fix: Set /home/old_admin to 0755 after setup.

4. Maintainer testing files

   • Updated the test script where needed, including the Challenge 9 DNS check, the Challenge 12 single-flag extraction, and the reboot progress helper call.

Where we are now:

• Azure contributor mode has been manually tested successfully.
• All challenges were verified manually.
• verify export was verified manually.
• The next meaningful validation step is release mode. To test that properly, these changes need to be merged and included in a GitHub Release asset, because release mode downloads the packaged setup from a release rather than using the local working tree.

[madebygps]

Progress update:

We merged the main Python setup refactor in PR #89:
#89

After that, we created the first release, v0.1.0:
https://github.com/learntocloud/linux-ctfs/releases/tag/v0.1.0

We then tested the release path from a fresh learner-style Azure Terraform deployment. The release package worked, and the setup did complete successfully, but we found an important timing issue: Terraform could finish and print the VM IP before the first-boot CTF setup was fully done. That made it possible for a learner to SSH in too early and see an empty home directory or the default Ubuntu login message before the custom lab setup, MOTD, and challenge files were ready.

We researched the better long-term direction and created issue #90 to track provider-specific readiness signals, such as Azure VM extensions or AWS SSM-based readiness where appropriate:
#90

For the immediate fix, we created PR #91:
#91

That PR adds a cross-provider readiness wait so Terraform waits for the CTF setup marker before returning the VM connection details. It also keeps the learner README files simpler by removing contributor and release-mode details from the provider setup instructions.

[madebygps]

Manual verification update: all challenge solution commands have now been tested successfully against the live lab.\n\nNotes:\n- All 18 challenge flags were captured and verified manually.\n- The setup readiness fix from PR #91 worked during release testing.\n- The MOTD displayed correctly after setup completed.\n- Setup completed in about 1 minute 48 seconds during the latest Azure test.\n- A local SOLUTIONS.md helper file was used for manual verification only and was not committed.