diff --git a/README.md b/README.md index 6fe70064..38fabb4c 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ And all this automatically as a part of your GitHub Actions workflow. > > The bootstrap script that this action injects as EC2 `user-data` is hardcoded to use `yum`, `useradd`, `sudo`, `bash`, and a `tmpfs` `/tmp`. That means the AMI you pass via `ec2-image-id` **must** be a yum-based distribution — Amazon Linux 2023 (the tested baseline), Amazon Linux 2, or a RHEL-family image (RHEL / CentOS Stream / Rocky / Alma) whose `/tmp` is mounted as tmpfs. > -> **Debian, Ubuntu, Alpine, and any other non-yum distributions are not supported.** If you launch this action against such an AMI, the EC2 instance will boot but the runner bootstrap will fail silently inside cloud-init, and the action will eventually time out with a registration error. Cross-distro support is not on the roadmap — if you need it, fork and replace the `userData` array in `src/aws.js`. +> **Debian, Ubuntu, Alpine, and any other non-yum distributions are not supported.** If you launch this action against such an AMI, the EC2 instance will boot but the runner bootstrap will fail. The action now surfaces this quickly: it fails fast naming the failing step and prints the instance's console output (see [Troubleshooting a failed start](#troubleshooting-a-failed-start)). Cross-distro support is not on the roadmap — if you need it, fork and replace the `userData` bootstrap in `src/aws.js`. ![GitHub Actions self-hosted EC2 runner](docs/images/github-actions-runner.gif) @@ -142,7 +142,9 @@ This action reads AWS credentials from the environment. Two paths — pick one. "ec2:TerminateInstances", "ec2:DescribeInstances", "ec2:DescribeInstanceStatus", - "ec2:DescribeImages" + "ec2:DescribeImages", + "ec2:DescribeTags", + "ec2:GetConsoleOutput" ], "Resource": "*" } @@ -150,6 +152,30 @@ This action reads AWS credentials from the environment. Two paths — pick one. } ``` + `ec2:DescribeTags` and `ec2:GetConsoleOutput` power the [bootstrap diagnostics](#troubleshooting-a-failed-start): the action reads the instance's bootstrap phone-home tag to fail fast on cloud-init errors, and captures the serial-console output when a start fails. `ec2:TerminateInstances` also covers the default cleanup of a failed start (see `cleanup-on-start-failure`). + + **Bootstrap phone-home (optional, recommended).** For the instance to tag its own bootstrap progress — which lets the action fail fast and name the failing step instead of waiting out the full registration timeout — the IAM role attached to the runner via `iam-role-name` needs permission to tag itself: + + ``` + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "ec2:CreateTags", + "Resource": "arn:aws:ec2:*::instance/*", + "Condition": { + "StringEquals": { + "aws:ARN": "${ec2:SourceInstanceARN}" + } + } + } + ] + } + ``` + + The condition scopes the permission so an instance can tag only itself. This is best-effort: if you don't set `iam-role-name`, or omit this permission, phone-home tagging is skipped and the action falls back to registration-timeout detection with no error. + If you plan to attach an IAM role to the EC2 runner with the `iam-role-name` parameter, you will need to allow additional permissions: ``` @@ -290,6 +316,7 @@ Now you're ready to go! | `runner-version` | Optional. Used only with the `start` mode. | Version of the `actions/runner` binary to download and register (default `2.335.1`).

Must have a matching entry in `src/runner-checksums.js`; the action verifies the downloaded tarball's SHA-256 against that table before extraction. | | `http-tokens` | Optional. Used only with the `start` mode. | Instance Metadata Service (IMDS) token mode (default `required`).

- `required` — IMDSv2 only; mitigates SSRF-style credential theft.
- `optional` — also allows IMDSv1; set only if a workload on the runner needs it. | | `encrypt-ebs` | Optional. Used only with the `start` mode. | When `true`, the root EBS volume is created with SSE-EBS encryption using the account's default AWS-managed key (default `false`). Volume size / type / IOPS are preserved from the AMI. | +| `cleanup-on-start-failure` | Optional. Used only with the `start` mode. | When `true` (default), a runner that fails to bootstrap or register has its console output captured and is then terminated so the failed start doesn't leak a billing instance. Set `false` to leave the instance running for interactive debugging.

**Behavior change:** older versions left the instance running after a registration timeout; the default is now to terminate it. See [Troubleshooting a failed start](#troubleshooting-a-failed-start). | | `debug` | Optional. | When `true`, the action emits extra diagnostic output to the Actions log — inputs (secrets redacted), AWS SDK response metadata, and runner-registration poll details. Default `false`. | ### Environment variables @@ -383,6 +410,39 @@ In [this discussion](https://github.com/machulav/ec2-github-runner/discussions/1 If you use this action in your workflow, feel free to add your story there as well 🙌 +## Troubleshooting a failed start + +When a runner fails to come up, the `start` step now diagnoses the failure itself instead of silently waiting out the registration timeout. + +### Fast-fail with a named step + +During bootstrap, the EC2 instance tags itself with its current phase in the `ec2-github-runner:bootstrap` tag as it advances through: + +`preparing` → `installing` → `creating-user` → `downloading` → `configuring` → `registered` + +If a phase aborts, the instance writes `failed:` (e.g. `failed:downloading`) and the `start` step fails within one poll interval, naming the step — so you know immediately whether the problem was, say, the `yum install` (`installing`), the runner-tarball download or checksum (`downloading`), or `config.sh` registration (`configuring`), rather than waiting five minutes for a generic timeout. + +This phone-home tagging needs `ec2:CreateTags` on the instance's own IAM role (set via `iam-role-name`) — see the [permissions policy](#2-prepare-the-aws-access-credentials). It is best-effort: without `iam-role-name` or the permission, tagging is skipped and the action falls back to timeout-based detection with no error, and reads the tag with `ec2:DescribeTags`. + +### Console output on failure + +On any failed start — fast-fail **or** registration timeout — the action fetches the instance's serial console output (`ec2:GetConsoleOutput`), and prints the tail (last 200 lines) into a collapsible group in the Actions log. This is the cloud-init/bootstrap log you would previously have had to fetch by hand with `aws ec2 get-console-output --latest`. The GitHub runner registration token is redacted from the captured output. + +### Cleanup on failure + +By default (`cleanup-on-start-failure: true`), the instance is **terminated** after its console output is captured, so a failed start does not leave a billing instance running. + +> **Behavior change:** older versions left the instance running after a registration timeout. If you relied on that (for example, to SSH in and debug), set `cleanup-on-start-failure: false`. The action then leaves the instance running and prints its instance id along with ready-to-paste `get-console-output` and `terminate-instances` commands. + +```yml +- name: Start EC2 runner + uses: machulav/ec2-github-runner@v2 + with: + mode: start + # ... other inputs ... + cleanup-on-start-failure: false # keep the instance for interactive debugging +``` + ## Self-hosted runner security with public repositories > We recommend that you do not use self-hosted runners with public repositories. diff --git a/action.yml b/action.yml index 961c368a..2e273799 100644 --- a/action.yml +++ b/action.yml @@ -103,6 +103,19 @@ inputs: Passed through to RunInstances MetadataOptions.HttpTokens. required: false default: 'required' + cleanup-on-start-failure: + description: >- + Used only with the 'start' mode. When 'true' (default), if the + runner fails to bootstrap or register, the action captures the + instance's console output and then terminates the instance so a + failed start does not leak a billing instance. Set 'false' to + leave the instance running for interactive debugging — the action + prints its instance id and a ready-to-paste 'get-console-output' + command instead of terminating it. + NOTE: 'true' is a behavior change from older versions, which left + the instance running after a registration timeout. + required: false + default: 'true' debug: description: >- When 'true', the action emits extra diagnostic output to the diff --git a/dist/index.js b/dist/index.js index 0655cb54..1058fb14 100644 --- a/dist/index.js +++ b/dist/index.js @@ -104651,6 +104651,8 @@ module.exports = { const { EC2Client, DescribeImagesCommand, + DescribeTagsCommand, + GetConsoleOutputCommand, RunInstancesCommand, TerminateInstancesCommand, AssociateAddressCommand, @@ -104663,6 +104665,16 @@ const { withRetry } = __nccwpck_require__(6759); const { sortByCreationDate } = __nccwpck_require__(5804); const checksums = __nccwpck_require__(2874); +// Instance tag the bootstrap script writes to phone home its progress. +// The start action polls it to fail fast on cloud-init errors instead of +// waiting out the full registration timeout. See buildUserData(). +const BOOTSTRAP_TAG_KEY = 'ec2-github-runner:bootstrap'; + +// Console-output capture caps: keep the printed tail useful but bounded so a +// runaway boot log can't flood the Actions run output. +const CONSOLE_TAIL_LINES = 200; +const CONSOLE_TAIL_BYTES = 64 * 1024; + // EC2Client reads region + credentials from the environment (set by // aws-actions/configure-aws-credentials or by the instance profile on // self-hosted runners). A single shared client is fine — commands are @@ -104748,59 +104760,88 @@ function buildEncryptedRootMapping(image) { }]; } -async function startEc2Instance(label, githubRegistrationToken) { - const client = ec2Client(); - - // Bootstrap design notes (fix-forward after ec2-github-runner#18/#19/#20): - // - // - Hashes for the runner tarball come from src/runner-checksums.js - // (hardcoded table, cross-checked against the release body in CI). - // The earlier `curl -fsSL .sha256` approach died because - // actions/runner doesn't publish per-tarball .sha256 sidecars. - // - // - Dedicated 'runner' user via useradd + sudo -u. The old - // RUNNER_ALLOW_RUNASROOT=1 escape hatch is gone. Runner has its - // own home under /home/runner/ and writes config.sh state there. - // - // - --ephemeral --unattended --disableupdate on config.sh: one-job - // runner, no interactive prompts, no runtime self-update during - // the session. GitHub auto-deregisters ephemeral runners after - // their job, making the removeRunner() API call in gh.js become - // belt-and-braces rather than the primary deregister path. - // - // - set -euo pipefail across both the outer and inner (runner-user) - // shells so ANY failure kills the bootstrap immediately. Made - // failures diagnosable in the Phase 4.b attempt (see #20 for the - // `aws ec2 get-console-output --latest` recipe). - const runnerVersion = config.input.runnerVersion; - const owner = config.githubContext.owner; - const repo = config.githubContext.repo; - const shaX64 = checksums.lookup('x64', runnerVersion); - const shaArm64 = checksums.lookup('arm64', runnerVersion); - if (!shaX64 || !shaArm64) { - throw new Error( - `No SHA-256 entry in src/runner-checksums.js for runner-version ${runnerVersion}. ` + - 'Add the x64 + arm64 hashes from the release body at ' + - `https://github.com/actions/runner/releases/tag/v${runnerVersion}`, - ); - } +// Build the cloud-init user-data bootstrap script. +// +// Design notes (fix-forward after ec2-github-runner#18/#19/#20): +// +// - Hashes for the runner tarball come from src/runner-checksums.js +// (hardcoded table, cross-checked against the release body in CI). +// The earlier `curl -fsSL .sha256` approach died because +// actions/runner doesn't publish per-tarball .sha256 sidecars. +// +// - Dedicated 'runner' user via useradd + sudo -u. The old +// RUNNER_ALLOW_RUNASROOT=1 escape hatch is gone. Runner has its own +// home under /home/runner/ and writes config.sh state there. +// +// - --ephemeral --unattended --disableupdate on config.sh: one-job +// runner, no interactive prompts, no runtime self-update during the +// session. GitHub auto-deregisters ephemeral runners after their job, +// making the removeRunner() API call in gh.js belt-and-braces rather +// than the primary deregister path. +// +// - set -euo pipefail across both the outer and inner (runner-user) +// shells so ANY failure kills the bootstrap immediately. +// +// - Bootstrap diagnostics (#41): each phase writes an instance tag +// (`ec2-github-runner:bootstrap` = preparing → installing → +// creating-user → downloading → configuring → registered) and an ERR +// trap writes `failed:` on abort, so the start action can fail +// fast and name the failing step instead of waiting out the full +// registration timeout. Tagging is best-effort: it needs +// `ec2:CreateTags` on the instance profile and degrades to +// timeout-only detection when absent (every write is `|| true`). +function buildUserData({ runnerVersion, owner, repo, label, githubRegistrationToken, shaX64, shaArm64 }) { + // Shared shell helpers, emitted verbatim into both the outer (root) and + // inner (runner-user) shells — each shell re-derives instance identity + // from IMDS so it can keep phoning home independently. + const phoneHomeHelpers = [ + 'gh_runner_imds() {', + ' local token', + ' token=$(curl -fsS -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 120" 2>/dev/null || true)', + ' curl -fsS -H "X-aws-ec2-metadata-token: $token" "http://169.254.169.254/latest/meta-data/$1" 2>/dev/null || true', + '}', + 'GH_RUNNER_IID=$(gh_runner_imds instance-id)', + 'GH_RUNNER_REGION=$(gh_runner_imds placement/region)', + 'gh_runner_phone_home() {', + ' [ -n "$GH_RUNNER_IID" ] && [ -n "$GH_RUNNER_REGION" ] || return 0', + ` aws ec2 create-tags --region "$GH_RUNNER_REGION" --resources "$GH_RUNNER_IID" --tags "Key=${BOOTSTRAP_TAG_KEY},Value=$1" >/dev/null 2>&1 || true`, + '}', + ]; - const userData = [ + return [ '#!/bin/bash', 'set -euo pipefail', '', + '# --- ec2-github-runner: bootstrap diagnostics (phone-home) ----------', + ...phoneHomeHelpers, + 'GH_RUNNER_STEP=preparing', + "trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR", + '', '# Root-required setup.', + 'GH_RUNNER_STEP=preparing', + 'gh_runner_phone_home preparing', 'mount -o remount,size=1G /tmp', + 'GH_RUNNER_STEP=installing', + 'gh_runner_phone_home installing', 'yum install -y libicu make sudo', '', '# Create the non-root runner user (idempotent).', + 'GH_RUNNER_STEP=creating-user', + 'gh_runner_phone_home creating-user', 'if ! id runner >/dev/null 2>&1; then', ' useradd -m -s /bin/bash runner', 'fi', '', - '# Drop to the runner user for download + configure + run.', + '# The runner-user shell owns the download/configure/register phases and', + '# reports them itself; drop the outer ERR trap so it does not overwrite', + '# the inner shell\'s more specific failed: tag.', + 'trap - ERR', "sudo -u runner -H bash <<'RUNNER_BOOTSTRAP'", 'set -euo pipefail', + ...phoneHomeHelpers, + 'GH_RUNNER_STEP=downloading', + "trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR", + 'gh_runner_phone_home downloading', 'cd "$HOME"', 'mkdir -p actions-runner && cd actions-runner', '', @@ -104829,11 +104870,34 @@ async function startEc2Instance(label, githubRegistrationToken) { 'tar xzf "$TARBALL"', '', 'export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1', + 'GH_RUNNER_STEP=configuring', + 'gh_runner_phone_home configuring', `./config.sh --url "https://github.com/${owner}/${repo}" --token "${githubRegistrationToken}" --labels "${label}" --ephemeral --unattended --disableupdate`, + 'GH_RUNNER_STEP=registered', + 'gh_runner_phone_home registered', './run.sh', 'RUNNER_BOOTSTRAP', '', - ]; + ].join('\n'); +} + +async function startEc2Instance(label, githubRegistrationToken) { + const client = ec2Client(); + + const runnerVersion = config.input.runnerVersion; + const owner = config.githubContext.owner; + const repo = config.githubContext.repo; + const shaX64 = checksums.lookup('x64', runnerVersion); + const shaArm64 = checksums.lookup('arm64', runnerVersion); + if (!shaX64 || !shaArm64) { + throw new Error( + `No SHA-256 entry in src/runner-checksums.js for runner-version ${runnerVersion}. ` + + 'Add the x64 + arm64 hashes from the release body at ' + + `https://github.com/actions/runner/releases/tag/v${runnerVersion}`, + ); + } + + const userData = buildUserData({ runnerVersion, owner, repo, label, githubRegistrationToken, shaX64, shaArm64 }); const resolved = await resolveImage(client); config.input.ec2ImageId = resolved.id; @@ -104843,7 +104907,7 @@ async function startEc2Instance(label, githubRegistrationToken) { InstanceType: config.input.ec2InstanceType, MinCount: 1, MaxCount: 1, - UserData: Buffer.from(userData.join('\n')).toString('base64'), + UserData: Buffer.from(userData).toString('base64'), SubnetId: config.input.subnetId, SecurityGroupIds: [config.input.securityGroupId], IamInstanceProfile: { Name: config.input.iamRoleName }, @@ -104912,32 +104976,141 @@ async function startEc2Instance(label, githubRegistrationToken) { return ec2InstanceId; } -async function terminateEc2Instance() { +async function terminateInstanceById(ec2InstanceId) { const client = ec2Client(); const start = Date.now(); - log.info('terminate_instance', { instance_id: config.input.ec2InstanceId }); + log.info('terminate_instance', { instance_id: ec2InstanceId }); try { await withRetry('terminate_instance', () => client.send(new TerminateInstancesCommand({ - InstanceIds: [config.input.ec2InstanceId], + InstanceIds: [ec2InstanceId], })), ); - log.info('terminate_instance', { instance_id: config.input.ec2InstanceId, elapsed_ms: Date.now() - start }); - core.info(`AWS EC2 instance ${config.input.ec2InstanceId} is terminated`); + log.info('terminate_instance', { instance_id: ec2InstanceId, elapsed_ms: Date.now() - start }); + core.info(`AWS EC2 instance ${ec2InstanceId} is terminated`); } catch (error) { - log.error('terminate_instance', { instance_id: config.input.ec2InstanceId, error: error.name, message: error.message }); - core.error(`AWS EC2 instance ${config.input.ec2InstanceId} termination error`); + log.error('terminate_instance', { instance_id: ec2InstanceId, error: error.name, message: error.message }); + core.error(`AWS EC2 instance ${ec2InstanceId} termination error`); throw error; } } +async function terminateEc2Instance() { + return terminateInstanceById(config.input.ec2InstanceId); +} + +// Read the instance's bootstrap phone-home tag. Returns the tag value +// (e.g. 'downloading', 'failed:configuring') or null when the tag is not +// yet set. Missing ec2:DescribeTags permission (or a transient API error) +// degrades to null so the caller falls back to timeout-based detection +// rather than surfacing a spurious error. +async function getBootstrapStatus(ec2InstanceId) { + const client = ec2Client(); + try { + const resp = await client.send(new DescribeTagsCommand({ + Filters: [ + { Name: 'resource-id', Values: [ec2InstanceId] }, + { Name: 'key', Values: [BOOTSTRAP_TAG_KEY] }, + ], + })); + const tag = (resp.Tags || []).find((t) => t.Key === BOOTSTRAP_TAG_KEY); + return tag ? tag.Value : null; + } catch (error) { + log.debug('bootstrap_status', { instance_id: ec2InstanceId, error: error.name, message: error.message }); + return null; + } +} + +// Replace every occurrence of each secret value with '***'. Uses literal +// (non-regex) replacement so tokens containing regex metacharacters are +// still fully scrubbed. +function redactSecrets(text, secrets) { + let out = text; + for (const secret of secrets || []) { + if (secret) { + out = out.split(secret).join('***'); + } + } + return out; +} + +// Fetch the instance's serial-console output, decode it, and return the +// last CONSOLE_TAIL_LINES lines capped at CONSOLE_TAIL_BYTES, with any +// provided secret values redacted. Returns '' when no output is available +// yet or the call fails (best-effort diagnostics must never mask the +// original error). +async function getConsoleOutputTail(ec2InstanceId, opts = {}) { + const maxLines = opts.maxLines ?? CONSOLE_TAIL_LINES; + const maxBytes = opts.maxBytes ?? CONSOLE_TAIL_BYTES; + const client = ec2Client(); + + let output; + try { + const resp = await client.send(new GetConsoleOutputCommand({ InstanceId: ec2InstanceId, Latest: true })); + if (!resp || !resp.Output) { + return ''; + } + // EC2 returns the console output base64-encoded. + output = Buffer.from(resp.Output, 'base64').toString('utf8'); + } catch (error) { + log.debug('console_output', { instance_id: ec2InstanceId, error: error.name, message: error.message }); + return ''; + } + + let lines = output.split('\n'); + if (lines.length > maxLines) { + lines = lines.slice(-maxLines); + } + let tail = lines.join('\n'); + if (Buffer.byteLength(tail, 'utf8') > maxBytes) { + tail = tail.slice(-maxBytes); + } + return redactSecrets(tail, opts.redactValues); +} + +// Handle a failed start: capture and print the instance's console output +// (collapsible group, secrets redacted), then either terminate the +// instance (default, so failed starts don't leak billing) or preserve it +// for interactive debugging when cleanup-on-start-failure is 'false'. +// Order is capture-then-terminate so the diagnostics survive the cleanup. +async function handleStartFailure(ec2InstanceId, opts = {}) { + const tail = await getConsoleOutputTail(ec2InstanceId, { redactValues: opts.redactValues }); + core.startGroup(`EC2 instance ${ec2InstanceId} console output (last ${CONSOLE_TAIL_LINES} lines)`); + core.info(tail || '(no console output was available yet — the instance may have failed before cloud-init produced output)'); + core.endGroup(); + + if (config.input.cleanupOnStartFailure === 'true') { + log.info('cleanup_on_start_failure', { instance_id: ec2InstanceId, action: 'terminate' }); + try { + await terminateInstanceById(ec2InstanceId); + } catch (error) { + log.error('cleanup_on_start_failure', { instance_id: ec2InstanceId, error: error.name, message: error.message }); + core.warning(`Could not terminate failed instance ${ec2InstanceId}: ${error.message}. Terminate it manually to avoid charges.`); + } + } else { + log.info('cleanup_on_start_failure', { instance_id: ec2InstanceId, action: 'preserve' }); + core.warning( + `Instance ${ec2InstanceId} was left running for debugging (cleanup-on-start-failure: false).\n` + + `Inspect it with:\n aws ec2 get-console-output --latest --instance-id ${ec2InstanceId}\n` + + `Terminate it when done:\n aws ec2 terminate-instances --instance-ids ${ec2InstanceId}`, + ); + } +} + module.exports = { startEc2Instance, terminateEc2Instance, + terminateInstanceById, waitForInstanceRunning, + getBootstrapStatus, + getConsoleOutputTail, + handleStartFailure, // Exported for unit testing. buildEncryptedRootMapping, + buildUserData, + redactSecrets, + BOOTSTRAP_TAG_KEY, }; @@ -104967,6 +105140,7 @@ class Config { runnerVersion: core.getInput('runner-version') || '2.335.1', httpTokens: core.getInput('http-tokens') || 'required', encryptEbs: core.getInput('encrypt-ebs') || 'false', + cleanupOnStartFailure: core.getInput('cleanup-on-start-failure') || 'true', debug: core.getInput('debug') || 'false', }; @@ -105095,45 +105269,19 @@ async function removeRunner() { } } -async function waitForRunnerRegistered(label) { - const timeoutMinutes = 5; - const retryIntervalSeconds = 10; - const quietPeriodSeconds = 30; - let waitSeconds = 0; - - core.info(`Waiting ${quietPeriodSeconds}s for the AWS EC2 instance to be registered in GitHub as a new self-hosted runner`); - await new Promise(r => setTimeout(r, quietPeriodSeconds * 1000)); - core.info(`Checking every ${retryIntervalSeconds}s if the GitHub self-hosted runner is registered`); - - return new Promise((resolve, reject) => { - const interval = setInterval(async () => { - const runner = await getRunner(label); - log.debug('wait_for_runner_poll', { label, elapsed_s: waitSeconds, found: !!runner, status: runner ? runner.status : null }); - - if (waitSeconds > timeoutMinutes * 60) { - log.error('wait_for_runner', { label, timeout_minutes: timeoutMinutes }); - core.error('GitHub self-hosted runner registration error'); - clearInterval(interval); - reject(`A timeout of ${timeoutMinutes} minutes is exceeded. Your AWS EC2 instance was not able to register itself in GitHub as a new self-hosted runner.`); - } - - if (runner && runner.status === 'online') { - log.info('wait_for_runner', { label, runner_id: runner.id, elapsed_s: waitSeconds }); - core.info(`GitHub self-hosted runner ${runner.name} is registered and ready to use`); - clearInterval(interval); - resolve(); - } else { - waitSeconds += retryIntervalSeconds; - core.info('Checking...'); - } - }, retryIntervalSeconds * 1000); - }); +// True once the runner for `label` has registered with GitHub and reports +// as online. Used as the success signal by the start action's wait loop +// (see src/wait.js), polled alongside the instance's bootstrap tag. +async function isRunnerOnline(label) { + const runner = await getRunner(label); + log.debug('runner_status', { label, found: !!runner, status: runner ? runner.status : null }); + return !!(runner && runner.status === 'online'); } module.exports = { getRegistrationToken, removeRunner, - waitForRunnerRegistered, + isRunnerOnline, }; @@ -105363,6 +105511,92 @@ module.exports = { } +/***/ }), + +/***/ 8644: +/***/ ((module, __unused_webpack_exports, __nccwpck_require__) => { + +const core = __nccwpck_require__(7484); +const log = __nccwpck_require__(7223); + +// Terminal bootstrap phone-home states carry a `failed:` value; see +// src/aws.js buildUserData() for the producing side. +const FAILED_PREFIX = 'failed:'; + +// Wait for the EC2 runner to come online, watching two signals in lockstep: +// +// 1. The instance's `ec2-github-runner:bootstrap` phone-home tag. A +// `failed:` value means cloud-init aborted — fail fast within one +// poll interval, naming the step, instead of waiting out the full +// registration timeout. +// 2. GitHub runner registration. Success is authoritative here: the runner +// showing up as `online` is what lets downstream jobs run. +// +// All I/O and timing are injected so the loop is unit-testable without real +// AWS/GitHub calls or wall-clock sleeps: +// - deps.getBootstrapStatus(): Promise +// - deps.isRunnerOnline(): Promise +// - deps.sleep(ms): Promise (defaults to setTimeout) +// +// Throws on `failed:` (err.bootstrapStep set) or on timeout +// (err.timedOut set) so the caller can capture diagnostics and clean up. +async function waitForRunnerReady(deps, opts = {}) { + const { getBootstrapStatus, isRunnerOnline } = deps; + const sleep = deps.sleep || ((ms) => new Promise((resolve) => setTimeout(resolve, ms))); + + const timeoutMinutes = opts.timeoutMinutes ?? 5; + const intervalSeconds = opts.intervalSeconds ?? 10; + const quietSeconds = opts.quietSeconds ?? 30; + + core.info(`Waiting ${quietSeconds}s for the AWS EC2 instance to bootstrap and register as a self-hosted runner`); + await sleep(quietSeconds * 1000); + core.info(`Checking every ${intervalSeconds}s for bootstrap failures and runner registration`); + + const deadlineSeconds = timeoutMinutes * 60; + let waitedSeconds = 0; + + for (;;) { + // 1. Fast-fail on a phoned-home bootstrap failure. + const status = await getBootstrapStatus(); + if (typeof status === 'string' && status.startsWith(FAILED_PREFIX)) { + const step = status.slice(FAILED_PREFIX.length); + log.error('wait_for_runner', { outcome: 'bootstrap_failed', step }); + core.error(`EC2 runner bootstrap failed during the "${step}" step`); + const error = new Error(`EC2 runner bootstrap failed during the "${step}" step. See the captured console output below.`); + error.bootstrapStep = step; + throw error; + } + + // 2. Success when the runner has registered and is online. + if (await isRunnerOnline()) { + log.info('wait_for_runner', { outcome: 'online', elapsed_s: waitedSeconds }); + core.info('GitHub self-hosted runner is registered and ready to use'); + return; + } + + // 3. Bounded wait — give up after the timeout. + if (waitedSeconds >= deadlineSeconds) { + log.error('wait_for_runner', { outcome: 'timeout', timeout_minutes: timeoutMinutes }); + const error = new Error( + `A timeout of ${timeoutMinutes} minutes is exceeded. Your AWS EC2 instance was not able to register itself in GitHub as a new self-hosted runner.`, + ); + error.timedOut = true; + throw error; + } + + log.debug('wait_for_runner_poll', { elapsed_s: waitedSeconds, bootstrap: status || null }); + core.info('Checking...'); + await sleep(intervalSeconds * 1000); + waitedSeconds += intervalSeconds; + } +} + +module.exports = { + waitForRunnerReady, + FAILED_PREFIX, +}; + + /***/ }), /***/ 2613: @@ -110210,6 +110444,7 @@ const aws = __nccwpck_require__(3776); const gh = __nccwpck_require__(5934); const config = __nccwpck_require__(1283); const log = __nccwpck_require__(7223); +const { waitForRunnerReady } = __nccwpck_require__(8644); const core = __nccwpck_require__(7484); // Write directly to the $GITHUB_OUTPUT file. The bundled @actions/core @@ -110235,7 +110470,21 @@ async function start() { const ec2InstanceId = await aws.startEc2Instance(label, githubRegistrationToken); setOutput(label, ec2InstanceId); await aws.waitForInstanceRunning(ec2InstanceId); - await gh.waitForRunnerRegistered(label); + + // Watch the bootstrap phone-home tag and GitHub registration together. + // On a bootstrap failure or registration timeout, capture the console + // output and (by default) terminate the instance so failed starts + // don't leak billing. The registration token is redacted from the + // captured output. + try { + await waitForRunnerReady({ + getBootstrapStatus: () => aws.getBootstrapStatus(ec2InstanceId), + isRunnerOnline: () => gh.isRunnerOnline(label), + }); + } catch (waitError) { + await aws.handleStartFailure(ec2InstanceId, { redactValues: [githubRegistrationToken] }); + throw waitError; + } log.info('start', { label, instance_id: ec2InstanceId, outcome: 'registered' }); } finally { core.endGroup(); diff --git a/src/aws.js b/src/aws.js index 3b8e821d..9f50721f 100644 --- a/src/aws.js +++ b/src/aws.js @@ -1,6 +1,8 @@ const { EC2Client, DescribeImagesCommand, + DescribeTagsCommand, + GetConsoleOutputCommand, RunInstancesCommand, TerminateInstancesCommand, AssociateAddressCommand, @@ -13,6 +15,16 @@ const { withRetry } = require('./retry'); const { sortByCreationDate } = require('./utils'); const checksums = require('./runner-checksums'); +// Instance tag the bootstrap script writes to phone home its progress. +// The start action polls it to fail fast on cloud-init errors instead of +// waiting out the full registration timeout. See buildUserData(). +const BOOTSTRAP_TAG_KEY = 'ec2-github-runner:bootstrap'; + +// Console-output capture caps: keep the printed tail useful but bounded so a +// runaway boot log can't flood the Actions run output. +const CONSOLE_TAIL_LINES = 200; +const CONSOLE_TAIL_BYTES = 64 * 1024; + // EC2Client reads region + credentials from the environment (set by // aws-actions/configure-aws-credentials or by the instance profile on // self-hosted runners). A single shared client is fine — commands are @@ -98,59 +110,88 @@ function buildEncryptedRootMapping(image) { }]; } -async function startEc2Instance(label, githubRegistrationToken) { - const client = ec2Client(); - - // Bootstrap design notes (fix-forward after ec2-github-runner#18/#19/#20): - // - // - Hashes for the runner tarball come from src/runner-checksums.js - // (hardcoded table, cross-checked against the release body in CI). - // The earlier `curl -fsSL .sha256` approach died because - // actions/runner doesn't publish per-tarball .sha256 sidecars. - // - // - Dedicated 'runner' user via useradd + sudo -u. The old - // RUNNER_ALLOW_RUNASROOT=1 escape hatch is gone. Runner has its - // own home under /home/runner/ and writes config.sh state there. - // - // - --ephemeral --unattended --disableupdate on config.sh: one-job - // runner, no interactive prompts, no runtime self-update during - // the session. GitHub auto-deregisters ephemeral runners after - // their job, making the removeRunner() API call in gh.js become - // belt-and-braces rather than the primary deregister path. - // - // - set -euo pipefail across both the outer and inner (runner-user) - // shells so ANY failure kills the bootstrap immediately. Made - // failures diagnosable in the Phase 4.b attempt (see #20 for the - // `aws ec2 get-console-output --latest` recipe). - const runnerVersion = config.input.runnerVersion; - const owner = config.githubContext.owner; - const repo = config.githubContext.repo; - const shaX64 = checksums.lookup('x64', runnerVersion); - const shaArm64 = checksums.lookup('arm64', runnerVersion); - if (!shaX64 || !shaArm64) { - throw new Error( - `No SHA-256 entry in src/runner-checksums.js for runner-version ${runnerVersion}. ` + - 'Add the x64 + arm64 hashes from the release body at ' + - `https://github.com/actions/runner/releases/tag/v${runnerVersion}`, - ); - } +// Build the cloud-init user-data bootstrap script. +// +// Design notes (fix-forward after ec2-github-runner#18/#19/#20): +// +// - Hashes for the runner tarball come from src/runner-checksums.js +// (hardcoded table, cross-checked against the release body in CI). +// The earlier `curl -fsSL .sha256` approach died because +// actions/runner doesn't publish per-tarball .sha256 sidecars. +// +// - Dedicated 'runner' user via useradd + sudo -u. The old +// RUNNER_ALLOW_RUNASROOT=1 escape hatch is gone. Runner has its own +// home under /home/runner/ and writes config.sh state there. +// +// - --ephemeral --unattended --disableupdate on config.sh: one-job +// runner, no interactive prompts, no runtime self-update during the +// session. GitHub auto-deregisters ephemeral runners after their job, +// making the removeRunner() API call in gh.js belt-and-braces rather +// than the primary deregister path. +// +// - set -euo pipefail across both the outer and inner (runner-user) +// shells so ANY failure kills the bootstrap immediately. +// +// - Bootstrap diagnostics (#41): each phase writes an instance tag +// (`ec2-github-runner:bootstrap` = preparing → installing → +// creating-user → downloading → configuring → registered) and an ERR +// trap writes `failed:` on abort, so the start action can fail +// fast and name the failing step instead of waiting out the full +// registration timeout. Tagging is best-effort: it needs +// `ec2:CreateTags` on the instance profile and degrades to +// timeout-only detection when absent (every write is `|| true`). +function buildUserData({ runnerVersion, owner, repo, label, githubRegistrationToken, shaX64, shaArm64 }) { + // Shared shell helpers, emitted verbatim into both the outer (root) and + // inner (runner-user) shells — each shell re-derives instance identity + // from IMDS so it can keep phoning home independently. + const phoneHomeHelpers = [ + 'gh_runner_imds() {', + ' local token', + ' token=$(curl -fsS -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 120" 2>/dev/null || true)', + ' curl -fsS -H "X-aws-ec2-metadata-token: $token" "http://169.254.169.254/latest/meta-data/$1" 2>/dev/null || true', + '}', + 'GH_RUNNER_IID=$(gh_runner_imds instance-id)', + 'GH_RUNNER_REGION=$(gh_runner_imds placement/region)', + 'gh_runner_phone_home() {', + ' [ -n "$GH_RUNNER_IID" ] && [ -n "$GH_RUNNER_REGION" ] || return 0', + ` aws ec2 create-tags --region "$GH_RUNNER_REGION" --resources "$GH_RUNNER_IID" --tags "Key=${BOOTSTRAP_TAG_KEY},Value=$1" >/dev/null 2>&1 || true`, + '}', + ]; - const userData = [ + return [ '#!/bin/bash', 'set -euo pipefail', '', + '# --- ec2-github-runner: bootstrap diagnostics (phone-home) ----------', + ...phoneHomeHelpers, + 'GH_RUNNER_STEP=preparing', + "trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR", + '', '# Root-required setup.', + 'GH_RUNNER_STEP=preparing', + 'gh_runner_phone_home preparing', 'mount -o remount,size=1G /tmp', + 'GH_RUNNER_STEP=installing', + 'gh_runner_phone_home installing', 'yum install -y libicu make sudo', '', '# Create the non-root runner user (idempotent).', + 'GH_RUNNER_STEP=creating-user', + 'gh_runner_phone_home creating-user', 'if ! id runner >/dev/null 2>&1; then', ' useradd -m -s /bin/bash runner', 'fi', '', - '# Drop to the runner user for download + configure + run.', + '# The runner-user shell owns the download/configure/register phases and', + '# reports them itself; drop the outer ERR trap so it does not overwrite', + '# the inner shell\'s more specific failed: tag.', + 'trap - ERR', "sudo -u runner -H bash <<'RUNNER_BOOTSTRAP'", 'set -euo pipefail', + ...phoneHomeHelpers, + 'GH_RUNNER_STEP=downloading', + "trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR", + 'gh_runner_phone_home downloading', 'cd "$HOME"', 'mkdir -p actions-runner && cd actions-runner', '', @@ -179,11 +220,34 @@ async function startEc2Instance(label, githubRegistrationToken) { 'tar xzf "$TARBALL"', '', 'export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1', + 'GH_RUNNER_STEP=configuring', + 'gh_runner_phone_home configuring', `./config.sh --url "https://github.com/${owner}/${repo}" --token "${githubRegistrationToken}" --labels "${label}" --ephemeral --unattended --disableupdate`, + 'GH_RUNNER_STEP=registered', + 'gh_runner_phone_home registered', './run.sh', 'RUNNER_BOOTSTRAP', '', - ]; + ].join('\n'); +} + +async function startEc2Instance(label, githubRegistrationToken) { + const client = ec2Client(); + + const runnerVersion = config.input.runnerVersion; + const owner = config.githubContext.owner; + const repo = config.githubContext.repo; + const shaX64 = checksums.lookup('x64', runnerVersion); + const shaArm64 = checksums.lookup('arm64', runnerVersion); + if (!shaX64 || !shaArm64) { + throw new Error( + `No SHA-256 entry in src/runner-checksums.js for runner-version ${runnerVersion}. ` + + 'Add the x64 + arm64 hashes from the release body at ' + + `https://github.com/actions/runner/releases/tag/v${runnerVersion}`, + ); + } + + const userData = buildUserData({ runnerVersion, owner, repo, label, githubRegistrationToken, shaX64, shaArm64 }); const resolved = await resolveImage(client); config.input.ec2ImageId = resolved.id; @@ -193,7 +257,7 @@ async function startEc2Instance(label, githubRegistrationToken) { InstanceType: config.input.ec2InstanceType, MinCount: 1, MaxCount: 1, - UserData: Buffer.from(userData.join('\n')).toString('base64'), + UserData: Buffer.from(userData).toString('base64'), SubnetId: config.input.subnetId, SecurityGroupIds: [config.input.securityGroupId], IamInstanceProfile: { Name: config.input.iamRoleName }, @@ -262,30 +326,139 @@ async function startEc2Instance(label, githubRegistrationToken) { return ec2InstanceId; } -async function terminateEc2Instance() { +async function terminateInstanceById(ec2InstanceId) { const client = ec2Client(); const start = Date.now(); - log.info('terminate_instance', { instance_id: config.input.ec2InstanceId }); + log.info('terminate_instance', { instance_id: ec2InstanceId }); try { await withRetry('terminate_instance', () => client.send(new TerminateInstancesCommand({ - InstanceIds: [config.input.ec2InstanceId], + InstanceIds: [ec2InstanceId], })), ); - log.info('terminate_instance', { instance_id: config.input.ec2InstanceId, elapsed_ms: Date.now() - start }); - core.info(`AWS EC2 instance ${config.input.ec2InstanceId} is terminated`); + log.info('terminate_instance', { instance_id: ec2InstanceId, elapsed_ms: Date.now() - start }); + core.info(`AWS EC2 instance ${ec2InstanceId} is terminated`); } catch (error) { - log.error('terminate_instance', { instance_id: config.input.ec2InstanceId, error: error.name, message: error.message }); - core.error(`AWS EC2 instance ${config.input.ec2InstanceId} termination error`); + log.error('terminate_instance', { instance_id: ec2InstanceId, error: error.name, message: error.message }); + core.error(`AWS EC2 instance ${ec2InstanceId} termination error`); throw error; } } +async function terminateEc2Instance() { + return terminateInstanceById(config.input.ec2InstanceId); +} + +// Read the instance's bootstrap phone-home tag. Returns the tag value +// (e.g. 'downloading', 'failed:configuring') or null when the tag is not +// yet set. Missing ec2:DescribeTags permission (or a transient API error) +// degrades to null so the caller falls back to timeout-based detection +// rather than surfacing a spurious error. +async function getBootstrapStatus(ec2InstanceId) { + const client = ec2Client(); + try { + const resp = await client.send(new DescribeTagsCommand({ + Filters: [ + { Name: 'resource-id', Values: [ec2InstanceId] }, + { Name: 'key', Values: [BOOTSTRAP_TAG_KEY] }, + ], + })); + const tag = (resp.Tags || []).find((t) => t.Key === BOOTSTRAP_TAG_KEY); + return tag ? tag.Value : null; + } catch (error) { + log.debug('bootstrap_status', { instance_id: ec2InstanceId, error: error.name, message: error.message }); + return null; + } +} + +// Replace every occurrence of each secret value with '***'. Uses literal +// (non-regex) replacement so tokens containing regex metacharacters are +// still fully scrubbed. +function redactSecrets(text, secrets) { + let out = text; + for (const secret of secrets || []) { + if (secret) { + out = out.split(secret).join('***'); + } + } + return out; +} + +// Fetch the instance's serial-console output, decode it, and return the +// last CONSOLE_TAIL_LINES lines capped at CONSOLE_TAIL_BYTES, with any +// provided secret values redacted. Returns '' when no output is available +// yet or the call fails (best-effort diagnostics must never mask the +// original error). +async function getConsoleOutputTail(ec2InstanceId, opts = {}) { + const maxLines = opts.maxLines ?? CONSOLE_TAIL_LINES; + const maxBytes = opts.maxBytes ?? CONSOLE_TAIL_BYTES; + const client = ec2Client(); + + let output; + try { + const resp = await client.send(new GetConsoleOutputCommand({ InstanceId: ec2InstanceId, Latest: true })); + if (!resp || !resp.Output) { + return ''; + } + // EC2 returns the console output base64-encoded. + output = Buffer.from(resp.Output, 'base64').toString('utf8'); + } catch (error) { + log.debug('console_output', { instance_id: ec2InstanceId, error: error.name, message: error.message }); + return ''; + } + + let lines = output.split('\n'); + if (lines.length > maxLines) { + lines = lines.slice(-maxLines); + } + let tail = lines.join('\n'); + if (Buffer.byteLength(tail, 'utf8') > maxBytes) { + tail = tail.slice(-maxBytes); + } + return redactSecrets(tail, opts.redactValues); +} + +// Handle a failed start: capture and print the instance's console output +// (collapsible group, secrets redacted), then either terminate the +// instance (default, so failed starts don't leak billing) or preserve it +// for interactive debugging when cleanup-on-start-failure is 'false'. +// Order is capture-then-terminate so the diagnostics survive the cleanup. +async function handleStartFailure(ec2InstanceId, opts = {}) { + const tail = await getConsoleOutputTail(ec2InstanceId, { redactValues: opts.redactValues }); + core.startGroup(`EC2 instance ${ec2InstanceId} console output (last ${CONSOLE_TAIL_LINES} lines)`); + core.info(tail || '(no console output was available yet — the instance may have failed before cloud-init produced output)'); + core.endGroup(); + + if (config.input.cleanupOnStartFailure === 'true') { + log.info('cleanup_on_start_failure', { instance_id: ec2InstanceId, action: 'terminate' }); + try { + await terminateInstanceById(ec2InstanceId); + } catch (error) { + log.error('cleanup_on_start_failure', { instance_id: ec2InstanceId, error: error.name, message: error.message }); + core.warning(`Could not terminate failed instance ${ec2InstanceId}: ${error.message}. Terminate it manually to avoid charges.`); + } + } else { + log.info('cleanup_on_start_failure', { instance_id: ec2InstanceId, action: 'preserve' }); + core.warning( + `Instance ${ec2InstanceId} was left running for debugging (cleanup-on-start-failure: false).\n` + + `Inspect it with:\n aws ec2 get-console-output --latest --instance-id ${ec2InstanceId}\n` + + `Terminate it when done:\n aws ec2 terminate-instances --instance-ids ${ec2InstanceId}`, + ); + } +} + module.exports = { startEc2Instance, terminateEc2Instance, + terminateInstanceById, waitForInstanceRunning, + getBootstrapStatus, + getConsoleOutputTail, + handleStartFailure, // Exported for unit testing. buildEncryptedRootMapping, + buildUserData, + redactSecrets, + BOOTSTRAP_TAG_KEY, }; diff --git a/src/config.js b/src/config.js index 68634de7..d50db57b 100644 --- a/src/config.js +++ b/src/config.js @@ -19,6 +19,7 @@ class Config { runnerVersion: core.getInput('runner-version') || '2.335.1', httpTokens: core.getInput('http-tokens') || 'required', encryptEbs: core.getInput('encrypt-ebs') || 'false', + cleanupOnStartFailure: core.getInput('cleanup-on-start-failure') || 'true', debug: core.getInput('debug') || 'false', }; diff --git a/src/gh.js b/src/gh.js index 5c346f09..6bbb3095 100644 --- a/src/gh.js +++ b/src/gh.js @@ -63,43 +63,17 @@ async function removeRunner() { } } -async function waitForRunnerRegistered(label) { - const timeoutMinutes = 5; - const retryIntervalSeconds = 10; - const quietPeriodSeconds = 30; - let waitSeconds = 0; - - core.info(`Waiting ${quietPeriodSeconds}s for the AWS EC2 instance to be registered in GitHub as a new self-hosted runner`); - await new Promise(r => setTimeout(r, quietPeriodSeconds * 1000)); - core.info(`Checking every ${retryIntervalSeconds}s if the GitHub self-hosted runner is registered`); - - return new Promise((resolve, reject) => { - const interval = setInterval(async () => { - const runner = await getRunner(label); - log.debug('wait_for_runner_poll', { label, elapsed_s: waitSeconds, found: !!runner, status: runner ? runner.status : null }); - - if (waitSeconds > timeoutMinutes * 60) { - log.error('wait_for_runner', { label, timeout_minutes: timeoutMinutes }); - core.error('GitHub self-hosted runner registration error'); - clearInterval(interval); - reject(`A timeout of ${timeoutMinutes} minutes is exceeded. Your AWS EC2 instance was not able to register itself in GitHub as a new self-hosted runner.`); - } - - if (runner && runner.status === 'online') { - log.info('wait_for_runner', { label, runner_id: runner.id, elapsed_s: waitSeconds }); - core.info(`GitHub self-hosted runner ${runner.name} is registered and ready to use`); - clearInterval(interval); - resolve(); - } else { - waitSeconds += retryIntervalSeconds; - core.info('Checking...'); - } - }, retryIntervalSeconds * 1000); - }); +// True once the runner for `label` has registered with GitHub and reports +// as online. Used as the success signal by the start action's wait loop +// (see src/wait.js), polled alongside the instance's bootstrap tag. +async function isRunnerOnline(label) { + const runner = await getRunner(label); + log.debug('runner_status', { label, found: !!runner, status: runner ? runner.status : null }); + return !!(runner && runner.status === 'online'); } module.exports = { getRegistrationToken, removeRunner, - waitForRunnerRegistered, + isRunnerOnline, }; diff --git a/src/index.js b/src/index.js index 7ce01e9f..df8f91fa 100644 --- a/src/index.js +++ b/src/index.js @@ -19,6 +19,7 @@ const aws = require('./aws'); const gh = require('./gh'); const config = require('./config'); const log = require('./log'); +const { waitForRunnerReady } = require('./wait'); const core = require('@actions/core'); // Write directly to the $GITHUB_OUTPUT file. The bundled @actions/core @@ -44,7 +45,21 @@ async function start() { const ec2InstanceId = await aws.startEc2Instance(label, githubRegistrationToken); setOutput(label, ec2InstanceId); await aws.waitForInstanceRunning(ec2InstanceId); - await gh.waitForRunnerRegistered(label); + + // Watch the bootstrap phone-home tag and GitHub registration together. + // On a bootstrap failure or registration timeout, capture the console + // output and (by default) terminate the instance so failed starts + // don't leak billing. The registration token is redacted from the + // captured output. + try { + await waitForRunnerReady({ + getBootstrapStatus: () => aws.getBootstrapStatus(ec2InstanceId), + isRunnerOnline: () => gh.isRunnerOnline(label), + }); + } catch (waitError) { + await aws.handleStartFailure(ec2InstanceId, { redactValues: [githubRegistrationToken] }); + throw waitError; + } log.info('start', { label, instance_id: ec2InstanceId, outcome: 'registered' }); } finally { core.endGroup(); diff --git a/src/wait.js b/src/wait.js new file mode 100644 index 00000000..b2a0b4ea --- /dev/null +++ b/src/wait.js @@ -0,0 +1,79 @@ +const core = require('@actions/core'); +const log = require('./log'); + +// Terminal bootstrap phone-home states carry a `failed:` value; see +// src/aws.js buildUserData() for the producing side. +const FAILED_PREFIX = 'failed:'; + +// Wait for the EC2 runner to come online, watching two signals in lockstep: +// +// 1. The instance's `ec2-github-runner:bootstrap` phone-home tag. A +// `failed:` value means cloud-init aborted — fail fast within one +// poll interval, naming the step, instead of waiting out the full +// registration timeout. +// 2. GitHub runner registration. Success is authoritative here: the runner +// showing up as `online` is what lets downstream jobs run. +// +// All I/O and timing are injected so the loop is unit-testable without real +// AWS/GitHub calls or wall-clock sleeps: +// - deps.getBootstrapStatus(): Promise +// - deps.isRunnerOnline(): Promise +// - deps.sleep(ms): Promise (defaults to setTimeout) +// +// Throws on `failed:` (err.bootstrapStep set) or on timeout +// (err.timedOut set) so the caller can capture diagnostics and clean up. +async function waitForRunnerReady(deps, opts = {}) { + const { getBootstrapStatus, isRunnerOnline } = deps; + const sleep = deps.sleep || ((ms) => new Promise((resolve) => setTimeout(resolve, ms))); + + const timeoutMinutes = opts.timeoutMinutes ?? 5; + const intervalSeconds = opts.intervalSeconds ?? 10; + const quietSeconds = opts.quietSeconds ?? 30; + + core.info(`Waiting ${quietSeconds}s for the AWS EC2 instance to bootstrap and register as a self-hosted runner`); + await sleep(quietSeconds * 1000); + core.info(`Checking every ${intervalSeconds}s for bootstrap failures and runner registration`); + + const deadlineSeconds = timeoutMinutes * 60; + let waitedSeconds = 0; + + for (;;) { + // 1. Fast-fail on a phoned-home bootstrap failure. + const status = await getBootstrapStatus(); + if (typeof status === 'string' && status.startsWith(FAILED_PREFIX)) { + const step = status.slice(FAILED_PREFIX.length); + log.error('wait_for_runner', { outcome: 'bootstrap_failed', step }); + core.error(`EC2 runner bootstrap failed during the "${step}" step`); + const error = new Error(`EC2 runner bootstrap failed during the "${step}" step. See the captured console output below.`); + error.bootstrapStep = step; + throw error; + } + + // 2. Success when the runner has registered and is online. + if (await isRunnerOnline()) { + log.info('wait_for_runner', { outcome: 'online', elapsed_s: waitedSeconds }); + core.info('GitHub self-hosted runner is registered and ready to use'); + return; + } + + // 3. Bounded wait — give up after the timeout. + if (waitedSeconds >= deadlineSeconds) { + log.error('wait_for_runner', { outcome: 'timeout', timeout_minutes: timeoutMinutes }); + const error = new Error( + `A timeout of ${timeoutMinutes} minutes is exceeded. Your AWS EC2 instance was not able to register itself in GitHub as a new self-hosted runner.`, + ); + error.timedOut = true; + throw error; + } + + log.debug('wait_for_runner_poll', { elapsed_s: waitedSeconds, bootstrap: status || null }); + core.info('Checking...'); + await sleep(intervalSeconds * 1000); + waitedSeconds += intervalSeconds; + } +} + +module.exports = { + waitForRunnerReady, + FAILED_PREFIX, +}; diff --git a/tests/diagnostics.test.js b/tests/diagnostics.test.js new file mode 100644 index 00000000..aea53082 --- /dev/null +++ b/tests/diagnostics.test.js @@ -0,0 +1,181 @@ +// Tests for the bootstrap-diagnostics helpers in src/aws.js: +// getBootstrapStatus, getConsoleOutputTail, redactSecrets, handleStartFailure. +// +// The EC2 client is mocked so a single mockSend records every command in +// call order — used to assert the capture-then-terminate ordering. withRetry +// is stubbed to call through once so terminate failures don't incur backoff +// delays. jest.mock is hoisted, so the mockSend name is `mock`-prefixed to +// satisfy the out-of-scope-reference guard. +const mockSend = jest.fn(); +jest.mock('@aws-sdk/client-ec2', () => ({ + EC2Client: jest.fn(() => ({ send: mockSend })), + DescribeImagesCommand: jest.fn((p) => ({ __command: 'DescribeImages', ...p })), + DescribeTagsCommand: jest.fn((p) => ({ __command: 'DescribeTags', ...p })), + GetConsoleOutputCommand: jest.fn((p) => ({ __command: 'GetConsoleOutput', ...p })), + RunInstancesCommand: jest.fn((p) => ({ __command: 'RunInstances', ...p })), + TerminateInstancesCommand: jest.fn((p) => ({ __command: 'TerminateInstances', ...p })), + AssociateAddressCommand: jest.fn((p) => ({ __command: 'AssociateAddress', ...p })), + waitUntilInstanceRunning: jest.fn(), +})); +jest.mock('../src/retry', () => ({ withRetry: (_step, fn) => fn() })); +jest.mock('../src/config', () => ({ + input: { mode: 'start', debug: 'false', cleanupOnStartFailure: 'true' }, + githubContext: { owner: 'o', repo: 'r' }, + tagSpecifications: null, +})); +jest.mock('@actions/core', () => ({ + info: jest.fn(), warning: jest.fn(), error: jest.fn(), setFailed: jest.fn(), + getInput: jest.fn(), startGroup: jest.fn(), endGroup: jest.fn(), debug: jest.fn(), +})); + +const core = require('@actions/core'); +const config = require('../src/config'); +const aws = require('../src/aws'); + +const b64 = (s) => Buffer.from(s, 'utf8').toString('base64'); +const commandsSent = () => mockSend.mock.calls.map((c) => c[0].__command); + +beforeEach(() => { + mockSend.mockReset(); + core.info.mockClear(); + core.warning.mockClear(); + core.error.mockClear(); + core.startGroup.mockClear(); + core.endGroup.mockClear(); + config.input.cleanupOnStartFailure = 'true'; +}); + +describe('redactSecrets', () => { + test('replaces every occurrence of each secret with ***', () => { + expect(aws.redactSecrets('a TOK b TOK c', ['TOK'])).toBe('a *** b *** c'); + }); + + test('ignores empty / undefined secrets', () => { + expect(aws.redactSecrets('hello', ['', undefined])).toBe('hello'); + expect(aws.redactSecrets('hello', [])).toBe('hello'); + expect(aws.redactSecrets('hello', undefined)).toBe('hello'); + }); + + test('treats secrets literally, not as regex', () => { + expect(aws.redactSecrets('x a.b.c y', ['a.b.c'])).toBe('x *** y'); + expect(aws.redactSecrets('keep aXbXc', ['a.b.c'])).toBe('keep aXbXc'); + }); +}); + +describe('getBootstrapStatus', () => { + test('returns the bootstrap tag value when present', async () => { + mockSend.mockResolvedValueOnce({ Tags: [{ Key: aws.BOOTSTRAP_TAG_KEY, Value: 'downloading' }] }); + await expect(aws.getBootstrapStatus('i-123')).resolves.toBe('downloading'); + // Scoped to the instance and the bootstrap key. + const cmd = mockSend.mock.calls[0][0]; + expect(cmd.__command).toBe('DescribeTags'); + expect(cmd.Filters).toEqual([ + { Name: 'resource-id', Values: ['i-123'] }, + { Name: 'key', Values: [aws.BOOTSTRAP_TAG_KEY] }, + ]); + }); + + test('surfaces a failed: value verbatim', async () => { + mockSend.mockResolvedValueOnce({ Tags: [{ Key: aws.BOOTSTRAP_TAG_KEY, Value: 'failed:configuring' }] }); + await expect(aws.getBootstrapStatus('i-123')).resolves.toBe('failed:configuring'); + }); + + test('returns null when the tag is not set yet', async () => { + mockSend.mockResolvedValueOnce({ Tags: [] }); + await expect(aws.getBootstrapStatus('i-123')).resolves.toBeNull(); + }); + + test('degrades to null when DescribeTags is denied (missing permission)', async () => { + mockSend.mockRejectedValueOnce(Object.assign(new Error('not authorized'), { name: 'UnauthorizedOperation' })); + await expect(aws.getBootstrapStatus('i-123')).resolves.toBeNull(); + }); +}); + +describe('getConsoleOutputTail', () => { + test('decodes base64 and returns the last N lines', async () => { + const lines = Array.from({ length: 10 }, (_, i) => `line${i + 1}`).join('\n'); + mockSend.mockResolvedValueOnce({ Output: b64(lines) }); + const tail = await aws.getConsoleOutputTail('i-1', { maxLines: 3 }); + expect(tail).toBe('line8\nline9\nline10'); + }); + + test('caps the tail at maxBytes', async () => { + mockSend.mockResolvedValueOnce({ Output: b64('a'.repeat(1000)) }); + const tail = await aws.getConsoleOutputTail('i-1', { maxLines: 200, maxBytes: 100 }); + expect(tail.length).toBe(100); + }); + + test('redacts secret values from the captured output', async () => { + const token = 'AAAA-registration-token-BBBB'; + mockSend.mockResolvedValueOnce({ Output: b64(`cloud-init: config.sh --token ${token} done`) }); + const tail = await aws.getConsoleOutputTail('i-1', { redactValues: [token] }); + expect(tail).not.toContain(token); + expect(tail).toContain('***'); + }); + + test('returns empty string when no console output is available', async () => { + mockSend.mockResolvedValueOnce({ Output: undefined }); + await expect(aws.getConsoleOutputTail('i-1')).resolves.toBe(''); + }); + + test('returns empty string when GetConsoleOutput fails', async () => { + mockSend.mockRejectedValueOnce(new Error('throttled')); + await expect(aws.getConsoleOutputTail('i-1')).resolves.toBe(''); + }); +}); + +describe('handleStartFailure', () => { + test('captures console output BEFORE terminating (default cleanup)', async () => { + mockSend.mockImplementation((cmd) => { + if (cmd.__command === 'GetConsoleOutput') return Promise.resolve({ Output: b64('boot failed here') }); + return Promise.resolve({}); + }); + + await aws.handleStartFailure('i-abc', { redactValues: [] }); + + expect(commandsSent()).toEqual(['GetConsoleOutput', 'TerminateInstances']); + expect(core.startGroup).toHaveBeenCalledTimes(1); + expect(core.endGroup).toHaveBeenCalledTimes(1); + }); + + test('redacts the registration token from the printed console output', async () => { + const token = 'AAAA-registration-token-BBBB'; + mockSend.mockImplementation((cmd) => { + if (cmd.__command === 'GetConsoleOutput') return Promise.resolve({ Output: b64(`log --token ${token} log`) }); + return Promise.resolve({}); + }); + + await aws.handleStartFailure('i-abc', { redactValues: [token] }); + + const allInfo = core.info.mock.calls.map((c) => String(c[0])).join('\n'); + expect(allInfo).not.toContain(token); + expect(allInfo).toContain('***'); + }); + + test('preserves the instance and prints a debug command when cleanup is disabled', async () => { + config.input.cleanupOnStartFailure = 'false'; + mockSend.mockImplementation((cmd) => { + if (cmd.__command === 'GetConsoleOutput') return Promise.resolve({ Output: b64('x') }); + return Promise.resolve({}); + }); + + await aws.handleStartFailure('i-keep', { redactValues: [] }); + + expect(commandsSent()).toEqual(['GetConsoleOutput']); + expect(commandsSent()).not.toContain('TerminateInstances'); + const warnings = core.warning.mock.calls.map((c) => String(c[0])).join('\n'); + expect(warnings).toContain('i-keep'); + expect(warnings).toContain('get-console-output'); + }); + + test('does not throw if termination itself fails after capture', async () => { + mockSend.mockImplementation((cmd) => { + if (cmd.__command === 'GetConsoleOutput') return Promise.resolve({ Output: b64('x') }); + return Promise.reject(new Error('terminate boom')); + }); + + await expect(aws.handleStartFailure('i-abc', { redactValues: [] })).resolves.toBeUndefined(); + const warnings = core.warning.mock.calls.map((c) => String(c[0])).join('\n'); + expect(warnings).toContain('i-abc'); + }); +}); diff --git a/tests/userdata.test.js b/tests/userdata.test.js new file mode 100644 index 00000000..ad9047b3 --- /dev/null +++ b/tests/userdata.test.js @@ -0,0 +1,91 @@ +// Tests for buildUserData — the cloud-init bootstrap script generator. +// It's a pure function, but aws.js requires config.js + @actions/core at +// module load, so those are mocked before require() (jest.mock is hoisted). +jest.mock('../src/config', () => ({ + input: { mode: 'start', debug: 'false' }, + githubContext: { owner: 'o', repo: 'r' }, + tagSpecifications: null, +})); +jest.mock('@actions/core', () => ({ + info: jest.fn(), warning: jest.fn(), error: jest.fn(), setFailed: jest.fn(), getInput: jest.fn(), + startGroup: jest.fn(), endGroup: jest.fn(), +})); + +const { buildUserData, BOOTSTRAP_TAG_KEY } = require('../src/aws'); + +const args = { + runnerVersion: '2.335.1', + owner: 'my-org', + repo: 'my-repo', + label: 'runner-abc12', + githubRegistrationToken: 'AAAA-secret-registration-token-BBBB', + shaX64: 'x64deadbeef', + shaArm64: 'arm64deadbeef', +}; + +describe('buildUserData', () => { + test('returns a bash script string with strict mode', () => { + const ud = buildUserData(args); + expect(typeof ud).toBe('string'); + expect(ud.startsWith('#!/bin/bash\nset -euo pipefail')).toBe(true); + }); + + test('emits a phone-home success tag for every bootstrap phase', () => { + const ud = buildUserData(args); + for (const phase of ['preparing', 'installing', 'creating-user', 'downloading', 'configuring', 'registered']) { + expect(ud).toContain(`gh_runner_phone_home ${phase}`); + } + }); + + test('sets GH_RUNNER_STEP before each phase so the ERR trap can name it', () => { + const ud = buildUserData(args); + for (const phase of ['preparing', 'installing', 'creating-user', 'downloading', 'configuring', 'registered']) { + expect(ud).toContain(`GH_RUNNER_STEP=${phase}`); + } + }); + + test('installs an ERR trap that phones home failed: in both shells', () => { + const ud = buildUserData(args); + const trapLine = "trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR"; + // Once in the outer (root) shell, once in the inner (runner-user) shell. + const occurrences = ud.split(trapLine).length - 1; + expect(occurrences).toBe(2); + }); + + test('writes the bootstrap tag via ec2 create-tags, best-effort', () => { + const ud = buildUserData(args); + expect(ud).toContain(`--tags "Key=${BOOTSTRAP_TAG_KEY},Value=$1"`); + // Best-effort: create-tags failure must never abort the bootstrap. + expect(ud).toContain('create-tags'); + expect(ud).toMatch(/create-tags .*\|\| true/); + }); + + test('derives instance identity from IMDSv2 (token-authenticated)', () => { + const ud = buildUserData(args); + expect(ud).toContain('X-aws-ec2-metadata-token-ttl-seconds'); + expect(ud).toContain('GH_RUNNER_IID=$(gh_runner_imds instance-id)'); + expect(ud).toContain('GH_RUNNER_REGION=$(gh_runner_imds placement/region)'); + }); + + test('drops the outer ERR trap before handing off to the runner-user shell', () => { + const ud = buildUserData(args); + // The inner shell owns download/configure/register failure attribution. + expect(ud).toContain('trap - ERR'); + expect(ud.indexOf('trap - ERR')).toBeLessThan(ud.indexOf("sudo -u runner -H bash <<'RUNNER_BOOTSTRAP'")); + }); + + test('embeds runner version, checksums, repo, label, and token into the script', () => { + const ud = buildUserData(args); + expect(ud).toContain('RUNNER_VERSION="2.335.1"'); + expect(ud).toContain('EXPECTED_SHA="x64deadbeef"'); + expect(ud).toContain('EXPECTED_SHA="arm64deadbeef"'); + expect(ud).toContain('--url "https://github.com/my-org/my-repo"'); + expect(ud).toContain('--labels "runner-abc12"'); + expect(ud).toContain(`--token "${args.githubRegistrationToken}"`); + }); + + test('configures the runner ephemeral, unattended, and without self-update', () => { + const ud = buildUserData(args); + expect(ud).toContain('--ephemeral --unattended --disableupdate'); + }); +}); diff --git a/tests/wait.test.js b/tests/wait.test.js new file mode 100644 index 00000000..34852883 --- /dev/null +++ b/tests/wait.test.js @@ -0,0 +1,79 @@ +// Tests for waitForRunnerReady — the combined bootstrap-watch + runner- +// registration loop. All I/O and timing are injected, so these run +// instantly with a no-op sleep and no real AWS/GitHub calls. +jest.mock('@actions/core', () => ({ info: jest.fn(), warning: jest.fn(), error: jest.fn(), debug: jest.fn() })); +jest.mock('../src/config', () => ({ input: { mode: 'start', debug: 'false' } })); + +const { waitForRunnerReady } = require('../src/wait'); + +const noSleep = () => Promise.resolve(); + +describe('waitForRunnerReady', () => { + test('resolves as soon as the runner is online', async () => { + const getBootstrapStatus = jest.fn().mockResolvedValue(null); + const isRunnerOnline = jest.fn().mockResolvedValue(true); + + await expect( + waitForRunnerReady({ getBootstrapStatus, isRunnerOnline, sleep: noSleep }, { quietSeconds: 0 }), + ).resolves.toBeUndefined(); + + expect(isRunnerOnline).toHaveBeenCalledTimes(1); + }); + + test('resolves after the runner comes online on a later poll', async () => { + const getBootstrapStatus = jest.fn() + .mockResolvedValueOnce(null) + .mockResolvedValueOnce('downloading') + .mockResolvedValueOnce('configuring'); + const isRunnerOnline = jest.fn() + .mockResolvedValueOnce(false) + .mockResolvedValueOnce(false) + .mockResolvedValueOnce(true); + + await waitForRunnerReady({ getBootstrapStatus, isRunnerOnline, sleep: noSleep }, { quietSeconds: 0, intervalSeconds: 1 }); + + expect(isRunnerOnline).toHaveBeenCalledTimes(3); + }); + + test('fails fast on a failed: bootstrap tag, naming the step', async () => { + const getBootstrapStatus = jest.fn().mockResolvedValue('failed:configuring'); + const isRunnerOnline = jest.fn().mockResolvedValue(false); + + await expect( + waitForRunnerReady({ getBootstrapStatus, isRunnerOnline, sleep: noSleep }, { quietSeconds: 0 }), + ).rejects.toMatchObject({ bootstrapStep: 'configuring' }); + + // Fast-fail happens before the registration check on that poll. + expect(isRunnerOnline).not.toHaveBeenCalled(); + }); + + test('surfaces the failing step name in the error message', async () => { + const getBootstrapStatus = jest.fn().mockResolvedValue('failed:downloading'); + const isRunnerOnline = jest.fn().mockResolvedValue(false); + + await expect( + waitForRunnerReady({ getBootstrapStatus, isRunnerOnline, sleep: noSleep }, { quietSeconds: 0 }), + ).rejects.toThrow(/"downloading"/); + }); + + test('times out when the runner never registers', async () => { + const getBootstrapStatus = jest.fn().mockResolvedValue(null); + const isRunnerOnline = jest.fn().mockResolvedValue(false); + + await expect( + waitForRunnerReady( + { getBootstrapStatus, isRunnerOnline, sleep: noSleep }, + { quietSeconds: 0, intervalSeconds: 10, timeoutMinutes: 0.5 }, + ), + ).rejects.toMatchObject({ timedOut: true }); + }); + + test('does not treat a non-failed bootstrap tag as an error', async () => { + const getBootstrapStatus = jest.fn().mockResolvedValue('registered'); + const isRunnerOnline = jest.fn().mockResolvedValueOnce(false).mockResolvedValueOnce(true); + + await expect( + waitForRunnerReady({ getBootstrapStatus, isRunnerOnline, sleep: noSleep }, { quietSeconds: 0, intervalSeconds: 1 }), + ).resolves.toBeUndefined(); + }); +});