Repository navigation
Commit a45c0ec
feat(upload): split profile archive uploads into concurrent parts (#565)
* feat(upload): upload profile archives as concurrent multipart uploads
S3 rejects single uploads above 5 GiB, and a single connection to S3
only reaches about 20-25 MiB/s on GitHub-hosted runners, so large
profile archives were slow or impossible to upload.
Every archive is now sent as an S3 multipart upload, replacing the
single-request upload: about two parts per concurrent upload, each
between 16 MiB and 256 MiB, so a small archive is a single part. The
md5 of every part is computed in the same pass as the archive md5 and
sent in the upload metadata (version 12) as `profileMultipart`. The API
answers with `multipartUploadUrls`: the parts are uploaded 8 at a time
(overridable with `CODSPEED_UPLOAD_CONCURRENCY`), each with its own
retries, then the upload is completed with the part ETags in order.
This applies to both on-disk and in-memory (gzip) archives.
This requires an upload endpoint that accepts metadata version 12 and
answers with `multipartUploadUrls`.
Walltime profile folders above 5 GiB are no longer gzipped on disk to
fit in a single request, and the runner no longer caps the archive size
itself: the upload endpoint rejects archives above its limit, with the
reason shown in the runner output, so the limit can change without a
runner release. Archives are now hashed while streaming on the blocking
thread pool, instead of being read whole into memory on the async
runtime.
Closes COD-3700
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(upload): check multipart uploads against CRC64NVME checksums
Replace the md5 of each part and of the archive with CRC64NVME
checksums, and describe the profile archive with a single
`profileArchive` field in the upload metadata (version 12): its
encoding, size and CRC64NVME, and the size and CRC64NVME of its parts.
It replaces `profileEncoding`, `profileMd5` and `profileMultipart`.
Each part is hashed once, and the part CRCs are combined into the
archive's.
S3 can check a CRC64NVME on the whole archive of a multipart upload,
which md5 does not support, so a completion that leaves out a part or
assembles another archive is rejected, and not only a corrupted part.
It is also much cheaper to compute: about 0.5s for a 6 GiB archive on a
GitHub-hosted runner, against 24s for the part and archive md5s.
The upload endpoint now answers with `multipartUpload`, replacing
`multipartUploadUrls`: a presigned request per part and one to complete
the upload, each with the headers to send as is, as they are part of
the signature. The runner sends them without knowing which ones S3
checks, so the endpoint can change them without a runner release.
`crc-fast` is pinned to 1.9, the last release supporting Rust 1.88,
which requires `crc` 3.3.
Refs COD-3700
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* review changes (will squash merge)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>1 parent 90cc8e9 commit a45c0ec
11 files changed
Lines changed: 1000 additions & 250 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | | - | |
| 40 | + | |
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| 80 | + | |
80 | 81 | | |
81 | 82 | | |
82 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
139 | | - | |
140 | | - | |
| 139 | + | |
141 | 140 | | |
142 | 141 | | |
143 | 142 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
1 | 3 | | |
2 | 4 | | |
3 | 5 | | |
4 | 6 | | |
5 | 7 | | |
6 | 8 | | |
7 | 9 | | |
8 | | - | |
| 10 | + | |
9 | 11 | | |
10 | 12 | | |
11 | 13 | | |
12 | 14 | | |
13 | 15 | | |
14 | 16 | | |
15 | 17 | | |
16 | | - | |
17 | | - | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| |||
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
27 | 44 | | |
28 | 45 | | |
29 | 46 | | |
| |||
46 | 63 | | |
47 | 64 | | |
48 | 65 | | |
49 | | - | |
| 66 | + | |
50 | 67 | | |
51 | 68 | | |
52 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
53 | 87 | | |
54 | 88 | | |
55 | 89 | | |
56 | 90 | | |
57 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
0 commit comments