Repository navigation
Commit 6363b74
feat(upload): check multipart uploads against CRC64NVME checksums
Replace the md5 of each part and of the archive with CRC64NVME
checksums, and describe the profile archive with a single
`profileArchive` field in the upload metadata (version 12): its
encoding, size and CRC64NVME, and the size and CRC64NVME of its parts.
It replaces `profileEncoding`, `profileMd5` and `profileMultipart`.
Each part is hashed once, and the part CRCs are combined into the
archive's.
S3 can check a CRC64NVME on the whole archive of a multipart upload,
which md5 does not support, so a completion that leaves out a part or
assembles another archive is rejected, and not only a corrupted part.
It is also much cheaper to compute: about 0.5s for a 6 GiB archive on a
GitHub-hosted runner, against 24s for the part and archive md5s.
The upload endpoint now answers with `multipartUpload`, replacing
`multipartUploadUrls`: a presigned request per part and one to complete
the upload, each with the headers to send as is, as they are part of
the signature. The runner sends them without knowing which ones S3
checks, so the endpoint can change them without a runner release.
`crc-fast` is pinned to 1.9, the last release supporting Rust 1.88,
which requires `crc` 3.3.
Refs COD-3700
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>1 parent 4e4c21f commit 6363b74
10 files changed
Lines changed: 253 additions & 191 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
| 32 | + | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
139 | | - | |
140 | | - | |
141 | | - | |
| 139 | + | |
142 | 140 | | |
143 | 141 | | |
144 | 142 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
1 | 3 | | |
2 | 4 | | |
3 | 5 | | |
| |||
13 | 15 | | |
14 | 16 | | |
15 | 17 | | |
16 | | - | |
17 | | - | |
18 | | - | |
| 18 | + | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
29 | | - | |
30 | | - | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
34 | 36 | | |
| 37 | + | |
| 38 | + | |
35 | 39 | | |
36 | | - | |
37 | | - | |
| 40 | + | |
| 41 | + | |
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
| |||
59 | 63 | | |
60 | 64 | | |
61 | 65 | | |
62 | | - | |
| 66 | + | |
63 | 67 | | |
64 | 68 | | |
65 | 69 | | |
66 | 70 | | |
67 | 71 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
73 | 85 | | |
74 | 86 | | |
75 | 87 | | |
| |||
88 | 100 | | |
89 | 101 | | |
90 | 102 | | |
91 | | - | |
92 | | - | |
93 | | - | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
94 | 112 | | |
95 | 113 | | |
96 | 114 | | |
97 | 115 | | |
98 | 116 | | |
99 | 117 | | |
100 | | - | |
101 | | - | |
102 | | - | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
103 | 130 | | |
104 | 131 | | |
0 commit comments