Tags: ghzhost/gtfs-scorecard
Tags
perf(ci): publish only artifacts whose content changed (ChelseaKR#227) `aws s3 sync` transfers when the local mtime is newer than the object. CI checks the repository out fresh on every run, so the daily collect job re-uploaded all ~28,700 published objects to change about 3,100 of them. `scorecard publish-artifacts` compares content instead of clocks: one paginated ListObjectsV2 over the destination prefix, then each local file's MD5 against the object's ETag. The bucket is SSE-S3 and the command writes with a single PutObject, so a published object's ETag is the MD5 of its bytes. A file is skipped only when its size and its hash both match; a missing object, a different size, a non-MD5 ETag, or a hash mismatch all upload, so the comparison can over-upload but has no path that skips a changed file. `--size-only` would have cut the same requests and is not used: it cannot see a re-score that keeps the same byte length. Measured against the live bucket, 1,431 of 3,261 changed objects in one pass were exactly that shape. test_s3_publish.py proves a same-length change is still published and test_workflow_safety.py keeps `--size-only` out of every workflow. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>