Skip to content

Conversation

kaspersv
Copy link
Contributor

@kaspersv kaspersv commented Sep 10, 2025

This PR increases the size limit imposed on (uncompressed) overlay base databases for uploading to the Actions Cache from 6GB to 15GB. With an expected compression ratio of 2.5:1 the 15GB uncompressed limit translates into an effective 6GB limit.

Overlay analysis is currently limited to the github and dsp-testing orgs and controlled by feature flags (codeql_action_overlay_analysis_*). This size limit will be revisited before removing the restriction to the github and dsp-testing org.

Risk assessment

Low risk: Overlay analysis is under feature flags and restricted to the github and dsp-testing orgs.

@kaspersv kaspersv force-pushed the kaspersv/increase-overlay-base-size-limit branch from ef9af8d to c02e46b Compare September 10, 2025 10:18
@kaspersv kaspersv force-pushed the kaspersv/increase-overlay-base-size-limit branch from c02e46b to 8a84a62 Compare September 10, 2025 10:31
@kaspersv kaspersv marked this pull request as ready for review September 10, 2025 10:49
@kaspersv kaspersv requested a review from a team as a code owner September 10, 2025 10:49
@kaspersv kaspersv requested review from Copilot, mbg and cklin September 10, 2025 10:49
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR increases the size limit for cached overlay base databases from 6GB to 15GB to enable Ruby overlay analysis for the github/github repository, which produces an 11GB uncompressed overlay base database.

  • Increased OVERLAY_BASE_DATABASE_MAX_UPLOAD_SIZE_MB from 6000 to 15000 MB
  • Updated comments to reflect the new size limit and its implications
  • Added TODO comment noting the limit should be revisited before broader rollout

Comment on lines +35 to +36
* compared to the default 10GB Actions Cache capacity, but enforcement of Actions
* Cache quotas is not immediate.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the addition of a comment acknowledging that this is a large limit, but I don't think that justifying it based on the speed of the enforcement makes sense.

Suggested change
* compared to the default 10GB Actions Cache capacity, but enforcement of Actions
* Cache quotas is not immediate.
* compared to the default 10GB Actions Cache capacity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think the speed of enforcement makes quite a big difference in practice with a limit that is over 50% of the Actions Cache quota. If each compressed overlay base database was +5GB and existing cache entries were immediately evicted from the cache to make room before a new cache entry could be uploaded, then the cache wouldn't be able to fit two overlay base databases and other concurrent CodeQL action runs on PRs would effectively fall back to normal analysis while a new overlay base databases was being uploaded. That will not happen because enforcement of quotas is not immediate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the cache quota is not specific to CodeQL and is shared among all Actions caches for a repo. It may be that the speed of enforcement makes a difference for the overlay databases if those are the only caches for a repo and they are this large, but it would still hurt customers if we rapidly take up most of their quota and the rate of cache churn resulting from overlay caches competing with customer caches within the standard 10GB quota would plausibly also evict overlay databases before they can be useful.

@kaspersv kaspersv merged commit f374a62 into main Sep 10, 2025
574 checks passed
@kaspersv kaspersv deleted the kaspersv/increase-overlay-base-size-limit branch September 10, 2025 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants