Skip to content

Bypass hardlink caching for empty files - #257

Open
dzbarsky wants to merge 1 commit into
buildbarn:mainfrom
dzbarsky:upstream/pkg_cas_hardlinking_file_fetcher_empty_files
Open

dzbarsky wants to merge 1 commit into
buildbarn:mainfrom
dzbarsky:upstream/pkg_cas_hardlinking_file_fetcher_empty_files

Conversation

@dzbarsky

Copy link
Copy Markdown

Fetch empty files directly instead of sharing one cached inode. Input trees with many empty files can otherwise exhaust the filesystem's hardlink limit; empty blobs do not need a download. This also avoids cache bookkeeping for those files.

Added coverage for repeated executable and non-executable empty files. The focused TestHardlinkingFileFetcher test fails before the change and passes afterward in an isolated copy with generated mocks and a Darwin O_SEARCH dependency shim. The unmodified checkout's Go test command is blocked by those existing dependency/generation requirements.

@EdSchouten

Copy link
Copy Markdown
Member

Any type of file can currently exhaust the filesystem's hard link limit. Not just empty files. A change like this is therefore a workaround, which doesn't actually solve the problem at hand.

Somebody sent out a PR to address this issue some time ago, but didn't respond to feedback I had left: #237.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants