Clarify cache behavior with parallel test processes

[Yuhi-Sato]

Hi! Thanks for FixtureKit.

I'd like to clarify how the cache behaves with process-based parallel tests such as parallel_tests.

The docs mention that cache data is mounted per connection_pool. If multiple test processes use the same cache_path, can cache files generated by one process be reused by the other processes, or is the cache effectively scoped to each process/connection pool?

If the cache is shared across processes, would you consider mentioning this explicitly in the docs?

I think this would be helpful since teams using FixtureKit for large test suites are also likely to run tests in parallel.

[ngan]

Hi there! Very good question, but unfortunately, we don't have parallel testing in our suite so I can't test it out. I think the best to do is to set config.cache_path = "tmp/cache/fixture_kit/#{ENV["TEST_ENV_NUMBER"]}" so that each test process gets its own cache directory so they don't clobber each other. There might be some edge cases? I'm not sure. If anyone doing this has some other workaround, I'm happy to incorporate doc updates or fixes.

[navidemad]

We run FixtureKit on a large suite in parallel, so here is what we found. The answer to "can one process reuse another's cache files" turns out to be different for each runner, which is probably why this is confusing. I've opened #82 to put it in the reference.

Rails parallelize — files are shared and reused across processes, by design. ActiveSupport::Testing::Parallelization forks its workers in Minitest.run, before any suite runs, and each worker runs individual test methods (Minitest.run_one_method), never run_suite. FixtureKit generates from FixtureKit::Minitest::ClassMethods#run_suite, so Runner#start and every generate happen in the parent and the workers only mount what the parent wrote. One shared cache_path is correct here, and it is what the default gives you. There is also nothing to key the path on: Rails names the per-worker databases itself and never sets TEST_ENV_NUMBER (no reference to it in activerecord, activesupport or railties). Keying on a worker number would point the workers at directories the parent never writes to.

parallel_tests — your suggestion is right. Each worker is a full process that boots the app and calls Runner#start, so each one clears the directory the others are generating into or mounting from. config.cache_path = "tmp/cache/fixture_kit/#{ENV["TEST_ENV_NUMBER"]}" fixes it; the cost is that every worker generates its own copy of the fixtures its share of the suite needs.

One directory shared between processes — a warm-up run that fills the cache before the suite, or a CI cache restored between jobs. This needs FIXTURE_KIT_PRESERVE_CACHE, otherwise the next process to start deletes it, and preserving it hands invalidation to the caller: Cache#exists? is File.exist?, so nothing compares that file against the definitions, the factories they call, or the schema. This is where it bit us. A cache written before a PaperTrail guard landed in our test helper kept being mounted, and surfaced as PG::UniqueViolation: versions_pkey in tests with no visible relationship to fixtures. We bound it by folding a digest of the definitions, factories and schema into cache_path, so a cache written under any other state of the code is ignored and regenerated rather than read. #82 documents the pattern and its two costs (obsolete digest directories are not reclaimed, and one changed file regenerates everything).

I also opened #83 for one thing that can only be fixed in the gem: FileCache#write is a plain File.write, which truncates the destination and fills it back in, so a shared directory can hand a concurrent reader a prefix of the JSON and a JSON::ParserError that looks like a corrupt cache. Writing a sibling file and renaming it over the destination makes that window disappear.

Happy to adjust the wording in #82 if any of it reads wrong to you.

[Yuhi-Sato]

@ngan
Thanks for the reply!

As @navidemad suggested, our team enabled FIXTURE_KIT_PRESERVE_CACHE, generated the cache files before running the parallel tests, and then shared the same cache_path across multiple processes. This worked without issues and significantly improved our test performance.

Using a process-specific path such as config.cache_path = "tmp/cache/fixture_kit/#{ENV["TEST_ENV_NUMBER"]}" is certainly one possible workaround. However, each process would need to generate its own cache files, which may reduce the performance benefits of FixtureKit.

I think a brief mention of the shared-cache approach in the documentation would be sufficient. What do you think?