Skip to content

Reject invalid memory filesystem copy destinations - #2268

Open
dlowzzxx wants to merge 1 commit into
fsspec:masterfrom
dlowzzxx:fix/memory-copy-parent-collision
Open

dlowzzxx wants to merge 1 commit into
fsspec:masterfrom
dlowzzxx:fix/memory-copy-parent-collision

Conversation

@dlowzzxx

Copy link
Copy Markdown
Contributor

Summary

MemoryFileSystem.cp_file inserts directly into its store without validating the destination hierarchy. If /parent is a file, copy("/src", "/parent/child") still succeeds. Afterwards info("/parent") reports a directory while isfile("/parent") is true. mv also removes the original source after creating this invalid destination.

Direct file-to-directory and directory-to-file copies can similarly leave conflicting entries in store and pseudo_dirs.

Validate destination ancestors and the exact destination before changing either collection. The ancestor check uses the same FileExistsError behavior as MemoryFileSystem._open. Ordinary overwrites, missing destination parents, public copy-into-directory behavior, and idempotent directory copies remain supported.

Tests

Regression tests cover cp_file, copy, and mv with file ancestors; explicit and implicit directory destinations; directory-to-file collisions; and unchanged source/destination contents after rejection. Success cases cover normal overwrites and directory copies.

Validation

  • Memory filesystem module: 85 passed.
  • Memory, filesystem-specification, utility, and core suites: 538 passed, 129 skipped, 1 xfailed (Windows, Python 3.13).
  • Ruff 0.14.3 check and format check; git diff --check.
  • The broader run emits a Windows socket-server cleanup warning in an existing test_walk_on_error_reaches_subdirectories test; no tests fail.

Checking an implicit destination directory scans the memory store, as existing directory detection does. No remote requests or external services are needed for the new tests.

Assisted-by: OpenAI Codex. The failure was reproduced locally and the code and validation were reviewed before submission.

Check destination ancestors and file/directory collisions before mutating the memory store, preventing copy and mv from creating an inconsistent tree.

Assisted-by: OpenAI Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant