Skip to content

fix(httpfs): initialize HTTP client on size-cache hit to avoid SIGSEGV on repeat opens - #73

Merged
adsharma merged 2 commits into
mainfrom
fix/880-httpfs-null-client-sigsegv
Sep 1, 2026
Merged

fix(httpfs): initialize HTTP client on size-cache hit to avoid SIGSEGV on repeat opens#73
adsharma merged 2 commits into
mainfrom
fix/880-httpfs-null-client-sigsegv

Conversation

@adsharma

@adsharma adsharma commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes the crash reported in LadybugDB/ladybug#880.

Problem

On 0.20.x, every remote (httpfs) read over http(s):// segfaults at bind time on the first ranged GET; 0.19.1 and earlier are clean. Local reads of the same files are fine, and URLs that 404/403 raise a clean RuntimeError instead of crashing.

Backtrace (matches the one in #880):

#0 std::_Function_handler<httplib::Result (), HTTPFileSystem::getRangeRequest(...)::{lambda()#1}>::_M_invoke(...)
#1 HTTPFileSystem::runRequestWithRetry(...)
#2 HTTPFileSystem::getRangeRequest(...)
#3 HTTPFileSystem::readFromFile(...)
...
#15 binder::Binder::bind(...)

Root cause

05c63dd ("perf(httpfs): pool HTTP connections and cache remote file sizes") added a process-wide cache of remote file sizes so openFile() skips the HEAD round trip on repeat opens of the same URL. The fast-path early return in HTTPFileInfo::initMetadata() was placed before initializeClient():

{
    std::lock_guard<std::mutex> lck{HTTPFileSystem::httpSizeCacheMtx};
    auto it = HTTPFileSystem::httpSizeCache.find(path);
    if (it != HTTPFileSystem::httpSizeCache.end()) {
        length = it->second;
        return;              // ← skips initializeClient()
    }
}
...
initializeClient();

httpClient is only assigned in initializeClient(), so the second openFile() of the same URL returns a handle whose httpClient is null. Every scan opens the same remote URL at least twice (bind + execution), and the first ranged GET on the second handle dereferences the null client — hence the SIGSEGV on all remote reads.

This also explains the reporter's observations:

  • local parquet reads are unaffected (different file system),
  • 404/403 URLs raise a clean error because they fail during the first open's HEAD (which does initialize the client),
  • 0.19.1 (pre-05c63dd) is clean.

Reproduced locally with a python -m http.server file server and LOAD FROM 'http://127.0.0.1:8123/user.parquet' RETURN count(*): deterministic SIGSEGV (exit 139), with the crash point being mov (%r13),%rsi where r13 = 0 (the null client).

Fix

  1. HTTPFileInfo::initMetadata(): call initializeClient() above the size-cache lookup. It only constructs an httplib::Client (no network I/O), so the invariant "an initialized HTTPFileInfo always has a client" is restored while keeping the HEAD deduplication.
  2. Defensive null-client guards in headRequest / getRangeRequest / postRequest / putRequest so no code path can dereference an uninitialized client.

Testing

New regression test httpfs_httpfs_test (httpfs/test/httpfs_test.cpp), which serves a fixture file over a local range-capable httplib server and exercises the real VFS path:

  • HttpFileSystemTest.ReadAfterRepeatOpenOfSizeCachedUrl: opens the same URL twice, reads (including at a non-zero offset) from the second handle. Segfaults without the fix; passes with it.
  • HttpFileSystemTest.RepeatOpenDoesNotIssueSecondHead: asserts repeat opens still issue exactly one HEAD (the perf win of 05c63dd is preserved).

Verified:

  • without fix: test binary dies with SIGSEGV (exit 139) in the regression test
  • with fix: both new tests pass, existing httpfs_xetfs_test (4 tests) passes
  • end-to-end: repeated remote parquet reads and a 20000-row multi-block remote CSV return correct counts with the fixed extension

…V on repeat opens

Fixes LadybugDB/ladybug#880.

05c63dd introduced a process-wide cache of remote file sizes so that
openFile() skips the HEAD round trip on repeat opens of the same URL.
However, the fast-path early return in HTTPFileInfo::initMetadata() was
placed before initializeClient(), leaving httpClient null on every open
after the first. The first ranged GET on such a handle then dereferenced
the null httplib client and crashed with SIGSEGV. Because every scan
opens the same remote URL at least twice (bind + execution), *all*
remote reads on 0.20.x crashed - local reads were unaffected, and
404/403 URLs raised a clean error because they fail during the first
open's HEAD.

initializeClient() only constructs an httplib::Client (no network I/O),
so moving it above the size-cache lookup restores the invariant that an
initialized HTTPFileInfo always has a client while keeping the HEAD
deduplication. Also add defensive null-client guards to head/get/post/
put requests so no code path can dereference an uninitialized client.

Adds a gtest regression test (httpfs_httpfs_test) that serves a file
over a local HTTP server, opens it twice, and reads from the second
handle - it segfaults without the fix and passes with it - plus a test
asserting repeat opens still issue only one HEAD.

Reported-by: jfrench9
CI (manylinux_2_28, Release) crashed with SIGSEGV in
httplib::ClientImpl::create_and_connect_socket in the new httpfs tests.

Cause: httpfs_extension_source is compiled with CPPHTTPLIB_OPENSSL_SUPPORT
while the test TUs included httplib.h without it. httplib types change
layout depending on that flag (e.g. httplib::Socket gains an `SSL*`
member), so mixing both variants in one binary is an ODR violation; the
linker merges the inline member definitions and the wrong copy reads the
wrong offsets. Locally the linker happened to pick a compatible copy,
which is why this was not caught before pushing.

Compile all httpfs test targets with the same flag so every TU in the
binary sees the same httplib layout.
@adsharma
adsharma merged commit 12e3b39 into main Sep 1, 2026
2 checks passed
@adsharma
adsharma deleted the fix/880-httpfs-null-client-sigsegv branch September 1, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant