Fix Windows argument encoding by fetching UTF-8 command line#2293
Fix Windows argument encoding by fetching UTF-8 command line#2293alaotach wants to merge 1 commit into
Conversation
|
I don't have a Windows machine handle so I asked Claude to take a look - I think its review is correct. Apologies for the slop. Blocker: it does not fix the reported bug. The error in #2284 — Failed to open one of the input file(s): File does not exist. — comes from exactly one place, src/ccextractor.c:122, reached via get_total_file_size() at src/ccextractor.c:116. That runs because binary_concat defaults to true (src/rust/lib_ccxr/src/common/options.rs:558, and C ccx_common_option.c:30). get_total_file_size() is still C and opens with the narrow CRT: src/lib_ccx/file_functions.c:36: h = OPEN(ctx->inputfile[i], O_RDONLY | O_BINARY); // OPEN == _open on Win32 _open interprets char* in the CRT ANSI code page. Repo has no UTF-8 manifest (windows/ccextractor.vcxproj — no activeCodePage), and main() calls setlocale(LC_ALL, "") which gives the ACP, not UTF-8. So the byte sequence for £ just changes flavor of wrong:
Same fatal, same message. The PR moves the mangling from the Rust lossy conversion into the CRT. It also would not fix the console display (Input: line) — printing UTF-8 to a CP-1252 console shows £, not £. Note the direction is right — the primary demux path is Rust (src/rust/src/demuxer/demux.rs:225, File::open → CreateFileW), so UTF-8 argv is what that path wants. The gap is every remaining C narrow call: file_functions.c:36/38, ccx_demuxer.c:129/131, output creation at output.c:55/82/84, ccx_encoders_common.c:1373/1465, utility.c:402, params.c:744. Minimal complement: add on Windows, alongside the argv conversion: Secondary issues
CI: all 24 green, including the Windows regression suite — no ASCII regression. Compiles fine on Linux (whole block is #ifdef _WIN32). Recommendation: request changes. Ask for the CRT UTF-8 locale + console CP, plus the malloc checks and NULL terminator. Then it needs re-testing against the actual reporter path in #2284. I can't run Windows here, so the verdict is from code paths, not a live run — but the failing call site is unambiguous. Want me to draft the review comment, or move to #2297? |
[FIX] Resolve Windows ANSI argument mangling for non-ASCII paths
In raising this pull request, I confirm the following (please check boxes):
Reason for this PR:
Sanity check:
Fixes
Closes #2284
Problem
On Windows, the standard
main(int argc, char *argv[])entry point receives arguments encoded in the system's local legacy ANSI code page (e.g., CP-1252), not UTF-8.indows ANSI argument mangling for non-ASCII paths
In raising this pull request, I confirm the following (please check boxes):
Reason for this PR:
Sanity check:
Fixes
Closes #2284
Problem
On Windows, the standard
main(int argc, char *argv[])entry point receives arguments encoded in the system's local legacy ANSI code page (e.g., CP-1252), not UTF-8.When these arguments are passed to the new Rust parameter parser (
ccxr_parse_parameters), Rust strictly expects UTF-8 strings and attempts to parse them usingto_string_lossy(). Because non-ASCII ANSI characters (like the£symbol, which is0xA3) are invalid UTF-8 byte sequences, they get gracefully replaced with the Unicode Replacement Character (U+FFFD). When printed back to the console, this manifests as mangled, permanently mangling the file path and preventing CCExtractor from opening the file.Fix
Added a Windows-specific
#ifdef _WIN32block at the very top ofmain()inccextractor.cto intercept and sanitize the arguments.It uses the native Windows APIs
GetCommandLineW()andCommandLineToArgvW()to fetch the raw UTF-16 Unicode command line, safely converts the wide strings to standard UTF-8 usingWideCharToMultiByte(), and dynamically overrides the legacyargvarray before it is passed to the rest of the application.This perfectly preserves all non-ASCII characters without changing the Rust API, breaking backwards compatibility for libraries, or changing the main entry point signature.
Repro instructions
£symbol. Example:£5 Million Restoration.ts.ccextractorwinfull.exe "£5 Million Restoration.ts".Error: Failed to open one of the input file(s): File does not exist.and prints the mangled input file name as�5 Million Restoration.ts.£symbol, natively locates the file, and proceeds to extract the captions.