Skip to content

Fix Windows argument encoding by fetching UTF-8 command line#2293

Open
alaotach wants to merge 1 commit into
CCExtractor:masterfrom
alaotach:fix-windows-utf8-argv
Open

Fix Windows argument encoding by fetching UTF-8 command line#2293
alaotach wants to merge 1 commit into
CCExtractor:masterfrom
alaotach:fix-windows-utf8-argv

Conversation

@alaotach

@alaotach alaotach commented Jul 8, 2026

Copy link
Copy Markdown

[FIX] Resolve Windows ANSI argument mangling for non-ASCII paths

In raising this pull request, I confirm the following (please check boxes):
Reason for this PR:

  • This PR adds new functionality.
  • This PR fixes a bug that I have personally experienced or that a real user has reported and for which a sample exists.
  • This PR is porting code from C to Rust.

Sanity check:

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • If the PR adds new functionality, I've added it to the changelog. If it's just a bug fix, I have NOT added it to the changelog.
  • I am NOT adding new C code unless it's to fix an existing, reproducible bug.

Fixes

Closes #2284

Problem

On Windows, the standard main(int argc, char *argv[]) entry point receives arguments encoded in the system's local legacy ANSI code page (e.g., CP-1252), not UTF-8.
indows ANSI argument mangling for non-ASCII paths

In raising this pull request, I confirm the following (please check boxes):
Reason for this PR:

  • This PR adds new functionality.
  • This PR fixes a bug that I have personally experienced or that a real user has reported and for which a sample exists.
  • This PR is porting code from C to Rust.

Sanity check:

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • If the PR adds new functionality, I've added it to the changelog. If it's just a bug fix, I have NOT added it to the changelog.
  • I am NOT adding new C code unless it's to fix an existing, reproducible bug.

Fixes

Closes #2284

Problem

On Windows, the standard main(int argc, char *argv[]) entry point receives arguments encoded in the system's local legacy ANSI code page (e.g., CP-1252), not UTF-8.

When these arguments are passed to the new Rust parameter parser (ccxr_parse_parameters), Rust strictly expects UTF-8 strings and attempts to parse them using to_string_lossy(). Because non-ASCII ANSI characters (like the £ symbol, which is 0xA3) are invalid UTF-8 byte sequences, they get gracefully replaced with the Unicode Replacement Character (U+FFFD). When printed back to the console, this manifests as mangled, permanently mangling the file path and preventing CCExtractor from opening the file.

Fix

Added a Windows-specific #ifdef _WIN32 block at the very top of main() in ccextractor.c to intercept and sanitize the arguments.

It uses the native Windows APIs GetCommandLineW() and CommandLineToArgvW() to fetch the raw UTF-16 Unicode command line, safely converts the wide strings to standard UTF-8 using WideCharToMultiByte(), and dynamically overrides the legacy argv array before it is passed to the rest of the application.

This perfectly preserves all non-ASCII characters without changing the Rust API, breaking backwards compatibility for libraries, or changing the main entry point signature.

Repro instructions

  1. On a standard Windows installation without global UTF-8 beta features enabled, rename any video file to include the £ symbol. Example: £5 Million Restoration.ts.
  2. Run CCExtractor against it: ccextractorwinfull.exe "£5 Million Restoration.ts".
  3. Before fix: CCExtractor fails with Error: Failed to open one of the input file(s): File does not exist. and prints the mangled input file name as �5 Million Restoration.ts.
  4. After fix: CCExtractor successfully parses the £ symbol, natively locates the file, and proceeds to extract the captions.

@cfsmp3

cfsmp3 commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

I don't have a Windows machine handle so I asked Claude to take a look - I think its review is correct. Apologies for the slop.

Blocker: it does not fix the reported bug.

The error in #2284 — Failed to open one of the input file(s): File does not exist. — comes from exactly one place, src/ccextractor.c:122, reached via get_total_file_size() at src/ccextractor.c:116. That runs because binary_concat defaults to true (src/rust/lib_ccxr/src/common/options.rs:558, and C ccx_common_option.c:30).

get_total_file_size() is still C and opens with the narrow CRT:

src/lib_ccx/file_functions.c:36: h = OPEN(ctx->inputfile[i], O_RDONLY | O_BINARY); // OPEN == _open on Win32

_open interprets char* in the CRT ANSI code page. Repo has no UTF-8 manifest (windows/ccextractor.vcxproj — no activeCodePage), and main() calls setlocale(LC_ALL, "") which gives the ACP, not UTF-8.

So the byte sequence for £ just changes flavor of wrong:

  • before: Rust to_string_lossy → EF BF BD (U+FFFD) → _open fails
  • after: correct UTF-8 → C2 A3 → _open in CP-1252 looks for £ → still fails

Same fatal, same message. The PR moves the mangling from the Rust lossy conversion into the CRT. It also would not fix the console display (Input: line) — printing UTF-8 to a CP-1252 console shows £, not £.

Note the direction is right — the primary demux path is Rust (src/rust/src/demuxer/demux.rs:225, File::open → CreateFileW), so UTF-8 argv is what that path wants. The gap is every remaining C narrow call: file_functions.c:36/38, ccx_demuxer.c:129/131, output creation at output.c:55/82/84, ccx_encoders_common.c:1373/1465, utility.c:402, params.c:744.

Minimal complement: add on Windows, alongside the argv conversion:
setlocale(LC_ALL, ".UTF8"); // UCRT: narrow file APIs then treat char* as UTF-8
SetConsoleOutputCP(CP_UTF8); // for correct display of the path
(then keep the existing setlocale(LC_NUMERIC, "POSIX") after it).

Secondary issues

  1. src/ccextractor.c:445,449 — both malloc results used unchecked. argv[i] = malloc(size) then written by WideCharToMultiByte → NULL deref on OOM. Project policy requires alloc NULL checks.
  2. argv is not NULL-terminated — only wargc pointers allocated, no argv[argc] = NULL. Harmless today (ccxr_parse_parameters uses from_raw_parts(argv, argc) — src/rust/src/lib.rs:693), but it breaks the C standard guarantee for any future consumer.
  3. WideCharToMultiByte sizing call returning 0 on failure → malloc(0) then unchecked write.
  4. Array and strings never freed. Process-lifetime, low priority.

CI: all 24 green, including the Windows regression suite — no ASCII regression. Compiles fine on Linux (whole block is #ifdef _WIN32).

Recommendation: request changes. Ask for the CRT UTF-8 locale + console CP, plus the malloc checks and NULL terminator. Then it needs re-testing against the actual reporter path in #2284.

I can't run Windows here, so the verdict is from code paths, not a live run — but the failing call site is unambiguous.

Want me to draft the review comment, or move to #2297?

@cfsmp3
cfsmp3 self-requested a review July 25, 2026 22:13

@cfsmp3 cfsmp3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: Cannot open file - UNC path with Unicode character

2 participants