Tests: Gracefully skip Tests_Readme when external version APIs time out or are unavailable by devmuhib009 · Pull Request #14683 · WordPress/wordpress-develop

devmuhib009 · 2026-10-22T05:17:33Z

Trac ticket: https://core.trac.wordpress.org/ticket/63914

Description

In Tests_Readme, get_response_body() contacts external APIs (php.net, endoflife.date, and mariadb.org) to verify the minimum recommended versions in readme.html.

Previously, if a request failed due to a network timeout or connection failure, assertNotWPError() caused an immediate hard test failure. Furthermore, only HTTP 503 was handled for skipping; other server errors or rate limiting (e.g. 429, 500, 502, 504) resulted in hard test failures.

This PR updates Tests_Readme::get_response_body() to:

Check is_wp_error( $response ) and call markTestSkipped() with the error message when connection or timeout issues occur.
Gracefully skip on HTTP 429, 500, 502, 503, or 504 server responses.

This prevents intermittent external network outages and rate limits from causing CI test failures on trunk.

…ut or are unavailable. Updates get_response_body() in Tests_Readme to skip the test when a network timeout or connection error occurs (is_wp_error()) or when external servers return HTTP 429, 500, 502, 503, or 504 errors, preventing intermittent CI test failures caused by external service availability. Props devmuhib009. See #63914.

github-actions · 2026-10-22T05:17:43Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs.

Core Committers: Use this line as a base for the props when committing in SVN:

Props devmuhib, huzaifaalmesbah, lancewillett.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Props Bot commands

You can trigger Props Bot actions by commenting on this PR:

@propsbot add <username> to give props to a GitHub user. Requires write access.
@propsbot remove <username> to leave a user out of the props. Requires write access.
@propsbot refresh to refresh this list. Adding the props-bot label works too.

huzaifaalmesbah

Thanks for working on this, @devmuhib009! Skipping on 429 and 5xx responses should cut a lot of flaky readme failures. LGTM.

AI Review: Opus 5.5

lancewillett

One blocker: the blanket WP_Error skip masks errors beyond external availability failures. Details inline.

_{AI review · gpt-6}

lancewillett · 2026-10-26T16:36:25Z

+		$host       = isset( $parsed_url['host'] ) ? $parsed_url['host'] : $url;
+		$response   = $this->wp_remote_get( $url );
+
+		if ( is_wp_error( $response ) ) {


The explicit 429/500/502/503/504 skip list looks reasonable, but please retain failures for invalid URLs, locally blocked requests, and unknown errors instead of skipping every WP_Error. The inherited HTTP helper already handles recognized timeouts. A focused probe confirms that http_request_failed with “A valid URL was not provided.” and http_request_not_executed now both skip. Those can indicate broken test setup or a local HTTP regression rather than an unavailable upstream.

Please narrow the additional error skips to identified availability failures and check that invalid/local errors still fail. Checking only http_request_failed is not sufficient: invalid URLs use that code too.

_{AI review · gpt-6}

Copilot

🟢 Approval recommended

The focused changes implement the requested outage handling while preserving successful-response validation, with no blocking issues identified.

No open issues 🎉

What changed in this PR

This PR makes Tests_Readme resilient to external version API outages, preventing transient network failures and rate limits from failing CI.

Changes:

Skip tests when requests return a WP_Error, including the error message in the skip reason.
Skip HTTP 429, 500, 502, 503, and 504 responses while preserving failures for other unexpected responses.
Reuse a host label with a URL fallback in diagnostic messages.

File	Description
`tests/phpunit/tests/readme.php`	Adds graceful skips and diagnostic messages for external API failures.

_{Review effort: Balanced}

lancewillett · 2026-10-26T17:29:13Z

@devmuhib009 Could you please pick this one back up when you have time?

…s_Readme Retain failures for invalid URLs, locally blocked requests, and unknown errors via assertNotWPError(). Only skip on identified external network availability failures or 429/5xx status codes, with dedicated unit tests covering both skipped and failing outcomes. See #63914.

lancewillett

The narrowed availability classification addresses the previous concern. One issue remains in the regression checks: SkippedTestError extends AssertionFailedError in PHPUnit 9.6, so the three tests expecting AssertionFailedError also pass if the helper skips. A standalone check confirmed that false pass.

Please explicitly reject SkippedTestError before accepting the expected assertion failure in the invalid-URL, blocked-request and 404 cases. Optional: avoid the HTTP wrapper’s retry sleeps for these fixed error fixtures; they add 45 seconds without exercising a changing response.

_{AI review · gpt-6}

Copilot

🟡 Changes recommended

PHPUnit 5.7 exception mismatches break tests and bypass skip-message checks, while some network failures remain unrecognized and offline mocks incur substantial retry delays.

3 open issues

Use PHPUnit 5.7-compatible exception classes · New
Recognize additional connection failure messages · New
Avoid retry delays in mocked error tests · New

_{Review effort: Balanced}

lancewillett · 2026-10-31T00:34:23Z

One additional compatibility point from Copilot: the supported PHPUnit 5.7 path uses legacy assertion exceptions; the namespaced AssertionFailedError is not an alias there. Please preserve the supported-version behavior while explicitly rejecting skips in cases that expect a failure. No broader network-message matrix is needed.

_{AI review · gpt-6}

…d use compatible assertions in Tests_Readme. See #63914. Props devmuhib009.

Copilot

🟡 Changes recommended

The skip logic misses several cURL availability failures, so they can still cause hard CI failures without regression coverage.

1 open issue

Recognize additional cURL availability failures · New

3 resolved since last review

_{Review effort: Balanced}

+			$error_message           = $response->get_error_message();
+			$is_availability_failure = false;
+
+			if ( 'http_request_failed' === $response->get_error_code() ) {


lancewillett

The response checks now distinguish expected skips from assertion failures and retain failures for invalid requests and unexpected responses. All 12 deterministic response checks pass in both single-site and multisite (27 assertions each). Live external API checks and PHPUnit 5.7 were not run locally. Current-head Copilot feedback was assessed; broader availability classification can be considered separately.

CI is green at this head. No remaining blocking finding.

_{AI review · gpt-6}

Skip recognized availability failures in the readme version checks while retaining failures for invalid requests and unexpected responses. Add deterministic response checks that distinguish skipped tests from assertion failures. Developed in: #14683 Props devmuhib, huzaifaalmesbah. See #63914. git-svn-id: https://develop.svn.wordpress.org/trunk@65176 602fd350-edb4-49c9-b593-d223f7449a82

lancewillett · 2026-11-01T17:41:37Z

PR #14683 landed in https://core.trac.wordpress.org/changeset/65176

Skip recognized availability failures in the readme version checks while retaining failures for invalid requests and unexpected responses. Add deterministic response checks that distinguish skipped tests from assertion failures. Developed in: WordPress/wordpress-develop#14683 Props devmuhib, huzaifaalmesbah. See #63914. Built from https://develop.svn.wordpress.org/trunk@65176 git-svn-id: http://core.svn.wordpress.org/trunk@64325 1a063a9b-81f0-0310-95a4-ce76da25c4cd

huzaifaalmesbah approved these changes Oct 22, 2026

View reviewed changes

lancewillett requested a balanced review from Copilot October 26, 2026 16:34

Copilot started reviewing on behalf of lancewillett October 26, 2026 16:35

View session

lancewillett requested changes Oct 26, 2026

View reviewed changes

Copilot AI reviewed Oct 26, 2026

View reviewed changes

lancewillett requested a balanced review from Copilot October 31, 2026 00:14

Copilot started reviewing on behalf of lancewillett October 31, 2026 00:15

View session

lancewillett requested changes Oct 31, 2026

View reviewed changes

Copilot AI reviewed Oct 31, 2026

View reviewed changes

Comment thread tests/phpunit/tests/readme.php Resolved

Comment thread tests/phpunit/tests/readme.php Resolved

Comment thread tests/phpunit/tests/readme.php Resolved

Tests: Extract parse_response_body, recognize connection messages, an…

ff59e50

…d use compatible assertions in Tests_Readme. See #63914. Props devmuhib009.

lancewillett requested a balanced review from Copilot November 1, 2026 16:25

Copilot started reviewing on behalf of lancewillett November 1, 2026 16:26

View session

Copilot AI reviewed Nov 1, 2026

View reviewed changes

Comment thread tests/phpunit/tests/readme.php

$error_message = $response->get_error_message();

$is_availability_failure = false;

if ( 'http_request_failed' === $response->get_error_code() ) {

lancewillett approved these changes Nov 1, 2026

View reviewed changes

lancewillett closed this Nov 1, 2026

Conversation

devmuhib009 commented Oct 22, 2026

Description

Uh oh!

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huzaifaalmesbah left a comment

Choose a reason for hiding this comment

Uh oh!

lancewillett left a comment

Choose a reason for hiding this comment

Uh oh!

lancewillett Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

lancewillett commented Oct 26, 2026

Uh oh!

lancewillett left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lancewillett commented Oct 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

lancewillett left a comment

Choose a reason for hiding this comment

Uh oh!

lancewillett commented Nov 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Oct 22, 2026 •

edited

Loading