Rendering type detection treats a failed static run as 'client only'

[ayush27316]

Crawlee 1.8.4, _adaptive_playwright_crawler.py:431:

static_run = await self._crawl_one('static', context=context, state=old_state_copy)
if static_run.result and self.result_comparator(static_run.result, pw_run.result):
    detection_result = 'static'
else:
    detection_result = 'client only'

self.rendering_type_predictor.store_result(context.request, detection_result)

If the static sub-crawler raises (connection reset, timeout, SessionError from retry_on_blocked), static_run.result is None, so this falls to else and trains the predictor with 'client only'. static_run.exception isn't checked.

What confuses me is that the browser side, six lines earlier, does the opposite:

if pw_run.exception is not None:
    raise pw_run.exception

A browser failure is never turned into a label, but a static failure is.

Was this deliberate?

What made me look was that the label sticks: once 'client only' is stored, detection_probability_recommendation drops toward detection_ratio, so the URL shape gets re-tested less often. A transient failure during one detection seems like it can keep similar URLs on the browser path for a while, with nothing logged.

Am I reading that right?

---

Possibly related, same store_result call: with the default push_data_only_comparator, if both sub-crawlers succeed but the handler extracts nothing from either (both got served a challenge page), [] == [] compares equal and the request is labeled 'static'. Hit this on a Google search URL that returns a soft-block with HTTP 200. Happy to open separately if it's a different discussion.

[janbuchar]

Hi @ayush27316 and thanks for the bug report!

A browser failure is never turned into a label, but a static failure is.

Was this deliberate?

I'd say it is, mostly, but it'd make more sense to postpone storing the result until we know that the browser path actually succeeded. Do you agree?

What made me look was that the label sticks: once 'client only' is stored, detection_probability_recommendation drops toward detection_ratio, so the URL shape gets re-tested less often. A transient failure during one detection seems like it can keep similar URLs on the browser path for a while, with nothing logged.

Am I reading that right?

Yes, good point, this is not the kind of result that should contribute towards the crawler detecting rendering type less often.

Possibly related, same store_result call: with the default push_data_only_comparator, if both sub-crawlers succeed but the handler extracts nothing from either (both got served a challenge page), [] == [] compares equal and the request is labeled 'static'. Hit this on a Google search URL that returns a soft-block with HTTP 200. Happy to open separately if it's a different discussion.

I believe that this is in fact a different discussion. From the crawler's point of view, a soft-block with HTTP 200 is a valid result. If you expect to find a non-zero amount of dataset items in each request handler run, for instance, that's something you need to express using the result_checker. Does that make sense?