Fix agent source registration and retire legacy Gemini assessment pipeline

[xrendan]

Summary

The commitment evaluation agent (AgentEvaluateCommitmentJob) has been unable to register any new evidence since roughly March. Every canada.ca registration timed out and parl.ca pages were rejected outright, so the agent could only cite source URLs that already existed in the database. This PR fixes the fetch path, moves HTML extraction to the hosted defuddle worker, upgrades the agent model, and removes the old Gemini criteria-assessment pipeline that was writing the same fields as the agent.

Root causes found

1. canada.ca blocks the User-Agent BuildCanada-Tracker/1.0 (connection reset over HTTP/2, indefinite hang over HTTP/1.1). The pages endpoint hit its 20s read timeout and returned an unhandled 500. The RSS entry scraper sent http.rb's default agent, which is blocked the same way.
2. parl.ca was not allowlisted in the pages endpoint, although the system prompt, agent CLAUDE.md and WebFetch permissions all tell the agent it may use LEGISinfo.
3. defuddle CLI truncates piped stdout at 64KB, so any page over that size (departmental plans, budgets) failed with a JSON parse error.
4. Timeouts and connection errors escaped as 500s rather than a message the agent could act on.

Changes

• PageFetcher (new) wraps http.rb with a browser-compatible crawler User-Agent that canada.ca accepts. Used by Api::Agent::PagesController and Entry#fetch_data!.
• Pages endpoint: *.parl.ca allowed; HTTP::TimeoutError, HTTP::ConnectionError, SSL and DNS errors and parse failures return 502 with a readable error.
• Defuddle is now a client for the hosted worker at deffudler.svc.canadasbuilding.com (POST /api/convert with the already-fetched HTML). Same [markdown, html] return shape; callers pass the page URL. Requires DEFUDDLE_API_KEY; DEFUDDLE_API_URL is optional. The worker side (accept html in the request, fix its own User-Agent) is already deployed.
• Agent model defaults to claude-sonnet-5 in both jobs. They previously disagreed (claude-sonnet-4-6 for commitments, claude-opus-4-6 for entries). Sonnet 5 is cheaper than Sonnet 4.6.
• Matches created by the agent are marked assessed on creation.
• Removed CommitmentAssessmentCronJob (6-hourly), CommitmentAssessmentJob, CriterionAssessor, CommitmentStatusDeriver, and lib/tasks/reevaluate.rake; backfill phase 5 and a stale rake hint dropped. This Gemini pipeline re-wrote criteria statuses and evidence notes that the agent also owns.
• Chat loads unknown STI types as plain Chat so the ~13k historical rows from the removed classes still open in Avo.

Deployment notes

• Set DEFUDDLE_API_KEY on both the web and worker containers (passthrough added to docker-compose.prod.yml). Without it, page registration and entry scraping fail fast with a clear error.
• The Dockerfile still installs Node and defuddle-cli, and package.json lists defuddle. Nothing uses them now; left for a follow-up.
• agent/run_batch.sh and agent/.env.example still reference the removed Python agent and are dead; also left for a follow-up.

Testing

Local, against a copy of the production database and the deployed worker, with no defuddle CLI on PATH:

• POST /api/agent/pages/fetch now returns 200 for a canada.ca news release (was 500 after 20s), the 300KB ESDC departmental plan (was 500 after 20s, then a 64KB truncation error), a LEGISinfo bill page (was 422), and a Canada Gazette index. An unresolvable host returns 502 with a message.
• AgentEvaluateCommitmentJob.perform_now on three commitments (2365, 2392, 2394): every API call 2xx. On the submarine commitment the agent found and correctly sourced the July 2026 preferred-supplier announcement; on the drone-fleet commitment it registered five new sources and created four events. Status decisions followed the evidence hierarchy in all three runs.
• Rails.application.eager_load! succeeds; bin/rails -T lists the remaining rake tasks; a Chat row typed CriterionAssessor loads as Chat with its messages.

Not run locally: bundle exec rubocop and bin/rails test. This machine has Ruby 4.0.2 and the project pins 3.4.7; RuboCop's parser and Minitest do not load under 4.0.2. Relying on CI for both.