Repository navigation
Conversation
process_disclosure re-fetched the registrant's summary page for every disclosure, so a registrant with N disclosures cost 1 + 2N requests. It now takes the parsed summary from its caller. The backfill passes the one it already holds; the weekly run passes the one it fetched live, or, for a prior year served from the summary cache, fetches it once when the first unprocessed disclosure needs it (none when nothing is new). Closes #101 Verified: backfill from the local archive into empty emulators, old vs new code, one worker: 2016 (20 registrants) summary fetches 59 -> 20, 2011 (1,820 registrants) 4,517 -> 1,820, disclosure fetches unchanged; all 1,064 and 6,749 documents identical apart from fetchedAt and the order of the clients list, which also varies between two runs of the old code. Live weekly run, 2026 limit 5: summary GETs 10 -> 5, 152 documents identical. Scraper unit tests pass, including new per-registrant fetch counts. Also with 8 workers, full 2011 from the archive: two runs of the old code and one of the new write 6,749 identical documents (clients list order ignored), summary fetches 4,592 / 4,586 -> 1,820. Not verified: the weekly prior-year cache-hit path against the live portal (unit tests only). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The lobbying scraper fetched a registrant's summary page to list its disclosures, then fetched the same page again for every disclosure it processed, so a registrant with N disclosures cost 1 + 2N requests to the Secretary of the Commonwealth's portal instead of 1 + N. Each disclosure now reuses the summary parsed once per registrant: the backfill passes the one it already holds, and the weekly run passes the one it fetched live, or fetches it once, only when a new disclosure needs it, for prior years served from the summary cache. In archive runs of the backfill, summary fetches fell by 60–66% (full 2011: 4,517 to 1,820) and the documents written were identical; a small live weekly run showed 10 to 5.
Closes hyphacoop#101
What's in it
🤖 Generated with Claude Code