Repository navigation
Conversation
csv.reader(..., quoting=csv.QUOTE_NONE) stops a literal " in a GBIF free-text field from making the parser swallow following rows/newlines into one garbled record. newline="" on the input open(); the csv module’s own documented recommendation, so it (not the text layer) controls line-ending interpretation. GBIF exports are always UTF-8 by spec (https://techdocs.gbif.org/en/data-use/), so we can drop the OS-locale-guessed primary_encoding=None and collapsed primary_encoding/fallback_encoding into one encoding="utf-8" parameter. Removed a dead except UnicodeDecodeError branch under the replace-mode retry too, since errors="replace" can’t raise that.
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the GBIF input-parsing changes.
No open issues 🎉
What changed in this PR
This PR updates v.in.gbif input parsing to address malformed imports caused by literal quotation marks in GBIF text fields, targeting the “not enough columns” problem reported in #1655.
Changes:
- Disables quote interpretation when reading tab-separated GBIF data.
- Uses
newline=""so the CSV reader controls line-ending handling. - Defaults to UTF-8 and simplifies the retry logic for invalid characters.
| File | Description |
|---|---|
src/vector/v.in.gbif/v.in.gbif.py |
Adjusts GBIF parsing, encoding defaults, and decoding-error handling. |
Review effort: Balanced
|
GBIF data encoding stabilized over the years. Back then, it was often trial/error to get data in; encoding and new line (often within a cell) were the cause of the issues. What I've recently seen in other GBIF workflows, encoding seems to be stable, new line (within a cell) not always. Thus, I "clean" the GBIF data before import. Maybe such a check could be added to the tool. I can't test your changes at the moment, but they look reasonable. |
That is a good idea, |
csv.reader(..., quoting=csv.QUOTE_NONE) stops a literal " in a GBIF free-text field from making the parser swallow following rows/newlines into one garbled record.
newline="" on the input open(); the csv module’s own documented recommendation, so it (not the text layer) controls line-ending interpretation.
GBIF exports are always UTF-8 by spec (https://techdocs.gbif.org/en/data-use/), so we can drop the OS-locale-guessed primary_encoding=None and collapsed primary_encoding/fallback_encoding into one encoding="utf-8" parameter. Removed a dead except UnicodeDecodeError branch under the replace-mode retry too, since errors="replace" can’t raise that.
fixes #1655