Skip to content

v.in.gbif: solves problem of "not enough columns" at import - #1908

Open
ecodiv wants to merge 4 commits into
OSGeo:grass8from
ecodiv:v_in_gbif
Open

ecodiv wants to merge 4 commits into
OSGeo:grass8from
ecodiv:v_in_gbif

Conversation

@ecodiv

@ecodiv ecodiv commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

csv.reader(..., quoting=csv.QUOTE_NONE) stops a literal " in a GBIF free-text field from making the parser swallow following rows/newlines into one garbled record.

newline="" on the input open(); the csv module’s own documented recommendation, so it (not the text layer) controls line-ending interpretation.

GBIF exports are always UTF-8 by spec (https://techdocs.gbif.org/en/data-use/), so we can drop the OS-locale-guessed primary_encoding=None and collapsed primary_encoding/fallback_encoding into one encoding="utf-8" parameter. Removed a dead except UnicodeDecodeError branch under the replace-mode retry too, since errors="replace" can’t raise that.

fixes #1655

csv.reader(..., quoting=csv.QUOTE_NONE) stops a literal " in a GBIF free-text field from making the parser swallow following rows/newlines into one garbled record.

newline="" on the input open(); the csv module’s own documented recommendation, so it (not the text layer) controls line-ending interpretation.

GBIF exports are always UTF-8 by spec (https://techdocs.gbif.org/en/data-use/), so we can drop the OS-locale-guessed primary_encoding=None and collapsed primary_encoding/fallback_encoding into one encoding="utf-8" parameter. Removed a dead except UnicodeDecodeError branch under the replace-mode retry too, since errors="replace" can’t raise that.
@ecodiv
ecodiv requested review from hellik and a balanced review from Copilot November 14, 2026 21:04
@ecodiv ecodiv self-assigned this Nov 14, 2026
@ecodiv ecodiv added the bug Something isn't working label Nov 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the GBIF input-parsing changes.

No open issues 🎉

What changed in this PR

This PR updates v.in.gbif input parsing to address malformed imports caused by literal quotation marks in GBIF text fields, targeting the “not enough columns” problem reported in #1655.

Changes:

  • Disables quote interpretation when reading tab-separated GBIF data.
  • Uses newline="" so the CSV reader controls line-ending handling.
  • Defaults to UTF-8 and simplifies the retry logic for invalid characters.
File Description
src/​vector/​v.in.gbif/​v.in.gbif.py Adjusts GBIF parsing, encoding defaults, and decoding-error handling.

Review effort: Balanced

@hellik

hellik commented Nov 14, 2026

Copy link
Copy Markdown
Member

GBIF data encoding stabilized over the years. Back then, it was often trial/error to get data in; encoding and new line (often within a cell) were the cause of the issues. What I've recently seen in other GBIF workflows, encoding seems to be stable, new line (within a cell) not always. Thus, I "clean" the GBIF data before import.

Maybe such a check could be added to the tool.

I can't test your changes at the moment, but they look reasonable.

@ecodiv

ecodiv commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

GBIF data encoding stabilized over the years. Back then, it was often trial/error to get data in; encoding and new line (often within a cell) were the cause of the issues. What I've recently seen in other GBIF workflows, encoding seems to be stable, new line (within a cell) not always. Thus, I "clean" the GBIF data before import.

Maybe such a check could be added to the tool.

That is a good idea, I'll add it I added a check - when found, it will drop the record with newline in cell with warming.

Comment thread src/vector/v.in.gbif/v.in.gbif.py Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] r.vect.stats: not enough columns

3 participants