[abdel792]

Link to issue number:

Replaces #41.

Summary of the issue:

While upgrading an existing add-on using standard Git commands remains fully supported, some add-on developers prefer an automated and dedicated tool to streamline the process.
Manual step-by-step migration of metadata (from buildVars.py to pyproject.toml) and file merges can require extra care and verification.
Providing an automated script offers a convenient alternative for developers who wish to perform these updates quickly and with built-in AST-based metadata handling.

Description of developer facing changes:

• Introduces the syncAddonTool package at the repository root as an automated alternative to Git-based manual upgrades for add-on developers.
• Requires tomlkit for parsing and merging pyproject.toml configurations (can be run without prior installation via uv run --no-project --with tomlkit).
• Includes a PyInstaller specification file (syncAddonTool/syncAddonTool.spec) to build a standalone executable (syncAddonTool.exe).
• Protects template-specific files, including .github/workflows/tests.yml and tests/, ensuring they are excluded from add-on synchronization.
• Excludes both syncAddonTool and the tests/ directory from static analysis in pyproject.toml (ruff and pyright).
• Adds a comprehensive verification infrastructure including both a unit test suite (tests/unit/template/test_syncAddonTool.py) and an End-to-End (E2E) CLI integration test suite (tests/e2e/template/test_syncAddonTool.py).
• Adds docs/testing.md to provide unified execution guidelines for both unit and End-to-End (E2E) tests.

Description of development approach:

• Synchronization Engine Creation: Implemented syncAddonTool using an AST-aware approach and tomlkit to seamlessly migrate legacy buildVars.py metadata or modern AddonInfo setups into pyproject.toml.
  It supports timestamped backups, custom exclusions via .addonmergeignore, and includes .github/workflows/tests.yml in PROTECTED_ELEMENTS.
• Standalone Binary Support: Added syncAddonTool/syncAddonTool.spec for building a standalone executable using uv run --with pyinstaller pyinstaller syncAddonTool/syncAddonTool.spec, eliminating external Python environment requirements for end users.
• Linter Exclusion Rules: Configured pyproject.toml to exclude syncAddonTool and tests/ from ruff and pyright checks.
• Verification & E2E Suite: Built a dual test infrastructure featuring unit tests (tests/unit/template/test_syncAddonTool.py) for internal AST/synchronization logic, and an E2E test suite (tests/e2e/template/test_syncAddonTool.py) executing isolated subprocess workflows, CLI flags, buildVars migrations, and PyInstaller binary packaging.
• Documentation Integration: Added docs/testing.md to detail CLI usage, available options, and instructions for executing both unit and E2E test suites.

Testing strategy:

Validated the synchronization script and unit test suite locally using uv:

1. Executed syncAddonTool across various CLI modes:

   • Standard Mode (inside add-on repo, -ad is optional):

     uv run --no-project --with tomlkit python syncAddonTool

   • External Directory Mode:

     uv run --no-project --with tomlkit python -m syncAddonTool -ad /path/to/my-nvda-addon

   • Dry Run Simulation:

     uv run --no-project --with tomlkit python syncAddonTool --dry-run

2. Validated Standalone Executable Generation via PyInstaller:

   uv run --with pyinstaller pyinstaller syncAddonTool/syncAddonTool.spec

   Executed dist/syncAddonTool.exe -ad /path/to/my-nvda-addon --dry-run and confirmed expected behavior.

3. Verified that backup folders (<addon>_bak_<timestamp>) were generated, .addonmergeignore rules were respected, and protected elements (including .github/workflows/tests.yml and tests/) were not pushed to target add-on repositories.

4. Executed the complete verification test suite (both unit and E2E tests) via unittest:

   uv run python -m unittest discover -s tests -v

   Confirmed all tests (10 unit tests and 10 E2E tests) passed successfully.

5. Confirmed ruff and pyright ignore syncAddonTool and tests/.

Known issues with pull request:

None.

---

Code Review Checklist

• Testing: Tested manually across diverse execution paths and validated via local unit tests.
  Steps to reproduce are detailed in the testing strategy.
• Compatibility: Developer-facing infrastructure tool; introduces no breaking changes to existing repository behaviors.
• Documentation: Technical developer documentation has been explicitly updated.
• UX of all users considered: Not applicable (Developer-facing infrastructure tool only).
• Security precautions taken: Not applicable (Runs locally in isolated temporary directories during development).

[Copilot]

Pull request overview

Adds a developer-facing synchronization tool to help NVDA add-on authors update their repositories to the latest AddonTemplate structure, including AST-aware metadata handling and TOML merging, with accompanying unit tests and documentation updates.

Changes:

• Introduces syncAddonWithTemplate.py to sync template infrastructure while merging buildVars.py and pyproject.toml.
• Adds unit tests for key merge/formatting behavior and a small template sanity suite.
• Updates dependency groups/docs to support running the sync tool and the unit test suite.

Reviewed changes

Copilot reviewed 8 out of 10 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
syncAddonWithTemplate.py	New sync/merge engine (AST + tomlkit) for updating add-on repos from the template.
tests/unit/test_syncAddonWithTemplate.py	Unit tests for metadata extraction, buildVars merge, TOML indentation, and dependency merging.
tests/unit/template/sanity.py	Minimal sanity tests intended to validate CI test discovery/execution.
docs/managementFromGit/updatingExistingAddons.md	Expanded guidance for updating add-ons (automated tool + manual merge).
docs/unitTesting.md	Updated instructions for running the unit tests and describing the new suites.
pyproject.toml	Adds tomlkit, updates pyright, and excludes the new sync script/tests from ruff/pyright.
uv.lock	Locks tomlkit and bumps pyright to match pyproject.toml.
tests/__init__.py	Updates package docstring.
tests/unit/__init__.py	Updates package docstring.
.addonmergeignore	Present at repo root (empty in this PR context).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[abdel792]

Hi @seanbudd,
Thanks for your feedback! I have updated the branch accordingly:

• Docstrings adjustment: I moved the test infrastructure docstring from root unit tests to the template submodule and restored the root tests/__init__.py docstring to reflect all test types (system and unit).
• Sanity test removal: I removed the obsolete sanity.py test file.

[abdel792]

@Copilot,
Here are the updates I made regarding your remarks on normalization and .addonmergeignore:

• Package Name Normalization: I replaced underscores with hyphens in getBasePackageName and legacyToolingBases to consistently handle equivalent Python package name formats during dependency merges.
• Dynamic Bootstrapping & Self-Exclusion: Instead of adding .addonmergeignore to protectedElements, I updated syncAddonWithTemplate.setupAddonMergeIgnore to bootstrap .addonmergeignore from the template on the first sync if it is absent locally, ensuring it includes its own self-exclusion. I prefer this approach as it allows the file to self-reference and manage its own persistence dynamically.

[abdel792]

@Copilot,
Regarding your comment on the load_tests implementation in test_syncAddonWithTemplate.py:

• Robust Test Sorting: I updated load_tests to use an orderIndex lookup dictionary with orderIndex.get(a, defaultOrder) instead of methodOrder.index().
• Error Prevention: I added this to prevent potential ValueError exceptions if inherited or dynamic test methods are present during test execution.

[abdel792]

@Copilot,
Regarding your feedback on the documentation:

• Unit Test Path Fix: I fixed the incorrect path for test_syncAddonWithTemplate.py in docs/unitTesting.md.
• CLI Usage & Dependency Clarification: I clarified the tomlkit installation requirement and corrected the no-argument command example in docs/managementFromGit/updatingExistingAddons.md.

[abdel792]

Hi @seanbudd, @Copilot,

Regarding the test coverage for .addonmergeignore:

• Unit Tests: I added testSetupAddonMergeIgnore and testAddonMergeIgnore to test_syncAddonWithTemplate.py to verify both the creation/bootstrapping behavior when missing and the actual file exclusion during synchronization.
• Documentation: I documented both new test methods in docs/unitTesting.md.

[abdel792]

Hi @seanbudd,

When running pyright on version 1.1.407 via uv (e.g., uv run pyright addon), the underlying Python wrapper package (pyright-python) checks for upstream NPM release updates and systematically outputs a warning to stderr before executing the type checker:

WARNING: there is a new pyright version available (v1.1.407 -> v1.1.411).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`

I bumped the lock version to 1.1.411 in pyproject.toml to eliminate this noise during local check runs and developer workflows.

Let me know if you would still prefer me to revert it to keep the PR scope strictly minimal!

[abdel792]

Thanks for the review and all these helpful suggestions, @seanbudd!

I have addressed the quick fixes and documentation updates:

• Restored the exact casing for AddonTemplate in docs/managementFromGit/updatingExistingAddons.md.
• Removed the detailed test suite section from docs/unitTesting.md (relying on test docstrings instead).
• Fixed the missing newline at the end of test_syncAddonWithTemplate.py.

Regarding your suggestion to split syncAddonWithTemplate.py into submodules: I completely agree, it's a great idea that will make the code much cleaner and easier to maintain!

I'm currently refactoring the code into submodules, writing the requested additional test cases, and running all necessary tests. I will address all of your remaining feedback and suggestions throughout this process and post a full update over the weekend.

I'll leave the PR in draft mode until everything is ready for review.

[abdel792]

Sean, regarding your comment on legacyToolingBases:

I'm currently implementing a solution for this and will push it in my next commit over the weekend, as I need to finish up some work commitments first.

The previous legacyToolingBases set was indeed handling this, but as you pointed out, maintaining such a hardcoded list long-term is cumbersome.

Replacing it with a dynamic lookup alongside a minimal mapping dictionary for non-supported/replaced dependencies (like pre-commit) makes maintenance much simpler.

The updated logic now dynamically collects all tooling packages directly from the template's dependencies and dependency-groups in pyproject.toml.

Additionally, for deprecated tools that were replaced in the template (such as pre-commit being replaced by prek), I introduced a lightweight dictionary (REPLACED_PACKAGES = {"pre-commit": "prek"}).

This maps replaced tools to their modern template equivalents so they aren't accidentally preserved as custom dependencies.

Here is a snippet of how the dynamic collection works:

# Dynamically collect all template packages (dependencies + dependency-groups)
templateBases: set[str] = {
	getBasePackageName(d) for d in tplDeps if isinstance(d, str)
}
if "dependency-groups" in mergedData and isinstance(mergedData["dependency-groups"], MutableMapping):
	for grp in mergedData["dependency-groups"].values():
		if isinstance(grp, (list, MutableSequence)):
			for grpItem in grp:
				if isinstance(grpItem, str):
					templateBases.add(getBasePackageName(grpItem))

[abdel792]

Hi @seanbudd and @nvdaes,

I have reviewed and processed all 12 Copilot review findings for this pull request.

Each of the 11 code-based corrections has been implemented via a precise, dedicated commit.

• Fixed the escaping of control characters in generated double-quoted literals.
• Handled annotated "addon_info" assignments properly.
• Preserved or rejected non-literal metadata expressions.
• Preserved imports and definitions for global expressions.
• Honored exclusions during configuration merges.
• Restricted package matching to dependency arrays.
• Removed existing static versions from merged dynamic metadata.
• Preserved "i18nSources" during build variable extraction.
• Matched directory exclusions using trailing separators.
• Populated repository metadata before merging template defaults.
• Used temporary paths for packaging tests.
• Addressed the suggestion concerning "git clean -f" during synchronization.

This finding was resolved with a reason.

The documentation referencing "git clean -f" pre-exists on the master branch and is out of scope for the current pull request.

This update must be handled in a separate, dedicated PR.

What are your thoughts on this?

Thanks @nvdaes for offering your help!

Feel free to run some tests on your side whenever you like.

[nvdaes]

Hi @abdel792 , I've performed the test suite, and all test pass! Thanks for this impressive work.

[abdel792]

Hi @seanbudd and @nvdaes,

@seanbudd You are completely right, and I apologize for initially marking this as "won't fix" by mistake.

This section is indeed part of the documentation for syncAddonTool.

Using "git clean -f" can accidentally delete newly generated untracked files (such as a new "pyproject.toml" in legacy add-ons) before they are staged.

I have just pushed a commit to fix this and update the documentation to use "git status --short" instead.

@nvdaes Thank you so much for offering your help, your testing, and your encouragement!

I also wanted to take a moment to express my sincere gratitude to you: you are the one who introduced me to creating NVDA add-ons in the first place, and I am still deeply grateful to you for that.

It is a true honor for me to contribute to the NVDA project with my modest contributions, and I am really proud to be part of this wonderful community!

[nvdaes]

Hi @abdel792 , thanks for this amazing work. I've reviewed it, and I think it's almost perfect.

I need to use it in the real life for my urlShortener add-on. I've copied the syncTool package in the repo. After running uv run --no-project --with tomlkit python syncAddonTool, the syncronization works well, and the backup is created in the parent folder of the repo.
But, when I run: uv run scons, or uv sync, I get the following error:

Using CPython 3.13.13
error: The Python request from .python-version resolved to Python 3.13.13, which is incompatible with the project's
Python requirement: ==3.11.* (from project.requires-python)
Use uv python pin to update the .python-version file to a compatible version

Can this be fixed in this PR?

Also, in the workflow to run tests with GitHub Actions, Python 3.13 is required. Can we require the version listed in the .python-version file instead?

Lastly, I'd like to create a scheduled job to update the template automatically, in the .github/workflows folder. What do you think?

Thanks so much!

[nvdaes]

Hi again @abdel792 , for any reason, seems that the add-on description in buildVars.py is overriden with the default value: Description for the add-on.
Thanks for this work!

[abdel792]

Hi @nvdaes,

Thanks for your thorough review and testing!

Here is how your points have been addressed:

1. Python version conflict (.python-version)

This is now resolved.

The .python-version file has been added to protectedElementsSet in syncAddonTool/engine.py.

Although you could also configure it manually by adding it to your .addonmergeignore file, having it protected by default is much safer to prevent template updates from overwriting your add-on's specific Python requirement.

2. GitHub Actions workflow version

Resolved!

The .github/workflows/test.yml file now uses python-version-file: '.python-version' to dynamically pick up the required Python version, ensuring full compliance with your add-on's requirements.

3. Scheduled workflow job

That's a great idea!

That could definitely make a great subject for a future PR if you'd like to implement it.

4. Overridden metadata fields

This issue has also been fully fixed.

Note that not only addon_description, but addon_summary and addon_changelog were also affected by this bug.

The AST parsing logic has been corrected to properly handle gettext calls (_()) in the template, ensuring your custom values are preserved instead of being overwritten by defaults.

Thanks again for your help and feedback!

[nvdaes]

Hi @abdel792 , thanks for this update!
Anyway, I'm still finding an issue. My second surname, which contains an "í" character, is not properly writen in buildVars.py. It's replaced with "\xed".R.
Thanks.
Hope this can be fixed, and I think that I will be able to approve this P

[abdel792]

Hi @nvdaes,

I have provided an update to the tool that addresses almost all of your recent requests.

To validate these changes, I cloned the "urlShortener" repository for testing purposes.

It turns out that in the pyproject.toml file of this wonderful add-on, setuptools uses a dynamic build module configuration (py-modules for buildVersion).

In principle, to use the uv sync command in "urlShortener", the NVDA buildVersion.py module must be placed at the root of the repository.

Therefore, after synchronizing the add-on with the template, you will need to place buildVersion.py at the root of the repository to execute uv sync without issues.

At that point, it is preferable to use the Python version from the template, which is Python 3.13.

This is because Python 3.13 uses precompiled 64-bit binary wheels that Python 3.11 does not support, thereby preventing compilation failures on 32-bit systems.

In principle, the tool now handles all of these technical details automatically.

Here is a summary of the recent modifications I have completed:

Python Version Synchronization

• Enforced automatic synchronization of the "requires-python" field from the reference template.
• Guaranteed the adoption of Python 3.13 for precompiled wheels to prevent compilation failures on 32-bit systems.

Dependency Management and Migration

• Improved the mergeDependencyLists algorithm to ensure "requires" fields are processed as full dependency lists, avoiding version conflicts and duplicates.
• Implemented automated cleanup in processDependencyGroupsMigration to purge redundant string-based dependencies from the "dev" group when covered by included groups like build, lint, and l10n.

Protected Files and String Formatting

• Removed the ".python-version" path from protectedElementsSet to allow seamless template synchronization and ensure easy upgrades to recent Python releases like Python 3.13.
• Updated formatStringLiteral in buildVarsSync.py to preserve raw UTF-8 Unicode characters instead of escaping them.
• Ensured author names and descriptions remain fully readable while properly escaping control characters and quotes.

Please feel free to run any additional tests you might need before confirming your approval for the merge.