docs: publish an agent-readable entry point

[Kush-Meta]

Coding agents fetching this documentation hub currently receive themed HTML and have no concise map of the pages or AGNTCY's separately hosted product docs. This PR makes the normal MkDocs build publish /llms.txt plus rendered Markdown for every in-repo page, so macros and includes are resolved before agents read them.

The index groups all 19 hub pages with one-line descriptions and links to the canonical Directory, SLIM, Identity, and SHADI sites. Each HTML page advertises its Markdown twin and /llms.txt in the head and has a small on-page pointer. The build fails if the index is missing, exceeds 50,000 characters, omits a page or description, contains unresolved AGNTCY macros, or links to a missing local Markdown page. The contribution guide explains how to add pages.

Validation: built the site and checked all 19 Markdown pages; checked head and on-page discovery links on all 19 HTML pages; verified the four product URLs; ran codespell, pymarkdown, Black, Python compileall, and git diff --check. A negative check confirmed that an undescribed index entry fails validation. DCO passes. Upstream Actions are currently action_required with no jobs started.

Closes #537

[keraron]

Hi, thanks for the PR. A few notes:

• the link to llms.txt is quite awkward, there should be a more elegant way to include it, or maybe even strip it completely
• some code fences lose their language. Fix: add pygments_lang_class: true under pymdownx.highlight
• expected_pages uses docs.rglob("*.md"), so any non-nav Markdown under docs/ breaks the build