Manually create a sitemap.xml for docs SEO

[DimedS]

Description

Draft PR currently, checking RTD build

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[astrojuanlu]

👀 So this was it...

[DimedS]

Unfortunately, I checked the build, and it didn’t help. Something is still overwriting the sitemap.xml, even with that line included.

[DimedS]

After adding html_extra_path = [str(here / "sitemap.xml")] to conf.py, I observed the following post-build:

• The custom sitemap.xml is accessible at: https://kedro--4261.org.readthedocs.build/en/4261/sitemap.xml
• The autogenerated sitemap.xml is still located at: https://kedro--4261.org.readthedocs.build/sitemap.xml and is likely the version that will be used at https://docs.kedro.org/sitemap.xml.

I also tried adding a custom robots.txt, but the autogenerated version still appears at https://kedro--4261.org.readthedocs.build/robots.txt. This makes it difficult to verify the effect in this build, and it’s possible that the custom sitemap.xml and robots.txt will only take effect at a later stage, potentially being used at https://docs.kedro.org.