Check URL Updater issues

[jan-vcapgemini]

We are logging a summary of our URL Updater. It seems like a bunch of downloads failed. We need to check what is causing this.

2024-11-18T04:05:03.4932538Z android-studio/android-studio versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 3 failed, 1530 succeeded, 1533 total, 0.20% error
2024-11-18T04:05:03.4933870Z aws/aws versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 1 failed, 1608 succeeded, 1609 total, 0.06% error
2024-11-18T04:05:03.4935076Z az/az versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 13 failed, 64 succeeded, 77 total, 16.88% error
2024-11-18T04:05:03.4936299Z docker/docker versions added: 0 failed, 87 succeeded, 87 total, 0.00% error - versions verified: 0 failed, 0 succeeded, 0 total, 0.00% error
2024-11-18T04:05:03.4937561Z dotnet/dotnet versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 67 failed, 1509 succeeded, 1576 total, 4.25% error
2024-11-18T04:05:03.4938961Z eclipse/cpp versions added: 0 failed, 10 succeeded, 10 total, 0.00% error - versions verified: 12 failed, 84 succeeded, 96 total, 12.50% error
2024-11-18T04:05:03.4940268Z eclipse/jee versions added: 0 failed, 10 succeeded, 10 total, 0.00% error - versions verified: 12 failed, 84 succeeded, 96 total, 12.50% error
2024-11-18T04:05:03.4941542Z eclipse/eclipse versions added: 0 failed, 10 succeeded, 10 total, 0.00% error - versions verified: 11 failed, 86 succeeded, 97 total, 11.34% error
2024-11-18T04:05:03.4942809Z gcloud/gcloud versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 0 failed, 947 succeeded, 947 total, 0.00% error
2024-11-18T04:05:03.4944084Z gcviewer/gcviewer versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 1 failed, 11 succeeded, 12 total, 8.33% error
2024-11-18T04:05:03.4945341Z gh/gh versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 108 failed, 632 succeeded, 740 total, 14.59% error
2024-11-18T04:05:03.4946601Z graalvm/community versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 0 failed, 52 succeeded, 52 total, 0.00% error
2024-11-18T04:05:03.4947884Z graalvm/graalvm versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 99 failed, 109 succeeded, 208 total, 47.60% error
2024-11-18T04:05:03.4949266Z gradle/gradle versions added: 0 failed, 144 succeeded, 144 total, 0.00% error - versions verified: 0 failed, 144 succeeded, 144 total, 0.00% error
2024-11-18T04:05:03.4950537Z helm/helm versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 107 failed, 610 succeeded, 717 total, 14.92% error
2024-11-18T04:05:03.4951783Z intellij/intellij versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 0 failed, 396 succeeded, 396 total, 0.00% error
2024-11-18T04:05:03.4953189Z jasypt/jasypt versions added: 0 failed, 1 succeeded, 1 total, 0.00% error - versions verified: 0 failed, 16 succeeded, 16 total, 0.00% error
2024-11-18T04:05:03.4954436Z java/java versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 50 failed, 196 succeeded, 246 total, 20.33% error
2024-11-18T04:05:03.4955709Z jenkins/jenkins versions added: 0 failed, 163 succeeded, 163 total, 0.00% error - versions verified: 0 failed, 162 succeeded, 162 total, 0.00% error
2024-11-18T04:05:03.4956951Z jmc/jmc versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 6 failed, 18 succeeded, 24 total, 25.00% error
2024-11-18T04:05:03.4958183Z kotlinc/kotlinc versions added: 0 failed, 26 succeeded, 26 total, 0.00% error - versions verified: 12 failed, 32 succeeded, 44 total, 27.27% error
2024-11-18T04:05:03.4959617Z kotlinc-native/kotlinc-native versions added: 0 failed, 17 succeeded, 17 total, 0.00% error - versions verified: 68 failed, 100 succeeded, 168 total, 40.48% error
2024-11-18T04:05:03.4961017Z lazydocker/lazydocker versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 50 failed, 159 succeeded, 209 total, 23.92% error
2024-11-18T04:05:03.4962267Z mvn/mvn versions added: 0 failed, 37 succeeded, 37 total, 0.00% error - versions verified: 0 failed, 0 succeeded, 0 total, 0.00% error
2024-11-18T04:05:03.4963444Z mvn/mvn versions added: 0 failed, 13 succeeded, 13 total, 0.00% error - versions verified: 0 failed, 0 succeeded, 0 total, 0.00% error
2024-11-18T04:05:03.4964651Z node/node versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 49 failed, 2507 succeeded, 2556 total, 1.92% error
2024-11-18T04:05:03.4965853Z npm/npm versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 0 failed, 481 succeeded, 481 total, 0.00% error
2024-11-18T04:05:03.4967189Z oc/oc versions added: 0 failed, 398 succeeded, 398 total, 0.00% error - versions verified: 1 failed, 1193 succeeded, 1194 total, 0.08% error
2024-11-18T04:05:03.4968451Z pgadmin/pgadmin versions added: 0 failed, 11 succeeded, 11 total, 0.00% error - versions verified: 3 failed, 21 succeeded, 24 total, 12.50% error
2024-11-18T04:05:03.4969682Z pip/pip versions added: 0 failed, 8 succeeded, 8 total, 0.00% error - versions verified: 0 failed, 8 succeeded, 8 total, 0.00% error
2024-11-18T04:05:03.4970916Z python/python versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 300 failed, 634 succeeded, 934 total, 32.12% error
2024-11-18T04:05:03.4972196Z quarkus/quarkus versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 88 failed, 296 succeeded, 384 total, 22.92% error
2024-11-18T04:05:03.4973476Z docker/rancher versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 55 failed, 146 succeeded, 201 total, 27.36% error
2024-11-18T04:05:03.4974727Z sonar/sonar versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 69 failed, 143 succeeded, 212 total, 32.55% error
2024-11-18T04:05:03.4976010Z terraform/terraform versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 7 failed, 877 succeeded, 884 total, 0.79% error
2024-11-18T04:05:03.4977290Z tomcat/tomcat versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 120 failed, 1 succeeded, 121 total, 99.17% error
2024-11-18T04:05:03.4978526Z vscode/vscode versions added: 0 failed, 0 succeeded, 0 total, 0.00% error - versions verified: 47 failed, 733 succeeded, 780 total, 6.03% error
2024-11-18T04:05:03.4979404Z END OF FINAL URL UPDATER REPORT

[jan-vcapgemini]

The tools with the highest error rate tomcat and graalvm seem to have the same issue in common. Both tools use the GithubUrlUpdater and try to fetch the version numbers from the official github tags. Unfortunately the tag versions and the available downloads are not synchronized e.g. we can find tag versions for tomcat 7.0.7 but no download URLs on the official repository. Instead apache offers a download URL for tomcat 7.0.70.

[jan-vcapgemini]

Why is kotlin using the WebsiteUrlUpdater instead of the GithubUrlUpdater?
Instead we could use: https://api.github.com/repos/JetBrains/kotlin/releases
Follow the assets_url and grab the browser_download_url with content_types: "application/x-gzip" and "application/zip" from f.e. https://api.github.com/repos/JetBrains/kotlin/releases/131180439/assets
We would need a GithubUrlUpdater for releases instead of tags then though.

The same could be done for graalvm: https://api.github.com/repos/graalvm/graalvm-ce-builds/releases

[jan-vcapgemini]

For tomcat we could use the download URLs from: https://github.com/apache/tomcat/tags instead of the official apache ones.
F.e. https://github.com/apache/tomcat/archive/refs/tags/7.0.7.tar.gz