Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New runs between running git-write.js and one of the scoring scripts will fail #157

Open
foolip opened this issue Feb 22, 2023 · 2 comments

Comments

@foolip
Copy link
Member

foolip commented Feb 22, 2023

The problem is that first we do this to populate the most recent runs:

node git-write.js --max-time=300 --max-age-days=5

The in various scoring scripts, we use the wpt.fyi API to enumerate runs:

const alignedRuns = await lib.runs.fetchAlignedRunsFromServer(
products, from, to, experimental);

And then fail if any of those aren't in the "cache":

if (!localRunIds.has(run.id)) {
// If you see this, you probably need to run git-write.js or just update
// your results-analysis-cache.git repo; see the README.md.
console.error(`Run ${run.id} missing from local git repo (${date})`);
hadErrors = true;
}

This happened in https://github.com/web-platform-tests/results-analysis/actions/runs/4240081460/jobs/7368750202

This doesn't happen often, but it will happen occasionally.

A better setup would be one of:

  • Assume the cache is sufficiently up-to-date, don't try to update it, and ignore any run not in it
  • Don't run git-write.js at all, and instead handle cache misses by creating the tree from the full wptreport.json. This would work without the results-analysis-cache repo, just very slowly.
  • Get the runs from the wpt.fyi API once, ensure those are in the cache, and then run scoring scripts (storing the runs in some intermediate JSON file)
@gsnedders
Copy link
Member

Assume the cache is sufficiently up-to-date, don't try to update it, and ignore any run not in it

If we want to do this, we probably want to do substantially increase the frequency of the runs. But especially around midnight UTC there can be long delay of GitHub Actions jobs, which means even if the cache update job was hourly there would be a decent chance that we'd run the scoring update job before the cache job.

@foolip
Copy link
Member Author

foolip commented Mar 21, 2023

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants