test matrix really big

[williamcroberts]

@whooo our test matrix is super big, 31 jobs on the test matrix (5x6+1). It seems now it's causing things to hang in the CI. I'm looking at things to pair down the matrix.

Considering the python versions, we have 3.7, 3.8, 3.9, 3.10 and 3.11 which are all currently supported according to:

• https://endoflife.date/python

Considering tss2 versions: 'master', '2.4.0', '2.4.6', '3.0.0', '3.0.3', '3.1.0'. I want to keep master and oldest officially supported version of 2.4.0.

Looking at is_bug_fixed and _check_bug_fixed, it seems things are:

• fixed in 3.2
• backported fixes in "2.4.7", "3.0.5", "3.1.1"

I am thinking we drop:

• '2.4.6', '3.0.0', '3.0.3'

But add 3.2. So the final tss list would look like:
'master', '2.4.0', '3.1.0', '3.2.0'

master keeps us checking on current.
2.4.0 keeps our min version sane.
3.1.0 checks the major release and misses a backport fix
3.2.0 checks the most recent release

So we would get back down ro 5x4+1, 21 tests?

Thoughts on my sanity here?

[whooo]

I really like this idea.
When it comes to versions I don't have any strong preferences, but it might be good to think about which version different distribution LTS releases have, looking at the latest Ubuntu LTS is at 3.2.0, Debian stable and CentOS/RHEL9 is at 3.0.3.
Perhaps targeting the latest Ubuntu LTS release would be enough?

Also, should we look at using binaries instead of compiling the whole stack in some scenarios? The simulators might be good targets for that as most of the time is spent building the dependencies.

[williamcroberts]

I really like this idea. When it comes to versions I don't have any strong preferences, but it might be good to think about which version different distribution LTS releases have, looking at the latest Ubuntu LTS is at 3.2.0, Debian stable and CentOS/RHEL9 is at 3.0.3. Perhaps targeting the latest Ubuntu LTS release would be enough?

Good idea perhaps target or choose what's packaged.

Also, should we look at using binaries instead of compiling the whole stack in some scenarios? The simulators might be good targets for that as most of the time is spent building the dependencies.

We don't have a lot of options if we use the github runners themselves, as none of them have the deps:

• https://github.com/actions/runner-images

But we could reuse our images, but it makes testing the different versions of python harder.

[whooo]

Also, should we look at using binaries instead of compiling the whole stack in some scenarios? The simulators might be good targets for that as most of the time is spent building the dependencies.

We don't have a lot of options if we use the github runners themselves, as none of them have the deps:

* https://github.com/actions/runner-images

But we could reuse our images, but it makes testing the different versions of python harder.

Perhaps we could use https://github.com/actions/cache for some parts? I'll do some testing

[williamcroberts]

@whooo I looked at the frequency of versions in repoology:

curl https://repology.org/api/v1/project/tpm2-tss > data.json

And wrote a script to look at it:
analyze.py.txt

It outputs:

Top 5:
  - 3.2.0: 71
  - 3.0.3: 61
  - 2.3.2: 25
  - 3.1.0: 20
  - 2.4.1: 12
Oldest 5:
  - 1.2.0: 1
  - 2.4.6: 2
  - 3.0.2: 1
  - 3.0.3: 61
  - 3.2.0: 71
Common:
  - None

So considering that output and this comment below:

master keeps us checking on current.
2.4.0 keeps our min version sane.
3.1.0 checks the major release and misses a backport fix
3.2.0 checks the most recent release

I still think this looks good, their could be some argument to moving 3.1.0 to 3.0.3, but perhaps
with your caching PR we can keep it all. I see with caching enabled it's 1 hour 46mins and without is 3hr 40mins.

I'm pretty OK with caching. Why do we have PRs #385 and #387 will we need both?

[whooo]

@williamcroberts, #385 was for testing and for when I looked at why some CI flows where so slow, but it's a bit chaotic so I'm taking out the relevant parts and adding them to #387 (as well as #386 ).

I'm good with dropping 3.0.3, I think that keeping track of whatever version is in the latest Ubuntu LTS release will be enough as it's probably what most people use. We could always setup a CI job that only uses packages from the latest GH Ubuntu image.