Split tests to core/providers/task-sdk/integration/system

[potiuk]

The tests execution was traditionally using single "breeze testing tests" command and you could select which test to run via TEST_TYPE.

However the recent move of providers and adding task_sdk necessitates splitting the tests commands into separate commands for core, providers, task_sdk, helm, integration and system.

This is done via introducing "TEST_GROUP" - which determines which group of tests is being executed, and dedicated testing command for each of the groups - with "db" and "non-db" variants where applicable.

Cleanup and small refactoring has been done to make it easier to reason about parameters passed down from the command line to docker and in-container pytest command.

Related: #42632

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

OK. I moved it here from #43965 .

That one should be ready for review. I got it green, all the tests are nicely split between core and providers. I will be adding a little more diagnostics (Right now it is not obvious if the successful tests are doing what they are supposed to do) and running it for all versions of Python etc. to check that everythong is as expected + I will add runnig system tests - but other than that, I think it is in the shape that is pretty much ready to merge (and for sure ready to review).

Leter we can run a number optimizations, but the way it is and after I check it for all typos and run in multiple versions, we should be ready to go. I am quite away next week (in Ireland) - and will be less available again in the next week, but I am confident it's pretty stable (but will be able to fix things when on/off and with @gopidesupavan @romsharon98 @shahar1 @amoghrajesh @tirkarthi @ashb @kaxil @pierrejeambrun @vincbeck @o-nikolas and others who contributed recently, I think this way of running tests should be easier to reason about and fix.

Free screenshots to show the current state:

The "core" and "providers" tests now run in completely separate now - following the 'Stage 1` of #42632 (comment):

List of tests commands:

One thing I am not sure here is naming. Should it be:

• integration-providers-tests

or

• providers-integration-tests

Both have it's rationale, it's mostly about whether we want to group them by "type" or "what sub-system they are working on". Would love to have other's opinion on that.

Any and all comments are welcome! Yes. It's huge, but a lot of it is auto-generated and you can just mark it as "read" and it will go away.

[ashb]

Just so I understand the plan, this splits out the running of tests, but doesn't change to code location which will be a future PR?

[ashb]

Interface question should the db/non-db be a flag that operates like a filter, so something like breeze testing core-tests --non-db etc?

Of particular note, I imagine that soon (as part of TaskSDK work) all the db unit tests for the providers will go away/become non-db tests. I don't know if that changes your thinking at all

[ashb]

+6,413 / −3,614 😱

Oh, this is mostly in the SVG images isn't it

[vincbeck]

Interface question should the db/non-db be a flag that operates like a filter, so something like breeze testing core-tests --non-db etc?

Of particular note, I imagine that soon (as part of TaskSDK work) all the db unit tests for the providers will go away/become non-db tests. I don't know if that changes your thinking at all

I agree, a flag to filter db/non-db tests would make more sense to me instead of having separate commands

[potiuk]

Oh, this is mostly in the SVG images isn't it

Yeah. It's definitely less code than it was :)

[potiuk]

@ashb

Interface question should the db/non-db be a flag that operates like a filter, so something like breeze testing core-tests --non-db etc?

@vincbeck

I agree, a flag to filter db/non-db tests would make more sense to me instead of having separate commands

OK. That's the 3rd similar comment after @o-nikolas - and I explained the reasons in: #43965 (comment)

So if everyone around you seems to be mad, maybe it's you :)

It's really additional level of abstraction for the base command that makes it easier to run and has some defaults set differently, but it is all possible to be done with the base command - it's just a matter to pass a few more parameters. And it was meant as "easier to reproduce locally" - in CI I can easily make those commands more complex.

Let me try to remove it and see how it will be then, that's not a big change and will decrease amount of code.

[potiuk]

Of particular note, I imagine that soon (as part of TaskSDK work) all the db unit tests for the providers will go away/become non-db tests. I don't know if that changes your thinking at all

That's planned in stage 3 in #42632 (comment) - but it's pretty independent - there we would simply remove the db method and rename it, but yeah, I will try to remove db/non-db commands now.

[potiuk]

Just so I understand the plan, this splits out the running of tests, but doesn't change to code location which will be a future PR?

Absolutely. One thing at a time. We want to merge it, let it run for 2-3 weeks and fix all teething problems before we go to next stages. Same as with the past "providers" move. We will find things a week or two from now after we merge this one, I am quite sure of that. And we have to do it carefully while not breaking too much the workflows of people who work noow with a speed of litght - in parallel.

[jscheffl]

Two comments, otherwise looks good to me. Would prefer another approval besides the reviews already received.

[eladkal]

Very cool!

[gopidesupavan]

LGTM +1

This is a fantastic simplification and a significant step forward for the upcoming planned stages. Thanks, Jarek, for the great optimization!

Now that each test variation has its own groups, I believe it will be much easier to make changes or extend functionality for any specific variation as needed in the future.

[potiuk]

The whole set of test commands is simplified now:

I also updated docs and examples and contributing docs.

I also reviewed and updated docs and fixed and simplified how sytem tests are run.

There is no more --system SYSTEM or pytest.mark.system("SYSTEM") but simply --system and pytest.mark.system. Also the example dags in "tests/system" and "providers/tests/system" are automatically marked with the "system" marker.

So tests shoudl be run:

In venv/inside breeze:

pytest --system providers/tests/system/google/cloud/bigquery/example_bigquery_queries.py

via Breeze - there is one command only:

breeze testing system-tests  providers/tests/system/google/cloud/bigquery/example_bigquery_queries.py

cc: @ahidalgob @kosteev @pankajkoti @fdemiane @sc250072

nce we merge it you will have to update the dashboards

[potiuk]

And merged!