Feature/loosen pyproject deps

[saeliddp]

What does this PR do?

As part of the effort to expand the Cyto-DL API, this PR relaxes the requirements in the pyproject.toml to make installing from PyPI a bit easier for clients. Short justifications of the changes are below. The final test I ran was pytest -k segmentation from a fresh Python 3.10 venv where I installed the requirements as they are in pyproject.toml (plus pytest) using pip install -r. All of these tests passed in 15 minutes.

• hydra-core~=1.3.0 : homepage states 1.3 is the stable version
• hydra-colorlog>=1.2 : github homepage states 1.2 adds support for python 3.10, which we need
• hydra-optuna-sweeper>=1.2. : github homepage states 1.2 adds support for python 3.10, which we need
• torch~=2.0.0 : pinned because this is the only version I could test on the A100s (due to nvidia driver compatibility)
• numpy>=1.23 : see 1 below
• matplotlib>=3.7 : see 1 below
• pandas>=1.5 : see 1 below
• fire>=0.5 : see 1 below
• joblib (REMOVED) : this is a dependency of scikit-learn, we do not import it directly in Cyto-DL
• mlflow>=2.1 : see 1 below
• omegaconf>=2.3 : see 1 below
• pyarrow>=10.0 : see 1 below
• pyrootutils>=1.0 : see 1 below
• PyYAML>=6.0 : see 1 below
• scikit-learn>=1.2 : see 1 below
• aicsimageio>=4.11.0 : see 1 below
• universal-pathlib>=0.0 : see 1 below
• ome-zarr>=0.6 : see 1 below
• anndata>=0.8 : see 1 below
• bfio>=2.3.2 : I found that versions above 2.3.2 significantly sped up the time it took to run tests.
• monai-weekly>=1.2.dev2308 : see 2 below
• tifffile (REMOVED) : this is a dependency aicsimageio, we do not import it directly in Cyto-DL
• timm>=0.9.7 : see 2 below
• tqdm>=4.64 : see 1 below
• protobuf (REMOVED) : this is a dependency of mlflow, we do not import it directly in Cyto-DL
• lightning>=2.0 : see 1 below
• ostat (REMOVED) : this was not being used anywhere that I could find
• einops>=0.6.1 : see 2 below
• edt>=2.3.1 : see 2 below
• astropy>=5.2 : see 1 below

1: tested latest version by installing the old requirements, then pip install pkg==new_version, then running gpu-segmentation tests
2: kept as is since already lower-bounded

Notes

• moving to the newer (faster versions of bfio) causes some warnings to appear during testing (no exceptions, though). I think moving forward, we should port to bioio instead of aicsimageio since bioio is the version that will have dev support in the future (created issue here)
• my validation for testing was that all tests passed and the output test segmentation 'looked reasonable' after one epoch of training on one training image (from me and @benjijamorris ). It may be worthwhile to test segmentations on larger training datasets + more epochs of training.
• if we want the speed improvements from the new version of bfio to show up in our tests on github actions, we will need to update the requirements .txt files in the repo (I think these are generated using PDM, which I'm not super familiar with)
• boto3 clashes in a weird way with ome-zarr (actually fsspec[s3] from ome-zarr). The net result of this is that we end up cycling through a bunch of versions of s3fs until we find one that doesn't clash, which is nearly 4 years old at this point

Before submitting

• Did you make sure title is self-explanatory and the description concisely explains the PR?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you list all the breaking changes introduced by this pull request?
• Did you test your PR locally with pytest command?
• Did you run pre-commit hooks with pre-commit run -a command?

Did you have fun?

Make sure you had fun coding 🙃

[benjijamorris]

This looks great! any thoughts on why tests passed locally and failed here?

[saeliddp]

This looks great! any thoughts on why tests passed locally and failed here?

Good question. Unfortunately, I don't have an immediate answer--but from what I can tell, this change should not impact tests on github actions since github actions downloads dependencies from requirements/ not pyproject.toml

Edit: it looks like maybe we are using these new requirements since we pip install . when running tests?

I'm a bit confused why we have all of these installation steps:

grep -E '^numpy==' requirements/$PLATFORM/test-requirements.txt | xargs -n 1 pip install
pip install -r requirements/$PLATFORM/test-requirements.txt
pip install .

Here's a diff between the most recent passing PR's installed dependencies and this PR's installed dependencies: diff. It looks like the difference could be bfio or a few others. Not sure why it would be a problem on a GH runner and not on the machine I was using though...

[saeliddp]

If the test failure is caused by bfio (not sure yet), maybe it's best that we port aicsimageio to bioio before I merge this PR so that we don't have to pin bfio to an older, slower version

[benjijamorris]

bfio seems likely since it failed on image loading...
the complicated install instructions are legacy to install some tricky VAE (not im2im) packages. We've removed those from the test installs so they might no longer be needed.
I think adding bioio here is not a bad idea

[benjijamorris]

@saeliddp could you remove 3.8 from the test matrix as well?

[saeliddp]

@saeliddp could you remove 3.8 from the test matrix as well?

Good call, added in 28b0a3

[benjijamorris]

looks great, don't worry about the failing tests - the ubuntu failure is unrelated to this PR and something I need to fix, the windows tests just timed out (which I think this pr should help with?)