add conda-lock support to conda easyblock #3397

JinyuHan99 · 2024-07-28T21:43:52Z

(created using eb --new-pr)

ocaisa · 2024-07-31T12:26:39Z

I did a diff of this this with the existing conda easyblock and the changes are quite small, it would be better to add condalock support to the existing easyblock.

ocaisa · 2024-07-31T12:34:11Z

easybuild/easyblocks/generic/condalock.py

+                get_software_root('anaconda3') or get_software_root('miniconda3')):
+            conda_cmd = 'conda'
+        elif get_software_root('miniforge3'):
+            conda_cmd = 'conda-lock'


This is not really correct, miniforge3 provides conda/mamba (from what I understand) but conda-lock is a separate Python package.

ocaisa · 2024-07-31T12:47:22Z

conda-lock looks interesting for reproducibility, but more is needed to fully support it in the existing conda easyblock:

Add an option to trigger it's use
Install conda-lock using the conda_cmd the user asked for
Use conda-lock to generate the lock file (may require platform specific support)
Use the conda_cmd the user asked for to install the package with the conda-lock created lock file
Make sure the lock file is shipped in the installation

JinyuHan99 · 2024-08-04T23:38:09Z

Hi ocaisa,

Thank you for your review! I'm trying to modify this easyblock according to your wonderful suggestions.

Regarding "Use conda-lock to generate the lock file (may require platform specific support)", I couldn't find the part where EasyBuild could automatically recognise the system architechture. Could you please indicate where I can find this information or if I need to write my own logic to achieve that?

If I'm going to do it myself, I suppose I need to transform them into the way conda-lock could recognize, right? (Or the laziest way is to let users specify the platform)

FYI, the conda-lock supported platforms look like this:

platforms:
  - linux-64
  - osx-64
  - win-64
  - osx-arm64  # For Apple Silicon, e.g. M1/M2
  - linux-aarch64  # aka arm64, use for Docker on Apple Silicon
  - linux-ppc64le

Regarding "Use the conda_cmd the user asked for to install the package with the conda-lock created lock file", if I understand correctly, I need to use conda-lock install to create the environment through a lock file, rather than using the conda_cmd.

Thank you for your time and help!

All the best,
Leah

easybuild/easyblocks/generic/conda.py

…in conda.py

ocaisa · 2024-08-05T07:48:27Z

Regarding "Use conda-lock to generate the lock file (may require platform specific support)", I couldn't find the part where EasyBuild could automatically recognise the system architechture. Could you please indicate where I can find this information or if I need to write my own logic to achieve that?

get_system_info() will give you more than you need, see https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/tools/systemtools.py#L1128

Regarding "Use the conda_cmd the user asked for to install the package with the conda-lock created lock file", if I understand correctly, I need to use conda-lock install to create the environment through a lock file, rather than using the conda_cmd.

I'm no expert here, but reading the documentation for conda-lock:

# Install conda-lock (should use conda/mamba as requested by the easyconfig, make sure version >= 1.0)
<conda/mamba> install --channel=conda-forge --name=<expected name> conda-lock
# identify the platform using `get_system_info` and create the necessary mapping (there are some useful options here)
conda-lock render -p <platform> (-c <channel> --no-dev-dependencies)
# NOTE: There is a chance that the above command will fail, so need to verify that it worked
# Now that we have a lock file we can use the requested tool for the installation
<conda/mamba> create -n <expected name> --file conda-<platform>.lock

ocaisa · 2024-08-05T07:56:13Z

easybuild/easyblocks/generic/conda.py

@@ -50,6 +51,9 @@ def extra_options(extra_vars=None):
            'environment_file': [None, "Conda environment.yml file to use with 'conda env create'", CUSTOM],
            'remote_environment': [None, "Remote conda environment to use with 'conda env create'", CUSTOM],
            'requirements': [None, "Requirements specification to pass to 'conda install'", CUSTOM],
+            'use_conda_lock': [False, "Whether to use conda-lock for reproducibility", CUSTOM],
+            'lock_file': [None, "Path to a pre-generated conda-lock file", CUSTOM],


I'm not sure how this could work as you would need different lock files for different architectures (of course conda-lock can easily create these, so I guess they could be added as source files).

I agree it could be useful, and if they existed you wouldn't even need to install conda-lock, you could just use the lock files directly.

I think it would be better to make this a Boolean named lock_file_provided: given use_conda_lock is set to true, True implies that a platform lock file is provided via a patch (since we can ship patches in our easyconfig repository). That way people can go through some manual effort to get the "right" lock files. A False value means the lock file needs to be generated. You would need to verify that a lock file for the current platform has been provided.

ocaisa · 2024-08-05T07:57:00Z

easybuild/easyblocks/generic/conda.py

@@ -50,6 +51,9 @@ def extra_options(extra_vars=None):
            'environment_file': [None, "Conda environment.yml file to use with 'conda env create'", CUSTOM],
            'remote_environment': [None, "Remote conda environment to use with 'conda env create'", CUSTOM],
            'requirements': [None, "Requirements specification to pass to 'conda install'", CUSTOM],
+            'use_conda_lock': [False, "Whether to use conda-lock for reproducibility", CUSTOM],
+            'lock_file': [None, "Path to a pre-generated conda-lock file", CUSTOM],
+            'platform': [None, "Platform specified for conda-lock", CUSTOM],


Drop this, you need to figure out what platform you are actually on, defining it is the opposite of helpful if it doesn't actually match.

hound · 2024-08-13T03:38:24Z

easybuild/easyblocks/generic/conda.py

+
+
+            # use lock_file to create environment
+            cmd = "%s %s create --file %s -p %s -y" % (self.cfg['preinstallopts'],conda_cmd, platform_rendered_lock_file, self.installdir)


line too long (138 > 120 characters)
missing whitespace after ','

hound · 2024-08-13T03:38:24Z

easybuild/easyblocks/generic/conda.py

+                    platform_rendered_lock_file=lock_file                
+
+
+            # use lock_file to create environment


too many blank lines (2)

hound · 2024-08-13T03:38:24Z

easybuild/easyblocks/generic/conda.py

+                    run_cmd(cmd, log_all=True, simple=True)
+                else:
+
+                    platform_rendered_lock_file=lock_file                


missing whitespace around operator
trailing whitespace

hound · 2024-08-13T03:38:24Z

easybuild/easyblocks/generic/conda.py

+                    # install conda-lock
+                    cmd = "%s install conda-lock" % conda_cmd
+                    run_cmd(cmd, log_all=True, simple=True)
+


blank line contains whitespace

hound · 2024-08-13T03:38:24Z

easybuild/easyblocks/generic/conda.py

+                    raise EasyBuildError("The provided lock file does not match this platform")
+
+
+                if lock_file_type == "multi":


too many blank lines (2)

hound · 2024-08-13T03:38:24Z

easybuild/easyblocks/generic/conda.py

+                            return line.split(":", 1)[1].strip() == platform_name
+
+
+        if self.cfg['use_conda_lock']:


too many blank lines (2)

hound · 2024-08-13T03:38:24Z

easybuild/easyblocks/generic/conda.py

+                    lock_data = yaml.safe_load(f)
+                    return platform_name in lock_data.get('metadata', []).get('platforms',[])
+            else:
+            # for a single-platform rendered lock file like conda-linux64.lock


expected an indented block (comment)

hound · 2024-08-13T03:38:24Z

easybuild/easyblocks/generic/conda.py

+            # for a multi-platform lock file like conda-lock.yml
+                with open(lock_file, 'r') as f:
+                    lock_data = yaml.safe_load(f)
+                    return platform_name in lock_data.get('metadata', []).get('platforms',[])


missing whitespace after ','

hound · 2024-08-13T03:38:24Z

easybuild/easyblocks/generic/conda.py

+            file_type = check_lock_file_type(lock_file)
+
+            if file_type == "multi":
+            # for a multi-platform lock file like conda-lock.yml


expected an indented block (comment)

hound · 2024-08-13T03:38:25Z

easybuild/easyblocks/generic/conda.py

+            }
+
+            return platforms.get(system, {}).get(machine)
+


blank line contains whitespace

JinyuHan99 · 2024-08-13T03:57:42Z

Hi ocaisa,

Thanks for your review! This time I added a few things:

I realized there are two kinds of lock files: multi-platform and single-platform files. They need to be handled differently.
Used easybuild.tools.systemtools.get_system_info() to get platform info.
Added checking if the platform appeared in lock file matches with the actual platform
Shipped the related files in installation.
Used conda cmd to create the environment at the end.

Please let me know if there's anything wrong or need to be improved. Really appreciate it.

All the best,
Leah

JinyuHan99 · 2024-08-15T02:56:47Z

Hello,

The tests failed due to module yaml not found. Could you please indicate the right way to use yaml module in the test environment? Thanks!

All the best,
Leah

ocaisa · 2024-08-15T21:07:38Z

@JinyuHan99 I'm on leave until September, won't be able to get back to this until then

adding easyblocks: condalock.py

25bcd57

JinyuHan99 mentioned this pull request Jul 28, 2024

{geo}[SYSTEM] geoapps v0.11.1 easybuilders/easybuild-easyconfigs#21079

Draft

ocaisa reviewed Jul 31, 2024

View reviewed changes

boegel added the new label Jul 31, 2024

boegel added this to the 4.x milestone Jul 31, 2024

JinyuHan99 changed the title ~~new easyblock for condalock~~ add conda-lock support to conda easyblock Aug 5, 2024

LeahHan and others added 3 commits August 5, 2024 18:46

add conda-lock support to existing conda easyblock

2685185

Merge branch 'easybuilders:develop' into 20240728234349_new_pr_condalock

9682388

rename condaenhance.py to conda.py

3c38cd8

ocaisa reviewed Aug 5, 2024

View reviewed changes

easybuild/easyblocks/generic/conda.py Outdated Show resolved Hide resolved

remove condalock.py, deleted unnecessary comments and fixed mistakes …

b363c0b

…in conda.py

ocaisa requested changes Aug 5, 2024

View reviewed changes

add function to handle multi-platform and single-platform lock files

68eb020

hound bot reviewed Aug 13, 2024

View reviewed changes

boegel added enhancement and removed new labels Aug 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add conda-lock support to conda easyblock #3397

add conda-lock support to conda easyblock #3397

JinyuHan99 commented Jul 28, 2024

ocaisa commented Jul 31, 2024

ocaisa Jul 31, 2024

ocaisa commented Jul 31, 2024 •

edited

Loading

JinyuHan99 commented Aug 4, 2024 •

edited

Loading

ocaisa commented Aug 5, 2024

ocaisa Aug 5, 2024

ocaisa Aug 5, 2024

hound bot Aug 13, 2024

hound bot Aug 13, 2024

hound bot Aug 13, 2024

hound bot Aug 13, 2024

hound bot Aug 13, 2024

hound bot Aug 13, 2024

hound bot Aug 13, 2024

hound bot Aug 13, 2024

hound bot Aug 13, 2024

hound bot Aug 13, 2024

JinyuHan99 commented Aug 13, 2024

JinyuHan99 commented Aug 15, 2024

ocaisa commented Aug 15, 2024



		# use lock_file to create environment
		cmd = "%s %s create --file %s -p %s -y" % (self.cfg['preinstallopts'],conda_cmd, platform_rendered_lock_file, self.installdir)

		platform_rendered_lock_file=lock_file


		# use lock_file to create environment

		raise EasyBuildError("The provided lock file does not match this platform")


		if lock_file_type == "multi":

		return line.split(":", 1)[1].strip() == platform_name


		if self.cfg['use_conda_lock']:

add conda-lock support to conda easyblock #3397

Are you sure you want to change the base?

add conda-lock support to conda easyblock #3397

Conversation

JinyuHan99 commented Jul 28, 2024

ocaisa commented Jul 31, 2024

ocaisa Jul 31, 2024

Choose a reason for hiding this comment

ocaisa commented Jul 31, 2024 • edited Loading

JinyuHan99 commented Aug 4, 2024 • edited Loading

ocaisa commented Aug 5, 2024

ocaisa Aug 5, 2024

Choose a reason for hiding this comment

ocaisa Aug 5, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

hound bot Aug 13, 2024

Choose a reason for hiding this comment

JinyuHan99 commented Aug 13, 2024

JinyuHan99 commented Aug 15, 2024

ocaisa commented Aug 15, 2024

ocaisa commented Jul 31, 2024 •

edited

Loading

JinyuHan99 commented Aug 4, 2024 •

edited

Loading