Merge pull request #397 from singularityhub/updates/notation

Updates to several container and module bases

.github/workflows/main.yml

@@ -38,4 +38,5 @@ jobs:
           pyflakes shpc/main/modules
           pyflakes shpc/main/container/base.py
           pyflakes shpc/main/container/podman.py
+          pyflakes shpc/main/container/docker.py
           pyflakes shpc/main/container/singularity.py

.github/workflows/test.yml

@@ -73,6 +73,10 @@ jobs:
           shpc config set container_tech:${{ matrix.container_tech }}
           shpc config set module_sys:${{ matrix.module }}
           shpc config set enable_tty:false
+
+          printf "\n\shpc test ============================================\n"
+          shpc test python:3.9.5-alpine          
+
           shpc install python:3.9.5-alpine
 
           module use ./modules
@@ -96,7 +100,8 @@ jobs:
           cat test_output
           grep --quiet 'Python 3.9.5' test_output
           rm test_output
-
+          shpc uninstall --force python:3.9.5-alpine 
+        
       - name: Run python module tests (tcsh)
         shell: tcsh -e {0}
         run: |
@@ -114,6 +119,7 @@ jobs:
           shpc config set container_tech:${{ matrix.container_tech }}
           shpc config set module_sys:${{ matrix.module }}
           shpc config set enable_tty:false
+
           shpc install python:3.9.5-alpine
 
           module use ./modules
@@ -137,3 +143,4 @@ jobs:
           cat test_output
           grep --quiet 'Python 3.9.5' test_output
           rm test_output
+          shpc uninstall --force python:3.9.5-alpine

CHANGELOG.md

@@ -14,6 +14,13 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.scom/singularityhub/singularity-hpc/tree/master) (0.0.x)
+ - Add test for docker and podman (0.0.28)
+   - namespace as format string for command named renamed to repository
+   - shpc test/uninstall should be run for all tests
+   - bug with uninstall
+   - adding user and group ids to docker and podman commands
+   - remove conflict with entire module name, should only be for aliases
+   - fix documentation of commands in Lua module template
  - Cleanup of module files and docker requirement bugfix (0.0.27)
  - Docker support (0.0.26)
    - added an enable_tty setting to allow disabling add -t in recipes

docs/getting_started/user-guide.rst

@@ -139,7 +139,7 @@ A summary table of variables is included below, and then further discussed in de
      - if defined, add to module script to load this Singularity module first
      - null
    * - module_name
-     - Format string for module commands exec,shell,run (not aliases) can include ``{{ registry }}``, ``{{ namespace }}``, ``{{ tool }}`` and ``{{ version }}``
+     - Format string for module commands exec,shell,run (not aliases) can include ``{{ registry }}``, ``{{ repository }}``, ``{{ tool }}`` and ``{{ version }}``
      - ``{{ tool }}``
    * - bindpaths
      - string with comma separated list of paths to binds. If set, expored to SINGULARITY_BINDPATH
@@ -305,12 +305,12 @@ A container identifier is parsed as follows:
 .. code-block:: console
 
     # quay.io   /biocontainers/samtools:latest
-    # <registry>/  <namespace>/  <tool>/<version>
+    # <registry>/  <repository>/  <tool>/<version>
 
 
 So by default, we use tool because it's likely closest to the command that is wanted.
 But let's say you had two versions of samtools - the namespaces would conflict! You
-would want to change your format string to ``{{ namespace }}-{{ tool }}`` to be
+would want to change your format string to ``{{ repository }}-{{ tool }}`` to be
 perhaps "biocontainers-samtools-exec" and "another-samtools-exec." 
 If you change the format string to ``{{ tool }}-{{ version }}`` you would see:
 

paper/paper.md

@@ -25,14 +25,14 @@ bibliography: paper.bib
 
 Portability and reproducibility of complex software stacks is essential for researchers to perform their work. High Performance Computing (HPC) environments add another level of complexity, where possibly conflicting dependencies must co-exist. Although container technologies like Singularity [@Kurtzer2017-xj] make it possible to "bring your own environment," without any form of central strategy to manage containers, researchers that seek reproducibility via using containers are tasked with managing their own container collection, often not taking care to ensure that a particular digest or version is used. The reproducibility of the work is at risk, as they cannot easily install and use containers, nor can they share their software with others.
 
-Singularity Registry HPC (shpc) is the first of its kind to provide an easy means for a researcher to add their research software for sharing and collaboration with other researchers to an existing collection of over 200 popular scientific libraries [@da2017biocontainers; @noauthor_undated-kp, @gorgolewski2017bids; @gamblin2015spack; @autamus]. The software installs these containers as environment modules [@McLay2011-wu] that are easy to use and read documentation for, and exposes aliases for commands in the container that the researcher can add to his or her pipeline without thinking about complex interactions with a container. The simple addition of an entry to the registry maintained by shpc comes down to adding a yaml file, and after doing this, another researcher can easily install the same software, down to the digest, to reproduce the original work.
+Singularity Registry HPC (shpc) is the first of its kind to provide an easy means for a researcher to add their research software for sharing and collaboration with other researchers to an existing collection of over 200 popular scientific libraries [@da2017biocontainers; @noauthor_undated-kp, @gorgolewski2017bids; @gamblin2015spack; @autamus]. The software installs containers as environment modules [@McLay2011-wu] that are easy to use and read documentation for, and exposes aliases for commands in the container that the researcher can add to his or her pipeline without thinking about complex interactions with a container. The simple addition of an entry to the registry maintained by shpc comes down to adding a yaml file, and after doing this, another researcher can easily install the same software, down to the digest, to reproduce the original work.
 
 
 ## Statement of Need
 
 Using environment modules [@McLay2011-wu] on HPC clusters is a common
 trend. Although writing the recipes can be complex, it's a fairly common practice for cluster administrators to provide
-a set of natively installed recipes for their users [@noauthor_undated-bt], or for researchers to develop and deploy their own software via containers. Even well-known package managers like Spack [@noauthor_undated-ae] and EasyBuild [@noauthor_undated-dj] expose software as modules. However, these package manager approaches don't always ensure reproducibility, or ease of development for the researcher. They typically require relying on some subset of system software, the underlying operating system, or even making changes to the system, which is not under the researcher's control. Although using containers in this context has been discussed previously [@noauthor_undated-rj; @noauthor_undated-rc], the majority of these approaches and tools do not make the process of developing and installing container modules easy. The single researcher must either convince a cluster administrator to install dependencies needed for their software, or build a container and manually move and interact with it on the cluster. All of these small challenges come together to make it harder for a researcher to develop and manage their own software, and subsequently to share their approach to reproduce the work. Using Singularity containers installed via Singularity Registry HPC offers a solution to this challenge. The only requirement is the Singularity software, and writing a simple configuration file for the registry. By clearly defining commands, and pinning exact versions of scientific software, researchers on high performance computing
+a set of natively installed recipes for their users [@noauthor_undated-bt], or for researchers to develop and deploy their own software via containers. Even well-known package managers like Spack [@noauthor_undated-ae] and EasyBuild [@noauthor_undated-dj] expose software as modules. However, these package manager approaches don't always ensure reproducibility, or ease of development for the researcher. They typically require relying on some subset of system software, the underlying operating system, or even making changes to the system, which is not under the researcher's control. Although using containers in this context has been discussed previously [@noauthor_undated-rj; @noauthor_undated-rc], the majority of these approaches and tools do not make the process of developing and installing container modules easy. The single researcher must either convince a cluster administrator to install dependencies needed for their software, or build a container and manually move and interact with it on the cluster. All of these small challenges come together to make it harder for a researcher to develop and manage their own software, and subsequently to share their approach to reproduce the work. Using Singularity, Podman, or other container technologies installed via Singularity Registry HPC offers a solution to this challenge. The only requirement is the container technology software, and writing a simple configuration file for the registry. By clearly defining commands, and pinning exact versions of scientific software, researchers on high performance computing
 clusters can have more confidence in the reproducibility of their work [@Santana-Perez2015-wo; @Boettiger2014-cz; @Wandell2015-yt].
 
 ## Usage

shpc/client/config.py

@@ -10,8 +10,6 @@ def main(args, parser, extra, subparser):
 
     from shpc.main import get_client
 
-    cli = get_client(quiet=args.quiet, settings_file=args.settings_file)
-
     # If nothing provided, show help
     if not args.params:
         print(subparser.format_help())
@@ -20,6 +18,11 @@ def main(args, parser, extra, subparser):
     # The first "param" is either set of get
     command = args.params.pop(0)
 
+    validate = True if not command == "edit" else False
+    cli = get_client(
+        quiet=args.quiet, settings_file=args.settings_file, validate=validate
+    )
+
     # For each new setting, update and save!
     if command == "edit":
         return cli.settings.edit()

shpc/main/__init__.py

@@ -21,9 +21,10 @@ def get_client(quiet=False, **kwargs):
     """
     # The name of the module
     module = kwargs.get("module")
+    validate = kwargs.get("validate", True)
 
     # Load user settings to add to client, and container technology
-    settings = Settings(kwargs.get("settings_file"))
+    settings = Settings(kwargs.get("settings_file"), validate)
     container = kwargs.get("container_tech") or settings.container_tech
 
     # Use the user provided module OR the default

shpc/main/client.py

@@ -141,11 +141,7 @@ def cleanup(tmpdir):
         # Test all tags (this could be subsetted)
         for tag in tags:
 
-            # Install the recipe
-            sif = self.install(module_name, tag)
-
-            # Assert that the container exists
-            assert os.path.exists(sif)
+            image = self.install(module_name, tag)
 
             # Do we want to test loading?
             if not skip_module and hasattr(self, "_test"):
@@ -157,27 +153,17 @@ def cleanup(tmpdir):
             # Do we want to test the test commands?
             if test_commands and config.test:
                 utils.write_file(test_file, config.test)
-                command = ["singularity", "exec", sif, "/bin/bash", test_file]
-                result = utils.run_command(command)
-
-                # We can't run on incompatible hosts
-                if (
-                    "the image's architecture" in result["message"]
-                    and result["return_code"] != 0
-                ):
-                    logger.warning(
-                        "Cannot run test for container with incompatible architecture: %s"
-                        % result["message"]
-                    )
-                elif result["return_code"] != 0:
+                return_code = self.container.test_script(image, test_file)
+                if return_code != 0:
                     cleanup(tmpdir)
                     logger.exit("Test of %s was not successful." % module_name)
 
             # Test the commands
             if not test_exec:
                 continue
+
             for alias in config.get_aliases():
-                result = self._container.client.execute(sif, alias["command"])
+                result = self.client.execute(image, alias["command"])
 
         # Cleanup the test install
         if stage:

shpc/main/container/base.py

@@ -20,7 +20,7 @@ class ContainerName:
     def __init__(self, raw):
         self.raw = raw
         self.registry = None
-        self.namespace = None
+        self.repository = None
         self.tool = None
         self.version = None
         self.digest = None
@@ -90,13 +90,17 @@ def container_dir(self, name):
             return os.path.join(self.settings.module_base, name)
         return os.path.join(self.settings.container_base, name)
 
-    def guess_tag(self, module_name):
-        """If a user asks for a name without a tag, try to figure it out."""
+    def guess_tag(self, module_name, allow_fail=False):
+        """
+        If a user asks for a name without a tag, try to figure it out.
+        """
         if ":" in module_name:
             return module_name
         tags = os.listdir(os.path.join(self.settings.module_base, module_name))
-        if not tags:
+        if not tags and allow_fail:
             logger.exit("%s does not have any tags installed." % module_name)
+        elif (tags or len(tags) > 1) and allow_fail:
+            return
         elif len(tags) > 1:
             logger.exit(
                 "Multiple tags found for %s: %s." % (module_name, ", ".join(tags))

shpc/main/container/docker.py

@@ -99,9 +99,13 @@ def get(self, module_name):
         Determine if a container uri exists.
         """
         # If no module tag provided, try to deduce from install tree
-        module_name = self.guess_tag(module_name)
+        full_name = self.guess_tag(module_name, allow_fail=True)
 
-        uri = self.add_registry(module_name)
+        # The user already provided a tag
+        if not full_name:
+            full_name = module_name
+
+        uri = self.add_registry(full_name)
         # If there isn't a tag in the name, add it back
         if ":" not in uri:
             uri = ":".join(uri.rsplit("/", 1))
@@ -132,6 +136,25 @@ def check(self, module_name, config):
                     % (module_name, config.latest.name)
                 )
 
+    def test_script(self, image, test_script):
+        """
+        Given a test file, run it and respond accordingly.
+        """
+        command = [
+            "docker",
+            "run",
+            "-i",
+            "--entrypoint",
+            "/bin/bash",
+            "-t",
+            image,
+            test_script,
+        ]
+        result = shpc.utils.run_command(command)
+
+        # Return code
+        return result["return_code"]
+
     def install(
         self,
         module_path,
@@ -190,7 +213,7 @@ def install(
             name=name,
             tool=parsed_name.tool,
             registry=parsed_name.registry,
-            namespace=parsed_name.namespace,
+            repository=parsed_name.repository,
             envfile=self.settings.environment_file,
             command=self.command,
             tty=self.settings.enable_tty,

shpc/main/container/singularity.py

@@ -179,7 +179,7 @@ def install(
             name=name,
             tool=parsed_name.tool,
             registry=parsed_name.registry,
-            namespace=parsed_name.namespace,
+            repository=parsed_name.repository,
             envfile=self.settings.environment_file,
         )
         shpc.utils.write_file(module_path, out)
@@ -325,3 +325,23 @@ def _pull_github(self, uri, dest=None):
             dest = os.path.basename(url)
         name = os.path.basename(dest)
         return self.client.pull(url, name=name, pull_folder=os.path.dirname(dest))
+
+    def test_script(self, image, test_script):
+        """
+        Given a test file, run it and respond accordingly.
+        """
+        command = ["singularity", "exec", image, "/bin/bash", test_script]
+        result = shpc.utils.run_command(command)
+
+        # We can't run on incompatible hosts
+        if (
+            "the image's architecture" in result["message"]
+            and result["return_code"] != 0
+        ):
+            logger.warning(
+                "Cannot run test for incompatible architecture: %s" % result["message"]
+            )
+            return 0
+
+        # Return code
+        return result["return_code"]

shpc/main/modules/__init__.py

@@ -84,6 +84,7 @@ def uninstall(self, name, force=False):
         """
         # The name can either be a folder or an install directory
         name = self.add_namespace(name)
+
         # folder paths do not have :
         name = name.replace(":", os.sep)
         module_dir = os.path.join(self.settings.module_base, name)