Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAS-2180: Remove Conda dependency on docker images #20

Merged
merged 5 commits into from
Jun 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ HyBIG follows semantic versioning. All notable changes to this project will be
documented in this file. The format is based on [Keep a
Changelog](http://keepachangelog.com/en/1.0.0/).

## [v1.2.2] - 2024-06-18

### Changed
Removes internal dependency on conda.

## [v1.2.1] - 2024-06-10

### Changed
Expand Down Expand Up @@ -47,7 +52,8 @@ outlined by the NASA open-source guidelines.
For more information on internal releases prior to NASA open-source approval,
see legacy-CHANGELOG.md.

[unreleased]:https://github.com/nasa/harmony-browse-image-generator/compare/1.2.1..HEAD
[unreleased]:https://github.com/nasa/harmony-browse-image-generator/compare/1.2.2..HEAD
[v1.2.2]: https://github.com/nasa/harmony-browse-image-generator/compare/1.2.1..1.2.2
[v1.2.1]: https://github.com/nasa/harmony-browse-image-generator/compare/1.2.0..1.2.1
[v1.2.0]: https://github.com/nasa/harmony-browse-image-generator/compare/1.1.0..1.2.0
[v1.1.0]: https://github.com/nasa/harmony-browse-image-generator/compare/1.0.2..1.1.0
Expand Down
16 changes: 9 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,10 +138,10 @@ also with units of degrees.
|- CONTRIBUTING.md
|- LICENSE
|- README.md
|- conda_requirements.txt
|- dev-requirements.txt
|- legacy-CHANGELOG.md
|- pip_requirements.txt
|- pip_requirements_skip_snyk.txt
```

* `bin` - A directory containing utility scripts to build the service and test
Expand Down Expand Up @@ -173,9 +173,6 @@ also with units of degrees.

* `README.md` - This file, containing guidance on developing the service.

* `conda_requirements.txt` - A list of service dependencies, such as GDAL, that
cannot be installed via Pip.

* `dev-requirements.txt` - list of packages required for service development.

* `legacy-CHANGELOG.md` - Notes for each version that was previously released
Expand All @@ -184,6 +181,12 @@ also with units of degrees.

* `pip_requirements.txt` - A list of service Python package dependencies.

* `pip_requirements_skip_snyk.txt` - A list of service Python package
dependencies that are not scanned by snyk for vulnerabilities. This file
contains only the `GDAL` package. It is separated because snyk's scanning is
naive and cannot pre-install required libraries so that `pip install GDAL`
fails and we have no work around.


## Local development:

Expand All @@ -193,13 +196,12 @@ regarding creation of a local Harmony instance.

If testing small functions locally that do not require inputs from the main
Harmony application, it is recommended that you create a Python virtual
environment via conda, and then install the necessary dependencies for the
environment, and then install the necessary dependencies for the
service within that environment via conda and pip then install the pre-commit hooks.

```
> conda create --name hybig-env python==3.11
> conda install --file conda_requirements.txt
> pip install -r pip_requirements.txt
> pip install -r pip_requirements.txt -r pip_requirements_skip_snyk.txt
> pip install -r dev-requirements.txt

> pre-commit install
Expand Down
1 change: 0 additions & 1 deletion conda_requirements.txt

This file was deleted.

41 changes: 11 additions & 30 deletions docker/service.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,47 +10,28 @@
# 2023-01-26: Updated for the Harmony Regridding Service.
# 2023-04-04: Updated for HyBIG.
# 2023-04-23: Updated conda clean and pip install to keep Docker image slim.
# 2024-06-18: Updates to remove conda dependency.
#
###############################################################################
FROM continuumio/miniconda3
FROM python:3.11

WORKDIR "/home"

# Create Conda environment:
COPY conda_requirements.txt conda_requirements.txt
RUN conda create --yes --name hybig --file conda_requirements.txt \
python=3.11 --channel conda-forge --channel defaults -q && \
conda clean --all --force-pkgs-dirs --yes
RUN apt-get update
RUN apt-get install -y libgdal-dev

# Install additional Pip dependencies
COPY pip_requirements.txt pip_requirements.txt
RUN conda run --name hybig pip install --no-input --no-cache-dir \
-r pip_requirements.txt
# Install Pip dependencies
COPY pip_requirements*.txt /home/

RUN pip install --no-input --no-cache-dir \
-r pip_requirements.txt \
-r pip_requirements_skip_snyk.txt

# Copy service code.
COPY ./harmony_browse_image_generator harmony_browse_image_generator

# Set conda environment for HyBIG, as `conda run` will not stream logging.
# Setting these environment variables is the equivalent of `conda activate`.
ENV _CE_CONDA='' \
_CE_M='' \
CONDA_DEFAULT_ENV=hybig \
CONDA_EXE=/opt/conda/bin/conda \
CONDA_PREFIX=/opt/conda/envs/hybig \
CONDA_PREFIX_1=/opt/conda \
CONDA_PROMPT_MODIFIER=(hybig) \
CONDA_PYTHON_EXE=/opt/conda/hybig/python \
CONDA_ROOT=/opt/conda \
CONDA_SHLVL=2 \
PATH="/opt/conda/envs/hybig/bin:${PATH}" \
SHLVL=1

# Set GDAL related environment variables.
ENV CPL_ZIP_ENCODING=UTF-8 \
GDAL_DATA=/opt/conda/envs/hybig/share/gdal \
GSETTINGS_SCHEMA_DIR=/opt/conda/envs/hybig/share/glib-2.0/schemas \
GSETTINGS_SCHEMA_DIR_CONDA_BACKUP='' \
PROJ_LIB=/opt/conda/envs/hybig/share/proj
ENV CPL_ZIP_ENCODING=UTF-8

# Configure a container to be executable via the `docker run` command.
ENTRYPOINT ["python", "-m", "harmony_browse_image_generator"]
2 changes: 1 addition & 1 deletion docker/service_version.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.2.1
1.2.2
4 changes: 3 additions & 1 deletion docker/tests.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,15 @@
# 2023-04-04: Updated for HyBIG.
# 2023-04-17: Added --no-cache-dir to keep Docker images slim.
# 2024-01-22: Updated to use new open-source service image name.
# 2024-06-18: Removes Conda.
#
###############################################################################
FROM ghcr.io/nasa/harmony-browse-image-generator

# Install additional Pip requirements (for testing)
COPY tests/pip_test_requirements.txt .
RUN conda run --name hybig pip install --no-input --no-cache-dir \

RUN pip install --no-input --no-cache-dir \
-r pip_test_requirements.txt

# Copy test directory containing Python unittest suite, test data and utilities
Expand Down
3 changes: 3 additions & 0 deletions pip_requirements_skip_snyk.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Because snyk can't install gdal properly during automated scans, we have this
# file that includes the GDAL package alone which is not scanned by snyk
GDAL==3.6.2
Loading