Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

no module named inference, openfold with pytorch2 and cuda12 #259

Open
blakemertz opened this issue Oct 23, 2024 · 4 comments
Open

no module named inference, openfold with pytorch2 and cuda12 #259

blakemertz opened this issue Oct 23, 2024 · 4 comments

Comments

@blakemertz
Copy link

After working on setting up the proper environment for diffdock, I thought I had a system that would allow me to roll with pytorch 2 and cuda 12. I went to run a pdb/ligand pair out of the example folder and got the following error:

(diffdock) [centos@modbind-dy-modbind-cr-0-1 examples]$ python -m inference --config ../default_inference_args.yaml --protein_path 1a46_protein_processed.pdb --ligand 1a46_ligand.sdf --out_dir test
/shared/miniconda3/envs/diffdock/bin/python: No module named inference

I found one other closed issue #175 that had the exact same error message but no expalnation for how it was resolved, so I am opening up a new issue. To take this from the beginning.....

My cloud environment is using cuda 12:

(diffdock) [centos@modbind-dy-modbind-cr-0-1 DiffDock]$ nvidia-smi
Wed Oct 23 20:55:19 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA L4                      On  | 00000000:31:00.0 Off |                    0 |
| N/A   37C    P8              17W /  72W |      4MiB / 23034MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

and this was not downgradeable to cuda11. After looking through the many issues trying to install OpenFold to get DiffDock to work (and failing several times myself to set up a DiffDock environment first and then installing OpenFold), I decided to try to get a working version of OpenFold on my cluster. Following the meticulous work and advice from another contributor on the OpenFold git page, I was able to get a working version of OpenFold based on pytorch2 and cuda12 installed.

I then cloned my OpenFold environment into a DiffDock environment so I could work on getting DiffDock running successfully.

Next step was to pull in the remaining dependencies for DiffDock:

mamba install prody=2.2.0 -c conda-forge
pip install torch_geometric==2.2.0 torch-cluster==1.6.2 torch-scatter==2.1.2 torch-sparse==0.6.18 torch-spline-conv==1.2.2
pip install e3nn==0.5.1
pip install -q git+https://github.com/asarigun/esm.git
conda install conda-forge::python-devtools conda-forge::pybind11 conda-forge::rdkit=2022.03.3 conda-forge::gradio=3.50.*

I specified the versions of ProDy, e3nn, rdkit, and gradio to stay consstent with the environment.yml in the DiffDock source code. I specified the versions for the torch_ dependencies based on my installed version of pytorch (2.1.2). I had to pull a forked version of esm-fold to stay compatible with a pytorch2/cuda12 installation.

The following is my list of conda packages for this environment:

(diffdock) [centos@ip-10-0-0-244 ~]$ conda list
# packages in environment at /shared/miniconda3/envs/diffdock:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
absl-py                   2.1.0              pyhd8ed1ab_0    conda-forge
aiofiles                  22.1.0          py310h06a4308_0  
altair                    5.0.1           py310h06a4308_0  
annotated-types           0.7.0                    pypi_0    pypi
anyio                     4.6.2           py310h06a4308_0  
aom                       3.5.0                h27087fc_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
aria2                     1.36.0               h1e4e653_3    conda-forge
asttokens                 2.0.5              pyhd3eb1b0_0  
attrs                     24.2.0          py310h06a4308_0  
aws-c-auth                0.7.26               hc36b679_2    conda-forge
aws-c-cal                 0.7.4                h2abdd08_0    conda-forge
aws-c-common              0.9.27               h4bc722e_0    conda-forge
aws-c-compression         0.2.19               haa50ccc_0    conda-forge
aws-c-event-stream        0.4.3                h570d160_0    conda-forge
aws-c-http                0.8.8                h9b61739_1    conda-forge
aws-c-io                  0.14.18              h49c7fd3_7    conda-forge
aws-c-mqtt                0.10.4              h5c8269d_18    conda-forge
aws-c-s3                  0.6.4               h77088c0_11    conda-forge
aws-c-sdkutils            0.1.19               h038f3f9_2    conda-forge
aws-checksums             0.1.18              h038f3f9_10    conda-forge
awscli                    2.18.11         py310hff52083_0    conda-forge
awscrt                    0.21.2          py310h95a9d59_15    conda-forge
binutils                  2.43                 h4852527_1    conda-forge
binutils_impl_linux-64    2.43                 h4bf12b8_1    conda-forge
binutils_linux-64         2.43                 h4852527_1    conda-forge
biopython                 1.84            py310hc51659f_0    conda-forge
blas                      2.116                       mkl    conda-forge
blas-devel                3.9.0            16_linux64_mkl    conda-forge
boost                     1.74.0          py310h7c3ba0c_5    conda-forge
boost-cpp                 1.74.0               h75c5d50_8    conda-forge
brotli                    1.1.0                hd590300_1    conda-forge
brotli-bin                1.1.0                hd590300_1    conda-forge
brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.32.3               h4bc722e_0    conda-forge
c-compiler                1.7.0                hd590300_1    conda-forge
ca-certificates           2024.9.24            h06a4308_0  
cairo                     1.16.0            ha61ee94_1012    conda-forge
certifi                   2024.8.30          pyhd8ed1ab_0    conda-forge
cffi                      1.17.0          py310h2fdcea3_0    conda-forge
charset-normalizer        3.4.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contextlib2               21.6.0             pyhd8ed1ab_0    conda-forge
contourpy                 1.3.0           py310h3788b33_2    conda-forge
cryptography              43.0.0          py310h4909e49_0    conda-forge
cuda                      12.1.0                        0    nvidia
cuda-cccl                 12.4.127                      0    nvidia
cuda-command-line-tools   12.1.1                        0    nvidia
cuda-compiler             12.4.1               hbad6d8a_0    conda-forge
cuda-cudart               12.1.105                      0    nvidia
cuda-cudart-dev           12.1.105                      0    nvidia
cuda-cudart-static        12.1.105                      0    nvidia
cuda-cuobjdump            12.4.127                      0    nvidia
cuda-cupti                12.1.105                      0    nvidia
cuda-cupti-static         12.1.105                      0    nvidia
cuda-cuxxfilt             12.4.127                      0    nvidia
cuda-demo-suite           12.4.127                      0    nvidia
cuda-documentation        12.4.127                      0    nvidia
cuda-driver-dev           12.4.127                      0    nvidia
cuda-gdb                  12.4.127                      0    nvidia
cuda-libraries            12.1.0                        0    nvidia
cuda-libraries-dev        12.6.0                        0    nvidia
cuda-libraries-static     12.1.0                        0    nvidia
cuda-nsight               12.4.127                      0    nvidia
cuda-nvcc                 12.4.131                      0    nvidia
cuda-nvdisasm             12.4.127                      0    nvidia
cuda-nvml-dev             12.4.127                      0    nvidia
cuda-nvprof               12.4.127                      0    nvidia
cuda-nvprune              12.4.127                      0    nvidia
cuda-nvrtc                12.1.105                      0    nvidia
cuda-nvrtc-dev            12.1.105                      0    nvidia
cuda-nvrtc-static         12.1.105                      0    nvidia
cuda-nvtx                 12.1.105                      0    nvidia
cuda-nvvp                 12.4.127                      0    nvidia
cuda-opencl               12.4.127                      0    nvidia
cuda-opencl-dev           12.4.127                      0    nvidia
cuda-profiler-api         12.4.127                      0    nvidia
cuda-runtime              12.1.0                        0    nvidia
cuda-sanitizer-api        12.4.127                      0    nvidia
cuda-toolkit              12.1.0                        0    nvidia
cuda-tools                12.1.1                        0    nvidia
cuda-visual-tools         12.6.0                        0    nvidia
cudatoolkit               11.8.0              h4ba93d1_13    conda-forge
cxx-compiler              1.7.0                h00ab1b0_1    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
deepspeed                 0.12.4                   pypi_0    pypi
distro                    1.8.0              pyhd8ed1ab_0    conda-forge
dllogger                  1.0.0                    pypi_0    pypi
dm-tree                   0.1.6                    pypi_0    pypi
docker-pycreds            0.4.0                      py_0    conda-forge
docutils                  0.19            py310hff52083_1    conda-forge
e3nn                      0.5.1                    pypi_0    pypi
einops                    0.8.0                    pypi_0    pypi
exceptiongroup            1.2.0           py310h06a4308_0  
executing                 2.1.0              pyhd8ed1ab_0    conda-forge
expat                     2.6.2                h6a678d5_0  
fair-esm                  2.0.0                    pypi_0    pypi
fastapi                   0.112.2         py310h06a4308_0  
ffmpeg                    5.1.2           gpl_h8dda1f0_106    conda-forge
ffmpy                     0.3.0              pyhb6f538c_0    conda-forge
fftw                      3.3.10          nompi_hf1063bd_110    conda-forge
filelock                  3.16.1             pyhd8ed1ab_0    conda-forge
flash-attn                2.6.3                    pypi_0    pypi
font-ttf-dejavu-sans-mono 2.37                 hd3eb1b0_0  
font-ttf-inconsolata      2.001                hcb22688_0  
font-ttf-source-code-pro  2.030                hd3eb1b0_0  
font-ttf-ubuntu           0.83                 h8b1ccd4_0  
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-anaconda            1                    h8fa9717_0  
fonts-conda-ecosystem     1                    hd3eb1b0_0  
fonttools                 4.54.1          py310h89163eb_1    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
fsspec                    2024.10.0          pyhff2d567_0    conda-forge
gcc                       12.4.0               h236703b_1    conda-forge
gcc_impl_linux-64         12.4.0               hb2e57f8_0    conda-forge
gcc_linux-64              12.4.0               h6b7512a_5    conda-forge
gds-tools                 1.9.1.3                       0    nvidia
git                       2.46.0          pl5321hb5640b7_0    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.43             pyhd8ed1ab_0    conda-forge
glib                      2.82.2               h44428e9_0    conda-forge
glib-tools                2.82.2               h4833e2c_0    conda-forge
gmp                       6.3.0                hac33072_2    conda-forge
gmpy2                     2.1.5           py310hc7909c9_1    conda-forge
gnutls                    3.7.9                hb077bed_0    conda-forge
gradio                    3.50.2             pyhd8ed1ab_0    conda-forge
gradio-client             0.6.1              pyhd8ed1ab_0    conda-forge
greenlet                  3.0.1           py310h6a678d5_0  
gxx                       12.4.0               h236703b_1    conda-forge
gxx_impl_linux-64         12.4.0               h613a52c_0    conda-forge
gxx_linux-64              12.4.0               h8489865_5    conda-forge
h11                       0.14.0          py310h06a4308_0  
hhsuite                   3.3.0           py310pl5321hc31ed2c_12    bioconda
hjson                     3.1.0                    pypi_0    pypi
hmmer                     3.4                  hdbdd923_2    bioconda
httpcore                  1.0.2           py310h06a4308_0  
httpx                     0.27.0          py310h06a4308_0  
huggingface_hub           0.24.6          py310h06a4308_0  
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.10               pyhd8ed1ab_0    conda-forge
ihm                       1.3             py310h5b4e0ec_0    conda-forge
importlib_resources       6.4.0           py310h06a4308_0  
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
joblib                    1.4.2                    pypi_0    pypi
jsonschema                4.23.0          py310h06a4308_0  
jsonschema-specifications 2023.7.1        py310h06a4308_0  
kalign2                   2.04                 h031d066_7    bioconda
kernel-headers_linux-64   3.10.0              he073ed8_17    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.7           py310h3788b33_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
lame                      3.100                h7b6447c_0  
lcms2                     2.15                 haa2dc70_1    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_1    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20240116.2      cxx17_he02047a_1    conda-forge
libblas                   3.9.0            16_linux64_mkl    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcblas                  3.9.0            16_linux64_mkl    conda-forge
libcublas                 12.1.0.26                     0    nvidia
libcublas-dev             12.1.0.26                     0    nvidia
libcublas-static          12.1.0.26                     0    nvidia
libcufft                  11.0.2.4                      0    nvidia
libcufft-dev              11.0.2.4                      0    nvidia
libcufft-static           11.0.2.4                      0    nvidia
libcufile                 1.9.1.3                       0    nvidia
libcufile-dev             1.9.1.3                       0    nvidia
libcufile-static          1.9.1.3                       0    nvidia
libcurand                 10.3.5.147                    0    nvidia
libcurand-dev             10.3.5.147                    0    nvidia
libcurand-static          10.3.5.147                    0    nvidia
libcurl                   8.9.1                hdb1bdb2_0    conda-forge
libcusolver               11.4.4.55                     0    nvidia
libcusolver-dev           11.4.4.55                     0    nvidia
libcusolver-static        11.4.4.55                     0    nvidia
libcusparse               12.0.2.55                     0    nvidia
libcusparse-dev           12.0.2.55                     0    nvidia
libcusparse-static        12.0.2.55                     0    nvidia
libdeflate                1.18                 h0b41bf4_0    conda-forge
libdrm                    2.4.123              hb9d3cd8_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-devel_linux-64     12.4.0             ha4f9413_100    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgfortran-ng            14.1.0               h69a702a_0    conda-forge
libgfortran5              14.1.0               hc5f4f2c_0    conda-forge
libglib                   2.82.2               h2ff4ddf_0    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
libhwloc                  2.9.1                hd6dc26d_0    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libidn2                   2.3.4                h5eee18b_0  
libjpeg-turbo             2.1.5.1              hd590300_1    conda-forge
liblapack                 3.9.0            16_linux64_mkl    conda-forge
liblapacke                3.9.0            16_linux64_mkl    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnpp                    12.0.2.50                     0    nvidia
libnpp-dev                12.0.2.50                     0    nvidia
libnpp-static             12.0.2.50                     0    nvidia
libnsl                    2.0.1                hd590300_0    conda-forge
libnvfatbin               12.4.127                      0    nvidia
libnvfatbin-dev           12.4.127                      0    nvidia
libnvjitlink              12.1.105                      0    nvidia
libnvjitlink-dev          12.1.105                      0    nvidia
libnvjpeg                 12.1.1.14                     0    nvidia
libnvjpeg-dev             12.1.1.14                     0    nvidia
libnvjpeg-static          12.1.1.14                     0    nvidia
libnvvm-samples           12.1.105                      0    nvidia
libopus                   1.3.1                h5eee18b_1  
libpciaccess              0.18                 hd590300_0    conda-forge
libpng                    1.6.44               hadc24fc_0    conda-forge
libprotobuf               4.25.3               h08a7969_0    conda-forge
libsanitizer              12.4.0               h46f95d5_0    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-devel_linux-64  12.4.0             ha4f9413_100    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libtasn1                  4.19.0               h5eee18b_0  
libtiff                   4.5.1                h8b53f26_1    conda-forge
libunistring              0.9.10               h27cfd23_0  
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libva                     2.18.0               h0b41bf4_0    conda-forge
libvpx                    1.11.0               h295c915_0  
libwebp-base              1.4.0                hd590300_0    conda-forge
libxcb                    1.13                 h1bed415_1  
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.10.3               hca2bb57_4    conda-forge
libzlib                   1.3.1                h4ab18f5_1    conda-forge
lightning-utilities       0.11.8             pyhd8ed1ab_0    conda-forge
llvm-openmp               15.0.7               h0cdce71_0    conda-forge
markupsafe                2.1.5           py310h2372a71_0    conda-forge
matplotlib-base           3.9.2           py310h68603db_1    conda-forge
mkl                       2022.1.0           h84fe81f_915    conda-forge
mkl-devel                 2022.1.0           ha770c72_916    conda-forge
mkl-include               2022.1.0           h84fe81f_915    conda-forge
ml-collections            0.1.1              pyhd8ed1ab_0    conda-forge
modelcif                  0.7                pyhd8ed1ab_0    conda-forge
mpc                       1.3.1                h24ddda3_0    conda-forge
mpfr                      4.2.1                h38ae2d0_2    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.8           py310h25c7140_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.5                  he02047a_1    conda-forge
nettle                    3.9.1                h7ab15ed_0    conda-forge
networkx                  3.4.2              pyhd8ed1ab_0    conda-forge
ninja                     1.11.1.1                 pypi_0    pypi
nsight-compute            2024.1.1.4                    0    nvidia
numpy                     1.26.4          py310hb13e2d6_0    conda-forge
ocl-icd                   2.3.2                hd590300_1    conda-forge
ocl-icd-system            1.0.0                         1    conda-forge
openfold                  2.0.0                    pypi_0    pypi
openh264                  2.3.1                hcb278e6_2    conda-forge
openjpeg                  2.5.2                he7f1fd0_0  
openmm                    7.7.0           py310hccf1d78_1    conda-forge
openssl                   3.3.2                hb9d3cd8_0    conda-forge
opt-einsum                3.4.0                    pypi_0    pypi
opt-einsum-fx             0.1.4                    pypi_0    pypi
orjson                    3.10.7          py310h97a8848_0  
p11-kit                   0.24.1               hc5aa10d_0    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pandas                    2.2.2           py310hf9f9076_1    conda-forge
pcre2                     10.44                hba22ea6_2    conda-forge
pdbfixer                  1.8.1              pyh6c4a22f_0    conda-forge
perl                      5.32.1          7_hd590300_perl5    conda-forge
pillow                    9.5.0           py310h065c6d2_0    conda-forge
pip                       24.2               pyh8b19718_1    conda-forge
pixman                    0.40.0               h7f8727e_1  
platformdirs              4.3.6              pyhd8ed1ab_0    conda-forge
prody                     2.2.0           py310hd8f1fbe_1    conda-forge
prompt-toolkit            3.0.38             pyha770c72_0    conda-forge
prompt_toolkit            3.0.38               hd8ed1ab_0    conda-forge
protobuf                  4.25.3          py310ha8c1f0e_0    conda-forge
psutil                    6.0.0           py310hc51659f_0    conda-forge
pthread-stubs             0.4               hb9d3cd8_1002    conda-forge
py-cpuinfo                9.0.0                    pypi_0    pypi
pybind11                  2.13.6             pyh085cc03_1    conda-forge
pybind11-global           2.13.6             pyh085cc03_1    conda-forge
pycairo                   1.23.0          py310hd1222b9_0  
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pydantic                  2.9.2                    pypi_0    pypi
pydantic-core             2.23.4                   pypi_0    pypi
pydub                     0.25.1             pyhd8ed1ab_0    conda-forge
pygments                  2.15.1          py310h06a4308_1  
pynvml                    11.5.3                   pypi_0    pypi
pyparsing                 3.2.0              pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.14         hd12c33a_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-devtools           0.12.2             pyhd8ed1ab_0    conda-forge
python-multipart          0.0.9           py310h06a4308_0  
python-tzdata             2024.2             pyhd8ed1ab_0    conda-forge
python_abi                3.10                    5_cp310    conda-forge
pytorch                   2.1.2           py3.10_cuda12.1_cudnn8.9.2_0    pytorch
pytorch-cuda              12.1                 ha16c6d3_6    pytorch
pytorch-lightning         2.4.0              pyhd8ed1ab_0    conda-forge
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2024.2             pyhd8ed1ab_0    conda-forge
pyyaml                    5.4.1           py310h5764c6d_4    conda-forge
qhull                     2020.2               h434a139_5    conda-forge
rdkit                     2022.03.3       py310h1c297d8_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.30.2          py310h06a4308_0  
reportlab                 3.5.67          py310h0edfc0f_1  
requests                  2.32.3             pyhd8ed1ab_0    conda-forge
rpds-py                   0.10.6          py310hb02cf49_0  
ruamel.yaml               0.17.21         py310h1fa729e_3    conda-forge
ruamel.yaml.clib          0.2.8           py310h2372a71_0    conda-forge
s2n                       1.5.1                h3400bea_0    conda-forge
scikit-learn              1.5.2                    pypi_0    pypi
scipy                     1.14.1          py310ha3fb0e1_0    conda-forge
semantic_version          2.8.5              pyhd3eb1b0_0  
sentry-sdk                2.16.0             pyhd8ed1ab_0    conda-forge
setproctitle              1.3.3           py310h2372a71_0    conda-forge
setuptools                59.5.0          py310hff52083_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
sniffio                   1.3.0           py310h06a4308_0  
sqlalchemy                2.0.34          py310h00e1ef3_0  
starlette                 0.38.2          py310h06a4308_0  
svt-av1                   1.4.1                hcb278e6_0    conda-forge
sympy                     1.13.3          pypyh2585a3b_103    conda-forge
sysroot_linux-64          2.17                h4a8ded7_17    conda-forge
tbb                       2021.8.0             hdb19cb5_0  
threadpoolctl             3.5.0                    pypi_0    pypi
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toolz                     0.12.0          py310h06a4308_0  
torch-cluster             1.6.2                    pypi_0    pypi
torch-geometric           2.2.0                    pypi_0    pypi
torch-scatter             2.1.2                    pypi_0    pypi
torch-sparse              0.6.18                   pypi_0    pypi
torch-spline-conv         1.2.2                    pypi_0    pypi
torchmetrics              1.4.2              pyhd8ed1ab_0    conda-forge
torchtriton               2.1.0                     py310    pytorch
tqdm                      4.62.2             pyhd8ed1ab_0    conda-forge
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024b                hc8b5060_0    conda-forge
unicodedata2              15.1.0          py310ha75aee5_1    conda-forge
urllib3                   1.26.19            pyhd8ed1ab_0    conda-forge
uvicorn                   0.20.0          py310h06a4308_0  
wandb                     0.16.6             pyhd8ed1ab_1    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
websockets                10.4            py310h5eee18b_1  
wheel                     0.44.0             pyhd8ed1ab_0    conda-forge
x264                      1!164.3095           h166bdaf_2    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xorg-fixesproto           5.0               hb9d3cd8_1003    conda-forge
xorg-kbproto              1.0.7             hb9d3cd8_1003    conda-forge
xorg-libice               1.1.1                hb9d3cd8_1    conda-forge
xorg-libsm                1.2.4                he73a12e_1    conda-forge
xorg-libx11               1.8.4                h0b41bf4_0    conda-forge
xorg-libxau               1.0.11               hb9d3cd8_1    conda-forge
xorg-libxdmcp             1.1.5                hb9d3cd8_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            hb9d3cd8_1003    conda-forge
xorg-xextproto            7.3.0             hb9d3cd8_1004    conda-forge
xorg-xproto               7.0.31            h27cfd23_1007  
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

And here are my modules:

>>> help("modules")
Please wait a moment while I gather a list of all available modules...

/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/Bio/Application/__init__.py:39: BiopythonDeprecationWarning: The Bio.Application modules and modules relying on it have been deprecated.

Due to the on going maintenance burden of keeping command line application
wrappers up to date, we have decided to deprecate and eventually remove these
modules.

We instead now recommend building your command line and invoking it directly
with the subprocess module.
  warnings.warn(
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/Bio/codonalign/__init__.py:21: BiopythonExperimentalWarning: Bio.codonalign is an experimental module which may undergo significant changes prior to its future official release.
  warnings.warn(
[2024-10-23 21:09:40,015] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/shared/miniconda3/envs/diffdock/lib/python3.10/pkgutil.py:92: UserWarning: The numpy.array_api submodule is still experimental. See NEP 47.
  __import__(info.name)
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/Bio/pairwise2.py:278: BiopythonDeprecationWarning: Bio.pairwise2 has been deprecated, and we intend to remove it in a future release of Biopython. As an alternative, please consider using Bio.Align.PairwiseAligner as a replacement, and contact the Biopython developers if you still need the Bio.pairwise2 module.
  warnings.warn(
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/pydantic/experimental/__init__.py:7: PydanticExperimentalWarning: This module is experimental, its contents are subject to change and deprecation.
  warnings.warn(
Warning: importing 'simtk.openmm' is deprecated.  Import 'openmm' instead.
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/distributed/_shard/checkpoint/__init__.py:8: DeprecationWarning: torch.distributed._shard.checkpoint will be deprecated, use torch.distributed.checkpoint instead
  warnings.warn(
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/distributed/_sharded_tensor/__init__.py:8: DeprecationWarning: torch.distributed._sharded_tensor will be deprecated, use torch.distributed._shard.sharded_tensor instead
  warnings.warn(
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/distributed/_sharding_spec/__init__.py:8: DeprecationWarning: torch.distributed._sharding_spec will be deprecated, use torch.distributed._shard.sharding_spec instead
  warnings.warn(
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/cuda/__init__.py:611: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
No CUDA runtime is found, using CUDA_HOME='/shared/miniconda3/envs/diffdock'
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch_geometric/graphgym/config.py:19: UserWarning: Could not define global config object. Please install 'yacs' for using the GraphGym experiment manager via 'pip install yacs'.
  warnings.warn("Could not define global config object. Please install "
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/wandb/analytics/sentry.py:90: SentryHubDeprecationWarning: `sentry_sdk.Hub` is deprecated and will be removed in a future major release. Please consult our 1.x to 2.x migration guide for details on how to migrate `Hub` usage to the new API: https://docs.sentry.io/platforms/python/migration/1.x-to-2.x
  self.hub = sentry_sdk.Hub(client)
wandb: WARNING The v1 API is deprecated and will be removed in a future release.  Please move to v2 by setting the env var WANDB_REPORT_API_ENABLE_V2=True.  This will be on by default in a future release.
wandb: WARNING You can disable this message by setting the env var WANDB_REPORT_API_DISABLE_MESSAGE=True
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/wandb/apis/reports/v2/internal.py:383: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
  @validator("children", pre=True, each_item=True)
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/wandb/apis/reports/v2/interface.py:674: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
  @validator("panels")
wandb: ERROR kfp not found!  Please `pip install kfp`
wandb: ERROR Failed to patch kfp.components.create_component_from_func!  Please check if this package/module is installed!
wandb: ERROR Failed to patch kfp.components._python_op.create_component_from_func!  Please check if this package/module is installed!
wandb: ERROR Failed to patch kfp.components._python_op._get_function_source_definition!  Please check if this package/module is installed!
wandb: ERROR Failed to patch kfp.components._python_op.strip_type_hints!  Please check if this package/module is installed!
wandb: ERROR Failed to patch one or more kfp functions.  Patching @wandb_log decorator to no-op.
wandb: ERROR Failed to patch wandb.integration.kfp.wandb_log!  Please check if this package/module is installed!
Error importing optional module langchain
Traceback (most recent call last):
  File "/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/wandb/util.py", line 236, in import_module_lazy
    return sys.modules[name]
KeyError: 'langchain'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/wandb/util.py", line 265, in get_module
    return import_module_lazy(name)
  File "/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/wandb/util.py", line 240, in import_module_lazy
    raise ModuleNotFoundError
ModuleNotFoundError
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
Bio                 attn_core_inplace_cuda ihm                 runpy
BioSQL              attr                imaplib             sched
PIL                 attrs               imghdr              scipy
__future__          audioop             imp                 secrets
_abc                awscli              importlib           select
_aix_support        awscrt              importlib_resources selectors
_ast                base64              inspect             semantic_version
_asyncio            bdb                 io                  sentry_sdk
_awscrt             binascii            ipaddress           setproctitle
_bisect             binhex              isympy              setuptools
_blake2             bisect              itertools           shelve
_bootsubprocess     brotli              jinja2              shlex
_brotli             builtins            jmespath            shutil
_bz2                bz2                 joblib              signal
_cffi_backend       cProfile            json                simtk
_codecs             cairo               jsonschema          site
_codecs_cn          calendar            jsonschema_specifications six
_codecs_hk          certifi             keyword             sklearn
_codecs_iso2022     cffi                kiwisolver          smmap
_codecs_jp          cgi                 lib2to3             smtpd
_codecs_kr          cgitb               lightning_fabric    smtplib
_codecs_tw          charset_normalizer  lightning_utilities sndhdr
_collections        chunk               linecache           sniffio
_collections_abc    click               locale              socket
_compat_pickle      cmath               logging             socketserver
_compression        cmd                 lzma                socks
_contextvars        code                mailbox             sockshandler
_crypt              codecs              mailcap             spwd
_csv                codeop              markupsafe          sqlalchemy
_ctypes             collections         marshal             sqlite3
_ctypes_test        colorama            math                sre_compile
_curses             colorsys            matplotlib          sre_constants
_curses_panel       compileall          mimetypes           sre_parse
_datetime           concurrent          ml_collections      ssl
_decimal            configparser        mmap                starlette
_distutils_hack     contextlib          modelcif            stat
_elementtree        contextlib2         modulefinder        statistics
_functools          contextvars         mpmath              string
_hashlib            contourpy           msgpack             stringprep
_heapq              copy                multipart           struct
_imp                copyreg             multiprocessing     subprocess
_io                 cpuinfo             munkres             sunau
_json               crypt               netrc               sympy
_locale             cryptography        networkx            symtable
_lsprof             csv                 ninja               sys
_lzma               ctypes              nis                 sysconfig
_markupbase         curses              nntplib             syslog
_md5                cycler              ntpath              tabnanny
_multibytecodec     dataclasses         nturl2path          tarfile
_multiprocessing    datetime            numbers             telnetlib
_opcode             dateutil            numpy               tempfile
_operator           dbm                 nvfuser             termios
_osx_support        decimal             opcode              test
_pickle             deepspeed           openfold            textwrap
_posixshmem         devtools            openmm              this
_posixsubprocess    difflib             operator            threading
_py_abc             dis                 opt_einsum          threadpoolctl
_pydecimal          distro              opt_einsum_fx       time
_pyio               distutils           optparse            timeit
_queue              dllogger            orjson              tkinter
_random             dockerpycreds       os                  tlz
_remote_module_non_scriptable doctest             ossaudiodev         token
_ruamel_yaml        docutils            packaging           tokenize
_sha1               e3nn                pandas              toolz
_sha256             einops              pathlib             torch
_sha3               email               pdb                 torch_cluster
_sha512             encodings           pdbfixer            torch_geometric
_signal             ensurepip           pickle              torch_scatter
_sitebuiltins       enum                pickletools         torch_sparse
_socket             errno               pip                 torch_spline_conv
_sqlite3            esm                 pipes               torchgen
_sre                exceptiongroup      pkg_resources       torchmetrics
_ssl                executing           pkgutil             tqdm
_stat               fastapi             platform            trace
_statistics         faulthandler        platformdirs        traceback
_string             fcntl               plistlib            tracemalloc
_strptime           ffmpy               poplib              tree
_struct             filecmp             posix               triton
_symtable           fileinput           posixpath           tty
_sysconfigdata__linux_x86_64-linux-gnu filelock            pprint              turtle
_sysconfigdata_x86_64_conda_cos6_linux_gnu flash_attn          prody               turtledemo
_sysconfigdata_x86_64_conda_linux_gnu flash_attn_2_cuda   profile             types
_testbuffer         fnmatch             prompt_toolkit      typing
_testcapi           fontTools           pstats              typing_extensions
_testclinic         fractions           psutil              tzdata
_testimportmultiple fsspec              pty                 unicodedata
_testinternalcapi   ftplib              pwd                 unicodedata2
_testmultiphase     functools           py_compile          unittest
_thread             functorch           pybind11            urllib
_threading_local    gc                  pyclbr              urllib3
_tkinter            genericpath         pycparser           uu
_tracemalloc        getopt              pydantic            uuid
_uuid               getpass             pydantic_core       uvicorn
_warnings           gettext             pydoc               venv
_weakref            git                 pydoc_data          wandb
_weakrefset         gitdb               pydub               warnings
_xxsubinterpreters  glob                pyexpat             wave
_xxtestfuzz         gmpy2               pygments            wcwidth
_yaml               gradio              pylab               weakref
_zoneinfo           gradio_client       pynvml              webbrowser
abc                 graphlib            pynvml_utils        websockets
absl                greenlet            pyparsing           wheel
aifc                grp                 pytorch_lightning   wsgiref
aiofiles            gzip                pytz                xdrlib
altair              h11                 queue               xml
annotated_types     hashlib             quopri              xmlrpc
antigravity         heapq               random              xxlimited
anyio               hjson               rdkit               xxlimited_35
appdirs             hmac                re                  xxsubtype
argparse            hopper              readline            yaml
array               html                referencing         zipapp
ast                 http                reportlab           zipfile
asttokens           httpcore            reprlib             zipimport
asynchat            httpx               requests            zlib
asyncio             huggingface_hub     resource            zoneinfo
asyncore            idlelib             rlcompleter         
atexit              idna                rpds

You'll notice that inference is conspicuosly absent from my module list. Which brings up a couple of questions:

  1. Did I properly complete the setup and installation of DiffDock? I thought I had, but I must have missed a step somewhere. My understanding is that I didn't necessarily have to use the environment.yml file to generate the diffdock environment, and installation issue #203 makes it sound like there are several issues with missing packages following the standard installation procedure.
  2. Are there any other ways to validate the installation or run inference? As far as I can tell, there is no documentation on running DiffDock outside of the single paragraph in the README.

Thanks for the help -- been banging my head against this problem for nearly 3 weeks and thought I was THISCLOSE to a working install after getting OpenFold to work.
@blakemertz
Copy link
Author

Question for anyone out there -- does python -m inference assume that you are running in the DiffDock home directory where inference.py can be referenced to generate a module named 'inference'?

I tried the following:
python -m inference --config default_inference_args.yaml --protein_path examples/1a46_protein_processed.pdb --ligand examples/1a46_ligand.sdf --out_dir test
in the DiffDock home directory and am patiently waiting for results. However, it really looks like my terminal is just hanging (been waiting for 10+ minutes now....)

@blakemertz
Copy link
Author

UPDATE: DiffDock ran my example I just submitted, running out of the diffdock parent directory:

(diffdock) [centos@modbind-dy-modbind-cr-0-1 DiffDock]$ python -m inference --config default_inference_args.yaml --protein_path examples/1a46_protein_processed.pdb --ligand examples/1a46_ligand.sdf --out_dir test                                                    
/shared/binaries/github/DiffDock/utils/so3.py:59: RuntimeWarning: invalid value encountered in sqrt
  _exp_score_norms = np.sqrt(np.sum(_score_norms**2 * _pdf_vals, axis=1) / np.sum(_pdf_vals, axis=1) / np.pi)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 201/201 [01:27<00:00,  2.29it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 201/201 [01:36<00:00,  2.09it/s]
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/Bio/pairwise2.py:278: BiopythonDeprecationWarning: Bio.pairwise2 has been deprecated, and we intend to remove it in a future release of Biopython. As an alternative, please consider using Bio.Align.PairwiseAligner as a replacement, and contact the Biopython developers if you still need the Bio.pairwise2 module.
  warnings.warn(
Generating ESM language model embeddings
Downloading: "https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t33_650M_UR50D.pt" to /home/centos/.cache/torch/hub/checkpoints/esm2_t33_650M_UR50D.pt
Downloading: "https://dl.fbaipublicfiles.com/fair-esm/regression/esm2_t33_650M_UR50D-contact-regression.pt" to /home/centos/.cache/torch/hub/checkpoints/esm2_t33_650M_UR50D-contact-regression.pt
Processing 1 of 1 batches (2 sequences)
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/overrides.py:110: UserWarning: 'has_cuda' is deprecated, please use 'torch.backends.cuda.is_built()'
  torch.has_cuda,
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/overrides.py:111: UserWarning: 'has_cudnn' is deprecated, please use 'torch.backends.cudnn.is_available()'
  torch.has_cudnn,
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/overrides.py:117: UserWarning: 'has_mps' is deprecated, please use 'torch.backends.mps.is_built()'
  torch.has_mps,
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/overrides.py:118: UserWarning: 'has_mkldnn' is deprecated, please use 'torch.backends.mkldnn.is_available()'
  torch.has_mkldnn,
0it [00:00, ?it/s]/shared/binaries/github/DiffDock/datasets/parse_chi.py:91: RuntimeWarning: invalid value encountered in cast
  Y = indices.astype(int)
1it [00:57, 57.48s/it]

So success, right?!?! Can anyone tell me what the difference is in the .sdf files that are produced?

(base) [centos@modbind-dy-modbind-cr-0-1 complex_0]$ ls
rank10_confidence-4.57.sdf  rank1.sdf                  rank3_confidence-2.43.sdf  rank5_confidence-3.57.sdf  rank7_confidence-4.09.sdf  rank9_confidence-4.43.sdf
rank1_confidence-1.79.sdf   rank2_confidence-2.20.sdf  rank4_confidence-2.78.sdf  rank6_confidence-4.08.sdf  rank8_confidence-4.27.sdf

I couldn't find any clues in the preprint or the README here. Thanks.

@starwingc
Copy link

Hi so for my issue, I manually download all the file from the website, and I can run the diffdock calculation later on successfully. However, when I try to use the new one as you, with pytorch 2.5.1 and cuda 12.1, it takes me over 30 mins stuck at the first step of the new model. How long did you wait for the downloading parameter?

@starwingc
Copy link

UPDATE: DiffDock ran my example I just submitted, running out of the diffdock parent directory:

(diffdock) [centos@modbind-dy-modbind-cr-0-1 DiffDock]$ python -m inference --config default_inference_args.yaml --protein_path examples/1a46_protein_processed.pdb --ligand examples/1a46_ligand.sdf --out_dir test                                                    
/shared/binaries/github/DiffDock/utils/so3.py:59: RuntimeWarning: invalid value encountered in sqrt
  _exp_score_norms = np.sqrt(np.sum(_score_norms**2 * _pdf_vals, axis=1) / np.sum(_pdf_vals, axis=1) / np.pi)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 201/201 [01:27<00:00,  2.29it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 201/201 [01:36<00:00,  2.09it/s]
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/Bio/pairwise2.py:278: BiopythonDeprecationWarning: Bio.pairwise2 has been deprecated, and we intend to remove it in a future release of Biopython. As an alternative, please consider using Bio.Align.PairwiseAligner as a replacement, and contact the Biopython developers if you still need the Bio.pairwise2 module.
  warnings.warn(
Generating ESM language model embeddings
Downloading: "https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t33_650M_UR50D.pt" to /home/centos/.cache/torch/hub/checkpoints/esm2_t33_650M_UR50D.pt
Downloading: "https://dl.fbaipublicfiles.com/fair-esm/regression/esm2_t33_650M_UR50D-contact-regression.pt" to /home/centos/.cache/torch/hub/checkpoints/esm2_t33_650M_UR50D-contact-regression.pt
Processing 1 of 1 batches (2 sequences)
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/overrides.py:110: UserWarning: 'has_cuda' is deprecated, please use 'torch.backends.cuda.is_built()'
  torch.has_cuda,
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/overrides.py:111: UserWarning: 'has_cudnn' is deprecated, please use 'torch.backends.cudnn.is_available()'
  torch.has_cudnn,
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/overrides.py:117: UserWarning: 'has_mps' is deprecated, please use 'torch.backends.mps.is_built()'
  torch.has_mps,
/shared/miniconda3/envs/diffdock/lib/python3.10/site-packages/torch/overrides.py:118: UserWarning: 'has_mkldnn' is deprecated, please use 'torch.backends.mkldnn.is_available()'
  torch.has_mkldnn,
0it [00:00, ?it/s]/shared/binaries/github/DiffDock/datasets/parse_chi.py:91: RuntimeWarning: invalid value encountered in cast
  Y = indices.astype(int)
1it [00:57, 57.48s/it]

So success, right?!?! Can anyone tell me what the difference is in the .sdf files that are produced?

(base) [centos@modbind-dy-modbind-cr-0-1 complex_0]$ ls
rank10_confidence-4.57.sdf  rank1.sdf                  rank3_confidence-2.43.sdf  rank5_confidence-3.57.sdf  rank7_confidence-4.09.sdf  rank9_confidence-4.43.sdf
rank1_confidence-1.79.sdf   rank2_confidence-2.20.sdf  rank4_confidence-2.78.sdf  rank6_confidence-4.08.sdf  rank8_confidence-4.27.sdf

I couldn't find any clues in the preprint or the README here. Thanks.

Yes, your results are correct, be aware that the confidence score here are negative, and you can read the confidence score descrption from the README:


** c > 0 high confidence
** -1.5 < c < 0 moderate confidence
** c < -1.5 low confidence

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@blakemertz @starwingc