Python envs for ML(TensorFlow and PyTorch)

Download and install miniconda3

curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

Use source ~/.bashrc to refresh/restart terminal before using conda

Create env

(base)conda create -n py39 python=3.9
(base)conda activate py39
(py39)

Add library path to $LD_LIBRARY_PATH

mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

Use conda to install cudatoolkit and cudnn

(py39)conda install -c conda-forge cudatoolkit=11.3 cudnn=8.1

Use pip to install opencv-python

(py39)pip install opencv-python

Use pip to install TensorFlow

(py39)pip install tensorflow==2.11.*

Check if it is cuda enabled

(py39)python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Work around with TensorFlow2

• If the code is implemented by Tensorflow1, then change import tensorflow as tf to import tensorflow.compat.v1 as tf
• If slim is not working, then package tf_slim is needed, also, in the code, replace slim = tf.contrib.slim by import tf_slim as slim
• If libnvinfer.so.7/libnvinfer_plugin.so.7 symbol not found is reported, then download the package tensorRT to get libnvinfer.so.8/libnvinfer_plugin.so.8and create the soft-link of libnvinfer.so.7/libnvinfer_plugin.so.7 in path env/py39/lib/
• If libtinfo.so.6 contains no version information of vim, bash, etc. create a soft-link of /lib/x86_64-linux-gnu/libtinfo.so.6 in path env/py39/lib/
• If ValueError: Memory growth cannot differ between GPU devices is reported, please insert the following lines:

gpus = tf.config.list_physical_devices('GPU') # find this line
if gpus:
  try:
    # Currently, memory growth needs to be the same across GPUs
    for gpu in gpus:
      tf.config.experimental.set_memory_growth(gpu, True)
    logical_gpus = tf.config.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Memory growth must be set before GPUs have been initialized
    print(e)

Use pip to install PyTorch(cuda enabled ver.)

(py39)pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113

Check if it is cuda enabled

(py39)python -c "import torch; print(torch.cuda.is_available())"

Work around with PyTorch

• If cudaGetStatusString symbol not found is reported, then there could be something wrong with torch installation, considering using the line above instead of the name pytorch only in requirements.txt
• If libtinfo.so.6 contains no version information of vim, bash, etc. create a soft-link of /lib/x86_64-linux-gnu/libtinfo.so.6 in path env/py39/lib/

Summary

One possible combination

Ubuntu      20.04
python      3.9
cudadriver  525.x(up to cuda 12.0)
cudatookit  11.3
cuDNN       8.1
opencv      4.6
tensorflow  2.11
pytorch     1.12.1