Singularity no space left in tmp directory

[SR-Epi-lab]

Our lab is attempting to run distiller on cluster using slurm scheduler and singularity, and we are consistently getting the same error with regards to not having enough space for the staging file. We don’t get this issue on truncated test runs. I believe that this is an issue with singularity, we have tried to set the tmp directory locally outside of the cluster in the configuration files, as well as using export SINGULARITY_BINDPATH="$HOME/tmp:/tmp" in the .bashrc file, but still get the same issue. If staging the tmp file outside of the cluster is the correct direction, what is the definitive way to do this? Our lab has used the lsf previously without issue and are having difficulty with the change over to slurm, your help is significant to us since IBM-LSF is not available for our HPC resources.

Any assistance would be greatly appreciated.

Error executing process > 'map_parse_sort_chunks (library:S28 run:lane1 chunk:02)'

Caused by:
Process map_parse_sort_chunks (library:S28 run:lane1 chunk:02) terminated with an error exit status (255)

Command executed:

TASK_TMP_DIR=$(mktemp -d -p $HOME/tmp distiller.tmp.XXXXXXXXXX)
touch S28.lane1.hg19.02.bam

bwa mem -t 8 -SP hg19.fa S28.lane1.02.1.fastq.gz S28.lane1.02.2.fastq.gz | pairtools parse --drop-sam --drop-readid --add-columns mapq -c hg19.reduced.chrom.sizes | pairtools sort --nproc 8 -o S28.lane1.hg19.02.pairsam.gz --tmpdir $TASK_TMP_DIR | cat

rm -rf $TASK_TMP_DIR

Command exit status:
255

Command output:
(empty)

Command error:
INFO: Converting SIF file to temporary sandbox...
FATAL: while extracting /rs01/home/user/lab/09AUG/work/singularity/open2c-distiller_env-0.3.4.img: root filesystem extraction failed: failed to copy content in staging file: write /tmp/rootfs-3437578670/archive-716607293: no space left on device

Work dir:
/rs01/home/user/lab/09AUG/work/81/a77bd3cb2e6e918ddc59ca797a715c

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh