Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve error message reported on failed run #5437

Open
pditommaso opened this issue Oct 28, 2024 · 3 comments · May be fixed by #5513
Open

Improve error message reported on failed run #5437

pditommaso opened this issue Oct 28, 2024 · 3 comments · May be fixed by #5513
Assignees
@jorgee

Comments

@pditommaso
Copy link
Member

pditommaso commented Oct 28, 2024 •
edited
Loading

When a task execution fails, along with the main error message, are display several other error messages that likely are a side effect of the main exception.

ERROR ~ Error executing process > 'NFCORE_RNASEQ:RNASEQ:BAM_MARKDUPLICATES_PICARD:PICARD_MARKDUPLICATES (RAP1_UNINDUCED_REP2)'

Caused by:
  Wave provisioning for container 'wave.seqera.io/wt/bd73f9b3f459/wave/build:picard-3.1.1--332f206f295b500a' did not complete successfully
  - Reason: Container security scan operation found one or more vulnerabilities with severity: HIGH
  - Find out more here: https://wave.seqera.io/view/scans/sc-40b7ffea831b7e8b_2


Command executed:

  picard \
      -Xmx12288M \
      MarkDuplicates \
      --ASSUME_SORTED true --REMOVE_DUPLICATES false --VALIDATION_STRINGENCY LENIENT --TMP_DIR tmp \
      --INPUT RAP1_UNINDUCED_REP2.sorted.bam \
      --OUTPUT RAP1_UNINDUCED_REP2.markdup.sorted.bam \
      --REFERENCE_SEQUENCE genome_gfp.fasta \
      --METRICS_FILE RAP1_UNINDUCED_REP2.markdup.sorted.MarkDuplicates.metrics.txt
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_RNASEQ:RNASEQ:BAM_MARKDUPLICATES_PICARD:PICARD_MARKDUPLICATES":
      picard: $(echo $(picard MarkDuplicates --version 2>&1) | grep -o 'Version:.*' | cut -f2- -d:)
  END_VERSIONS

Command exit status:
  -

Command output:
  (empty)

Work dir:
  s3://nextflow-ci/test/fe/1f2b95271cdad59dbd5dc2aefa7ba2

Container:
  wave.seqera.io/wt/bd73f9b3f459/wave/build:picard-3.1.1--332f206f295b500a

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details
ERROR ~ Failed to sanitize XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler

 -- Check '.nextflow.log' file for details
ERROR ~ Unexpected error [AbortedException]

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details
-[nf-core/rnaseq] Pipeline completed with errors-
WARN: Killing running tasks (1)
ERROR ~ A DataflowVariable can only be assigned once. Use bind() to allow for equal values to be passed into already-bound variables.

 -- Check '.nextflow.log' file for details

Those error messages do not provide any value and are just confusing. This is also likely a regression of this PR (not 100% sure).

The goal of this issue is to better handle this error reporting to only show the main error message and skip secondary exception.

Log details

Oct-27 20:56:32.584 [Task submitter] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_RNASEQ:RNASEQ:BAM_MARKDUPLICATES_PICARD:PICARD_MARKDUPLICATES (RAP1_UNINDUCED_REP2)'

Caused by:
  Wave provisioning for container 'wave.seqera.io/wt/bd73f9b3f459/wave/build:picard-3.1.1--332f206f295b500a' did not complete successfully
  - Reason: Container security scan operation found one or more vulnerabilities with severity: HIGH
  - Find out more here: https://wave.seqera.io/view/scans/sc-40b7ffea831b7e8b_2


Command executed:

  picard \
      -Xmx12288M \
      MarkDuplicates \
      --ASSUME_SORTED true --REMOVE_DUPLICATES false --VALIDATION_STRINGENCY LENIENT --TMP_DIR tmp \
      --INPUT RAP1_UNINDUCED_REP2.sorted.bam \
      --OUTPUT RAP1_UNINDUCED_REP2.markdup.sorted.bam \
      --REFERENCE_SEQUENCE genome_gfp.fasta \
      --METRICS_FILE RAP1_UNINDUCED_REP2.markdup.sorted.MarkDuplicates.metrics.txt
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_RNASEQ:RNASEQ:BAM_MARKDUPLICATES_PICARD:PICARD_MARKDUPLICATES":
      picard: $(echo $(picard MarkDuplicates --version 2>&1) | grep -o 'Version:.*' | cut -f2- -d:)
  END_VERSIONS

Command exit status:
  -

Command output:
  (empty)

Work dir:
  s3://nextflow-ci/test/fe/1f2b95271cdad59dbd5dc2aefa7ba2

Container:
  wave.seqera.io/wt/bd73f9b3f459/wave/build:picard-3.1.1--332f206f295b500a

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
Oct-27 20:56:32.585 [Task submitter] INFO  nextflow.Session - Execution cancelled -- Finishing pending tasks before exit
Oct-27 20:56:32.593 [Task submitter] ERROR nextflow.Nextflow - Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting
Oct-27 20:56:32.594 [main] DEBUG nextflow.Session - Session await > all processes finished
Oct-27 20:56:32.605 [Actor Thread 60] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_RNASEQ:RNASEQ:QUANTIFY_STAR_SALMON:SE_GENE; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Oct-27 20:56:32.606 [Actor Thread 50] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_RNASEQ:RNASEQ:BAM_MARKDUPLICATES_PICARD:BAM_STATS_SAMTOOLS:SAMTOOLS_IDXSTATS; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Oct-27 20:56:32.612 [Actor Thread 47] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_RNASEQ:RNASEQ:QUANTIFY_PSEUDO_ALIGNMENT:SE_GENE; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Oct-27 20:56:32.617 [Actor Thread 61] ERROR nextflow.extension.OperatorImpl - @unknown
com.amazonaws.SdkClientException: Failed to sanitize XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
	at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:240)
	at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:339)
	at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:118)
	at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:107)
	at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
	at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)
	at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)
	at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1795)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5558)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5505)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:950)
	at nextflow.cloud.aws.nio.S3Client.listObjects(S3Client.java:140)
	at nextflow.cloud.aws.nio.util.S3ObjectSummaryLookup.lookup(S3ObjectSummaryLookup.java:89)
	at nextflow.cloud.aws.nio.S3FileSystemProvider.readAttr0(S3FileSystemProvider.java:702)
	at nextflow.cloud.aws.nio.S3FileSystemProvider.readAttributes(S3FileSystemProvider.java:686)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.FilesEx.createDirIfNotExists(FilesEx.groovy:1352)
	at nextflow.file.FileCollector.saveTo0(FileCollector.groovy:225)
	at nextflow.file.FileCollector.saveTo(FileCollector.groovy:251)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.CollectFileOp.emitItems(CollectFileOp.groovy:187)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.Closure.call(Closure.java:433)
	at groovy.lang.Closure.call(Closure.java:422)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.DataflowHelper$3.afterStop(DataflowHelper.groovy:257)
	at groovyx.gpars.dataflow.operator.DataflowProcessor.fireAfterStop(DataflowProcessor.java:324)
	at groovyx.gpars.dataflow.operator.DataflowProcessorActor.afterStop(DataflowProcessorActor.java:59)
	at jdk.internal.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:633)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:624)
	at groovyx.gpars.actor.Actor.callDynamic(Actor.java:369)
	at groovyx.gpars.actor.Actor.handleTermination(Actor.java:320)
	at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:86)
	at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.amazonaws.AbortedException: 
	at com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:61)
	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:89)
	at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
	at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:270)
	at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:313)
	at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:188)
	at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
	at java.base/java.io.BufferedReader.read1(BufferedReader.java:211)
	at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
	at java.base/java.io.Reader.read(Reader.java:250)
	at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:212)
	... 67 common frames omitted
Oct-27 20:56:32.626 [Actor Thread 65] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Oct-27 20:56:32.626 [Actor Thread 61] DEBUG nextflow.Session - Session aborted -- Cause: Failed to sanitize XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
Oct-27 20:56:32.633 [Actor Thread 66] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 17; slices: 1; internal sort time: 0.015 s; external sort time: 0.005 s; total time: 0.02 s
Oct-27 20:56:32.634 [Actor Thread 66] ERROR nextflow.extension.OperatorImpl - @unknown
com.amazonaws.AbortedException: 
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:880)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:757)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5558)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5505)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:950)
	at nextflow.cloud.aws.nio.S3Client.listObjects(S3Client.java:140)
	at nextflow.cloud.aws.nio.util.S3ObjectSummaryLookup.lookup(S3ObjectSummaryLookup.java:89)
	at nextflow.cloud.aws.nio.S3FileSystemProvider.readAttr0(S3FileSystemProvider.java:702)
	at nextflow.cloud.aws.nio.S3FileSystemProvider.readAttributes(S3FileSystemProvider.java:686)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.FilesEx.createDirIfNotExists(FilesEx.groovy:1352)
	at nextflow.file.FileCollector.saveTo0(FileCollector.groovy:225)
	at nextflow.file.FileCollector.saveTo(FileCollector.groovy:251)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.CollectFileOp.emitItems(CollectFileOp.groovy:187)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.Closure.call(Closure.java:433)
	at groovy.lang.Closure.call(Closure.java:422)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.DataflowHelper$3.afterStop(DataflowHelper.groovy:257)
	at groovyx.gpars.dataflow.operator.DataflowProcessor.fireAfterStop(DataflowProcessor.java:324)
	at groovyx.gpars.dataflow.operator.DataflowProcessorActor.afterStop(DataflowProcessorActor.java:59)
	at jdk.internal.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:633)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:624)
	at groovyx.gpars.actor.Actor.callDynamic(Actor.java:369)
	at groovyx.gpars.actor.Actor.handleTermination(Actor.java:320)
	at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:86)
	at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.amazonaws.http.timers.client.SdkInterruptedException: null
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:935)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:921)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1115)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	... 53 common frames omitted
Oct-27 20:56:32.695 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Oct-27 20:56:32.695 [main] DEBUG nextflow.Session - Session await > all barriers passed
Oct-27 20:56:32.699 [Actor Thread 61] DEBUG nextflow.Session - The following nodes are still active:
  [operator] collectFile

Oct-27 20:56:32.706 [Actor Thread 61] DEBUG nextflow.Session - 
Thread[Notification Thread,9,system]

Thread[Actor Thread 47,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 60,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[FileTransfer-9,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 68,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Reference Handler,10,system]
  [email protected]/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  [email protected]/java.lang.ref.Reference.processPendingReferences(Reference.java:253)
  [email protected]/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:215)

Thread[process reaper (pid 2632489),10,system]
  [email protected]/java.lang.ProcessHandleImpl.waitForProcessExit0(Native Method)
  [email protected]/java.lang.ProcessHandleImpl$1.run(ProcessHandleImpl.java:150)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 62,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[FileTransfer-4,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[HttpClient-1-Worker-5,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Thread-4,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)
  [email protected]/java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515)
  [email protected]/java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677)
  app//nextflow.util.SimpleAgent.run(SimpleAgent.groovy:89)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:412)
  app//groovy.lang.Closure.run(Closure.java:505)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 63,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 57,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 58,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Task submitter,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1757)
  app//nextflow.processor.TaskPollingMonitor.awaitSlots(TaskPollingMonitor.groovy:399)
  app//nextflow.processor.TaskPollingMonitor.submitLoop(TaskPollingMonitor.groovy:420)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:412)
  app//groovy.lang.Closure.run(Closure.java:505)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[pool-2-thread-4,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 64,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 67,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 26,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Signal Dispatcher,9,system]

Thread[JGit-FileStoreAttributeReader-1,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 59,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Finalizer,8,system]
  [email protected]/java.lang.Object.wait(Native Method)
  [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  [email protected]/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:172)

Thread[FileTransfer-7,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[pool-2-thread-3,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[pool-2-thread-1,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[FileTransfer-2,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[java-sdk-http-connection-reaper,5,main]
  [email protected]/java.lang.Thread.sleep(Native Method)
  com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:188)

Thread[FileTransfer-5,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[HttpClient-1-Worker-6,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[FileTransfer-10,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 50,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[S3TransferManager-1,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 69,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[FileTransfer-3,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[FileTransfer-6,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 65,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Common-Cleaner,8,InnocuousThreadGroup]
  [email protected]/java.lang.Object.wait(Native Method)
  [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  [email protected]/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
  [email protected]/java.lang.Thread.run(Thread.java:833)
  [email protected]/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:162)

Thread[FileTransfer-8,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[pool-2-thread-2,5,main]
  [email protected]/java.lang.Object.wait(Native Method)
  [email protected]/java.lang.Object.wait(Object.java:338)
  [email protected]/java.lang.ProcessImpl.waitFor(ProcessImpl.java:434)
  app//nextflow.executor.local.LocalTaskHandler$_submit_closure1.doCall(LocalTaskHandler.groovy:109)
  app//nextflow.executor.local.LocalTaskHandler$_submit_closure1.call(LocalTaskHandler.groovy)
  app//groovy.lang.Closure.run(Closure.java:505)
  [email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  [email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[main,5,main]
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1091)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:633)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:624)
  app//org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:391)
  app//org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:330)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  [email protected]/java.lang.invoke.LambdaForm$DMH/0x00007f8e880b2c00.invokeInterface(LambdaForm$DMH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e88175400.invoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e8892a000.invoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e884a9c00.guardWithCatch(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e884aa800.reinvoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e884aac00.guard(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e884aa800.reinvoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e884aac00.guard(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e884aa800.reinvoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e88d54800.invoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e880b9000.invokeExact_MT(LambdaForm$MH)
  app//org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
  [email protected]/java.lang.invoke.LambdaForm$DMH/0x00007f8e880a6c00.invokeStatic(LambdaForm$DMH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e884a8800.invoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
  Script_e83567872dab8752$_runScript_closure13$_closure15$_closure19.doCall(Script_e83567872dab8752:135)
  Script_e83567872dab8752$_runScript_closure13$_closure15$_closure19.doCall(Script_e83567872dab8752)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:279)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:412)
  app//nextflow.script.WorkflowMetadata$_invokeOnComplete_closure3.doCall(WorkflowMetadata.groovy:428)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:279)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:422)
  app//org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2394)
  app//org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2379)
  app//org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2420)
  app//nextflow.script.WorkflowMetadata.invokeOnComplete(WorkflowMetadata.groovy:426)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
  app//nextflow.script.WorkflowMetadata$_closure2.doCall(WorkflowMetadata.groovy:286)
  app//nextflow.script.WorkflowMetadata$_closure2.call(WorkflowMetadata.groovy)
  app//groovy.lang.Closure.run(Closure.java:505)
  app//nextflow.Session.shutdown0(Session.groovy:741)
  app//nextflow.Session.destroy(Session.groovy:694)
  app//nextflow.script.ScriptRunner.shutdown(ScriptRunner.groovy:260)
  app//nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:146)
  app//nextflow.cli.CmdRun.run(CmdRun.groovy:376)
  app//nextflow.cli.Launcher.run(Launcher.groovy:503)
  app//nextflow.cli.Launcher.main(Launcher.groovy:658)

Thread[FileTransfer-1,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[HttpClient-1-SelectorManager,5,main]
  [email protected]/sun.nio.ch.EPoll.wait(Native Method)
  [email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:205)
  [email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
  [email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  platform/[email protected]/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:889)

Thread[Actor Thread 55,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 66,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  [email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[S3TransferManager-2,5,main]
  [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
  [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
  [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Thread[Actor Thread 61,5,main]
  [email protected]/java.lang.Thread.dumpThreads(Native Method)
  [email protected]/java.lang.Thread.getAllStackTraces(Thread.java:1662)
  app//nextflow.util.SysHelper.dumpThreads(SysHelper.groovy:188)
  app//nextflow.Session.abort(Session.groovy:800)
  [email protected]/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e883de400.invoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e88105800.guardWithCatch(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e88106000.reinvoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e88106400.guard(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e88106000.reinvoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e88106400.guard(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e88106000.reinvoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e8816d400.invoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e880b9000.invokeExact_MT(LambdaForm$MH)
  app//org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
  [email protected]/java.lang.invoke.LambdaForm$DMH/0x00007f8e880a6c00.invokeStatic(LambdaForm$DMH)
  [email protected]/java.lang.invoke.LambdaForm$MH/0x00007f8e88105000.invoke(LambdaForm$MH)
  [email protected]/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
  app//nextflow.extension.DataflowHelper$3.afterStop(DataflowHelper.groovy:261)
  app//groovyx.gpars.dataflow.operator.DataflowProcessor.fireAfterStop(DataflowProcessor.java:324)
  app//groovyx.gpars.dataflow.operator.DataflowProcessorActor.afterStop(DataflowProcessorActor.java:59)
  jdk.internal.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
  [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:633)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:624)
  app//groovyx.gpars.actor.Actor.callDynamic(Actor.java:369)
  app//groovyx.gpars.actor.Actor.handleTermination(Actor.java:320)
  app//groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:86)
  app//groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
  [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  [email protected]/java.lang.Thread.run(Thread.java:833)

Oct-27 20:56:32.708 [Actor Thread 61] ERROR nextflow.Nextflow - Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting
Oct-27 20:56:32.710 [main] INFO  nextflow.Nextflow - -[nf-core/rnaseq] Pipeline completed with errors-
Oct-27 20:56:32.715 [main] WARN  n.processor.TaskPollingMonitor - Killing running tasks (1)
Oct-27 20:56:32.723 [Actor Thread 61] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Oct-27 20:56:32.724 [Actor Thread 61] ERROR nextflow.extension.DataflowHelper - @unknown
java.lang.IllegalStateException: A DataflowVariable can only be assigned once. Use bind() to allow for equal values to be passed into already-bound variables.
	at groovyx.gpars.dataflow.expression.DataflowExpression.bindUnique(DataflowExpression.java:383)
	at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:88)
	at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Oct-27 20:56:32.731 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=33; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=10; submittedCount=0; runningCount=-1; retriesCount=0; abortedCount=1; succeedDuration=22m 42s; failedDuration=0ms; cachedDuration=0ms;loadCpus=-4; loadMemory=0; peakRunning=2; peakCpus=8; peakMemory=30 GB; ]
Oct-27 20:56:32.731 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Oct-27 20:56:32.756 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Oct-27 20:56:33.697 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Oct-27 20:56:34.045 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Oct-27 20:56:34.073 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Oct-27 20:56:34.073 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-wave
Oct-27 20:56:34.073 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Oct-27 20:56:34.074 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-tower
Oct-27 20:56:34.074 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Oct-27 20:56:34.074 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-amazon
Oct-27 20:56:34.074 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Oct-27 20:56:34.074 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-schema
Oct-27 20:56:34.079 [main] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-15224585989101541662
Oct-27 20:56:34.081 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
@jorgee
Copy link
Contributor

jorgee commented Nov 12, 2024 •
edited
Loading

@pditommaso I was trying to reproduce the error but I am not able to reproduce the secondary errors. Could you provide more details about the execution that causes these secondary errors? Is it from a ci test?

@pditommaso
Copy link
Member Author

I think this happens when using data hosted on S3. I usually see when execution of nf-core/rnaseq with test profile has a failure.

The error stack should give some more hints. Likely this log should/could be avoided and leave to abort method the printing of the exception

@jorgee
Copy link
Contributor

jorgee commented Nov 14, 2024

I have been able to reproduce the error using the nf-core/rnaseq with awsbatch and wave with scan. The logs that where before aborted where the ones that produce the secondary exceptions. Moreover, when there is a cancellation and an abort in the same session, the error is notified twice, so this is also why the nf-core messages appear twice. I will create a PR with the changes. Regarding the secondary exceptions, I would move the exception messages to debug (the will be printed in .nextflow log but not in the console). Is it fine or is it better to do not print them?

@jorgee jorgee linked a pull request Nov 15, 2024 that will close this issue
Removing secondary exceptions from console #5513
Draft
@jorgee jorgee self-assigned this Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jorgee jorgee

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Removing secondary exceptions from console nextflow-io/nextflow
2 participants
@pditommaso @jorgee