Plugins offline: Nextflow self-contained distribution allows only core plugins

[ewels]

Talking to @Alexander-Stuckey about running offline with plugins, I pointed to these docs for fetching plugins for offline use, but it seems that it doesn't work if you use the -all distribution of Nextflow:

N E X T F L O W  ~  version 23.09.1-edge
Launching `/pgen_int_work/BRS/rif/helix_replacement_project/sarek/resources/nf-core-sarek_3.3.1/3_3_1/main.nf` [distracted_meninsky] DSL2 - revision: 19395081df
WARN: Nextflow self-contained distribution allows only core plugins -- User config plugins will be ignored: [email protected]

This is currently the recommended method from nf-core, so it's causing some confusion.

We're starting to see these kinds of problems a lot now, since @nf-core pipelines are adopting the nf-validation plugin. I think that this is probably some of the first very widespread adoption of a non-core Nextflow plugin.

[ewels]

Related to #4329

[bentsherman]

I think #4340 will make it possible to get around this problem by using the regular Nextflow distribution and downloading the desired set of core plugins. Otherwise we will need to change the self-contained distribution to support non-core plugins, and I don't know how difficult that would be.

[ewels]

Ok, yeah sounds like we probably need to drop the -all distribution at some point. I know it's been spoken about before hand.

I guess tied to this is that we could do with a user friendly way to distribute Nextflow in offline systems to replace that dist. Running Nextflow on another system and then transferring a bunch of directories is quite indirect and unintuitive.

Could we have some kind of "downloader" web pages, that generates an -all dist on the fly with the selected plugins? Or some other way that we can make it easy for people to easily get a Nextflow install that works offline?

Related to #2028 and #2951

[bentsherman]

Could we extend nf-core download for this purpose, or is it only used for container images?

You could write a bash script to install everything you need for offline usage:

1. install nextflow like normal
2. install nextflow plugins: nextflow plugin install <plugin>[@<version>]
3. pull pipeline repos: nextflow pull <project>
4. download container images
5. create an archive of items 1-4
6. transfer archive to offline system
7. extract that archive over ssh?

[ewels]

Could be an option, yeah 👍🏻

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bentsherman]

With Nextflow 24.07.0-edge, the "all" release has been replaced with the "dist" release which (1) includes fewer core plugins (since most of them aren't used offline anyway) and (2) has the ability to use third-party plugins. So it should now be possible to use the "dist" release offline with e.g. nf-validation.

Let's have some people test this out and if it works we can close this issue.

[pditommaso]

This should be solved as commented by Ben.

[ewels]

Reopening, as the docs are outdated and need to be fixed.

[ewels]

'cc @christopher-hakkaart - is this something you could add to your backlog please?

Note that I don't know what the correct procedure is now for running offline with custom plugins.

[ewels]

Reading again, the update may be as simple as basically s/all/dist - but I'm hoping that we have a better method for grabbing plugins for offline use now..? Hopefully? 🤞🏻

[pditommaso]

does nextflow plugin install not work for you?