Workflow output definition (second preview)

[bentsherman]

Close #5103

TODO:

language improvements:

• support dynamic path
• move publish options to config
• remove publish section from process definition

runtime improvements:

• prevent leading/trailing slashes
• report unused target names
• detect file collisions at runtime ?
• detect outputs that aren't published or used downstream ?

new features (might be handled separately):

• support JSON index file
• generate output schema ?
• include publish targets in inspect command ?

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	7d5d69b
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67194094986ac9000862eb27
😎 Deploy Preview	https://deploy-preview-5185--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[bentsherman]

Second preview is ready for review, POCs have also been updated:

• Proposal: DSL2+ nf-core/fetchngs#312
• Workflow output definition nf-core/rnaseq#1227

[nvnieuwk]

I just tried the dynamic publishing with some dummy code and it worked perfectly! I'm looking forward to implementing this in my pipelines 🥳

[christopher-hakkaart]

Added small language suggestion.

[pditommaso]

I'm really not sure about this, I see the need for modules but it will make the writing of workflows much more verbose. Could not both approach co-exist?

[bentsherman]

It's too confusing. If a process publishes some outputs by default and you don't want to publish them, then you have to explicitly disable them. And one of the goals of workflow outputs was to have all published outputs in one place, instead of being scattered across the pipeline

My initial feeling was to only allow publishing in the entry workflow, but that would also create a lot of boilerplate to pass all the channels up from the subworkflows. So a happy middle ground for now is to allow publishing from any workflow, but not processes

[bentsherman]

Also, the original motivation for process publishing is to provide some sensible publishing defaults for a module. I think this is better handled by providing an entry workflow with the module:

process ASPERA_CLI {
    input:
    // ...

    output:
    tuple val(meta), path("*fastq.gz"), emit: fastq
    tuple val(meta), path("*md5")     , emit: md5

    script:
    // ...
}

workflow {
    main:
    ASPERA_CLI ( params.input, params.args )

    publish:
    ASPERA_CLI.out.fastq >> 'fastq'
    ASPERA_CLI.out.md5 >> 'md5'
}

This gives an example of how to use the module, both params and publishing, but they are opt-in. The user can incorporate these publishing rules into their pipeline explicitly, if they want to.

[pditommaso]

@bentsherman can you please have a look at the conflicts