interleaved output is not reproducible with multiple threads

[briankegerreis]

When writing to interleaved stdout with the --stdout flag, reads appear to be written in random order as md5sums from repeat runs differ.

steps to reproduce with v0.23.3:

for TRY in 1 2 3; do fastp -w $THREADS --dont_eval_duplication -i in1.fq.gz -I in2.fq.gz -A -G -L -Q --stdout > interleave_attempt${TRY}.fastq; done
md5sum interleave*
for TRY in 1 2 3; do fastp -w $THREADS --dont_eval_duplication -i in1.fq.gz -I in2.fq.gz -A -G -L -Q -o split_attempt${TRY}_1.fastq -O split_attempt${TRY}_2.fastq; done
md5sum split*

[ckrushton]

I have also encountered this, and upon investigation, it appears to be the result of a race condition when writing to stdout with multiple threads. While writing to an output file, it FASTP seems to always be consistent, because a thread that completes faster than others will wait until the others complete before writing to the output. Unfortunately this logic is not applied when writing to stdout.

We ended up using --stdout with a single thread in a our workflow, then modifying the source code directly to allow FASTP to work with named pipes (currently it appears to write the output in a consistent order, but then block and hang after it finishes processing)

[tshtatland]

I also found fastp output to be irreproducible with the default number of threads (3), when writing to STDOUT, with fastp v0.23.2. Switching to single thread (-w 1), as suggested by @ckrushton, made the output reproducible.