Output redirected to a file appears to get truncated.

[pwrightkcl]

I am running a script that includes some Niftyreg commands and redirecting the output to a log file. After reg_aladin is complete, its output and several other lines before and after it are removed from the log. I tried monitoring the log live using tail -f and it output "file truncated" at the end of the reg_aladin output, followed by the content of the log file, starting from the first line, but missing the reg_aladin output and two lines before and after. The script includes subsequent lines using reg_aladin and reg_f3d, but these do not get truncated, and the truncation appears to occur before the subsequent lines are reached.

I do not know if this is caused by reg_aladin or how to determine whether it is but am raising the issue here in the hope someone can advise me.

I have tried running my script both with a standard redirect and prefixing it with stdbuf -oL. It is the latter that let me monitor the progress of the log file and catch the truncation taking place.

[pwrightkcl]

Here's what the log file looked like as it was getting truncated. You can see the two log lines prior to the first reg_aladin line are removed. There should be another three additional log lines after reg_aladin and before the line "Extracting volumes from GIF prior." that are also removed.

Note that reg_aladin runs again after "Aligning ICBM template to T1." but no further truncation occurs.

Wed Apr 19 09:57:53 UTC 2023
Creating local script:
/nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/ScriptBaMoS_sub-0_.sh
Running /nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/ScriptBaMoS_sub-0_.sh.
Scratch path: /tmp/sub-0_BaMoSCross__
Copying T1 image and converting to float.
Aligning FLAIR to T1.
[/usr/local/bin/reg_aladin] 
# snip
[reg_aladin_sym] * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
tail: sub-0/ses-v3/anat/sub-0_BaMoS.log: file truncated
Wed Apr 19 09:57:53 UTC 2023
Creating local script:
/nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/ScriptBaMoS_sub-0_.sh
Running /nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/ScriptBaMoS_sub-0_.sh.
Scratch path: /tmp/sub-0_BaMoSCross__
Extracting volumes from GIF prior.
Creating artefact mask based on GIF parcellation.
Aligning ICBM template to T1.
[/usr/local/bin/reg_aladin] 

[pwrightkcl]

Here is some actual code. I'm afraid it's long: there's a wrapper script containing the command with the redirection to the log file, that command generates a third script and executes it. I am trying to snip out irrelevant lines but show the parts relevant to the truncated log, including lines either side of reg_aladin.

Lines from the wrapper script:

BaMoS_log="${BaMoS_results_path}"/${id}_BaMoS.log
date > "$BaMoS_log"
stdbuf -oL "$script_path"/BaMoS_RISAPS.sh "$id" "$input_FLAIR" "$input_T1" "$GIF_results_path" "$BaMoS_results_path" >> "$BaMoS_log" 2>&1

Lines from the BaMoS_RISAPS.sh, which generates a third script:

echo "echo \"Scratch path: $PathScratch\"" >> "$NameScript"
# snip
{
  echo "cp ${PathResults}/* ${PathScratch}/."

  echo "echo \"Copying T1 image and converting to float.\""
  echo "cp ${ImageT1} ${PathScratch}/T1_${ID}.nii.gz"
  echo "${PathSeg}/seg_maths ${PathScratch}/T1_${ID}.nii.gz -odt float ${PathScratch}/T1_${ID}.nii.gz"

  echo "echo \"Aligning FLAIR to T1.\""
  echo "${PathReg}/reg_aladin -ref ${PathScratch}/T1_${ID}.nii.gz -flo ${ImageFLAIR} \
                              -aff ${PathScratch}/Aff_FLAIRtoT1.txt -res ${PathScratch}/FLAIR_${ID}.nii.gz -rigOnly"
  echo "${PathSeg}/seg_maths ${PathScratch}/FLAIR_${ID}.nii.gz -odt float ${PathScratch}/FLAIR_${ID}.nii.gz"
} >> "$NameScript"
# snip
{
  echo "echo \"Copying scratch directory to results directory.\""
  echo "cp ${PathScratch}/* ${PathResults}/."

  echo "echo \"Extracting volumes from GIF prior.\""
} >> "$NameScript"
#snip
  {
    echo "echo \"Aligning ICBM template to T1.\""
    echo "${PathReg}/reg_aladin -ref ${PathScratch}/T1_${ID}.nii.gz -flo ${PathICBM}/ICBM_Template.nii.gz \
                                -aff ${PathScratch}/${ID}_AffTransf.txt"
    echo "${PathReg}/reg_f3d -ref ${PathScratch}/T1_${ID}.nii.gz -flo ${PathICBM}/ICBM_Template.nii.gz \
                             -aff ${PathScratch}/${ID}_AffTransf.txt -cpp ${PathScratch}/${ID}_cpp.nii.gz"
    echo "cp ${PathScratch}/*cpp* ${PathResults}/."
  } >> "$NameScript"
#snip
bash "$NameScript"

Lines from the third script referenced by $NameScript. These show the three log lines that should appear between the first reg_aladin and the "Extracting volumes from GIF prior." The two lines before reg_aladin appeared but were then truncated along with the reg_aladin output. The three following lines never appeared at all: "Copying GIF priors." "Copying GIF results." "Copying scratch directory to results directory."

#!/bin/bash
echo "Scratch path: /tmp/sub-0_BaMoSCross__"
mkdir -p /tmp/sub-0_BaMoSCross__ 
cp /nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/* /tmp/sub-0_BaMoSCross__/.
echo "Copying T1 image and converting to float."
cp /nfs/project/RISAPS/rawdata/sub-0/ses-v3/anat/sub-0_ses-v3_T1.nii.gz /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz
/usr/local/bin/seg_maths /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz -odt float /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz
echo "Aligning FLAIR to T1."
/usr/local/bin/reg_aladin -ref /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz -flo /nfs/project/RISAPS/rawdata/sub-0/ses-v3/anat/sub-0_ses-v3_FLAIR.nii.gz                               -aff /tmp/sub-0_BaMoSCross__/Aff_FLAIRtoT1.txt -res /tmp/sub-0_BaMoSCross__/FLAIR_sub-0.nii.gz -rigOnly
/usr/local/bin/seg_maths /tmp/sub-0_BaMoSCross__/FLAIR_sub-0.nii.gz -odt float /tmp/sub-0_BaMoSCross__/FLAIR_sub-0.nii.gz
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/sub-0_ses-v3_T1_TIV.nii.gz /tmp/sub-0_BaMoSCross__/Mask_sub-0.nii.gz
/usr/local/bin/seg_maths /tmp/sub-0_BaMoSCross__/Mask_sub-0.nii.gz -bin -odt char /tmp/sub-0_BaMoSCross__/GIF_B1_sub-0.nii.gz
echo "Copying GIF priors."
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/*prior* /tmp/sub-0_BaMoSCross__/GIF_prior_sub-0.nii.gz
echo "Copying GIF results."
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/*Segmentation* /tmp/sub-0_BaMoSCross__/GIF_Segmentation_sub-0.nii.gz
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/*prior* /tmp/sub-0_BaMoSCross__/GIF_prior_sub-0.nii.gz
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/*Parcellation* /tmp/sub-0_BaMoSCross__/GIF_Parcellation_sub-0.nii.gz
echo "Copying scratch directory to results directory."
cp /tmp/sub-0_BaMoSCross__/* /nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/.
echo "Extracting volumes from GIF prior."
#snip (several seg_maths lines)
echo "Creating artefact mask based on GIF parcellation."
#snip (even more seg_maths_lines)
echo "Aligning ICBM template to T1."
/usr/local/bin/reg_aladin -ref /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz -flo /nfs/project/RISAPS/sourcedata/ICBM_priors/ICBM_Template.nii.gz                                 -aff /tmp/sub-0_BaMoSCross__/sub-0_AffTransf.txt
/usr/local/bin/reg_f3d -ref /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz -flo /nfs/project/RISAPS/sourcedata/ICBM_priors/ICBM_Template.nii.gz                              -aff /tmp/sub-0_BaMoSCross__/sub-0_AffTransf.txt -cpp /tmp/sub-0_BaMoSCross__/sub-0_cpp.nii.gz

[mmodat]

Hi Paul,

Sorry, this has been left for a long time. Do you still have an issue? I can't think of a reason why reg_aladin would yield truncated line.

[pwrightkcl]

I haven't examined it in detail since I did the testing above. I'd still prefer if this didn't happen, but like you I have no idea why it does, or even whether it is reg_aladin that causes it.