software-layer icon indicating copy to clipboard operation
software-layer copied to clipboard

fix ReFrame issues on NVIDIA Grace

Open trz42 opened this issue 8 months ago • 24 comments

Let's see if ~CI for eessi_container.sh is fixed by installing another package in install_apptainer_ubuntu.sh, and also if~ ReFrame tests succeed.

We should also add some package to be built to test if building still works after running containers with the additional argument --contain.

trz42 avatar Mar 28 '25 21:03 trz42

Instance eessi-bot-mc-aws is configured to build for:

  • architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/sapphirerapids, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
  • repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

eessi-bot[bot] avatar Mar 28 '25 21:03 eessi-bot[bot]

Instance eessi-bot-mc-azure is configured to build for:

  • architectures: x86_64/amd/zen4
  • repositories: eessi.io-2023.06-compat, eessi.io-2023.06-software

eessi-bot[bot] avatar Mar 28 '25 21:03 eessi-bot[bot]

Instance trz42-GH200-jr is configured to build for:

  • architectures: aarch64/nvidia/grace
  • repositories: eessi.io-2023.06-software

eessi-bot-trz42[bot] avatar Mar 28 '25 21:03 eessi-bot-trz42[bot]

Instance rt-Grace-jr is configured to build for:

  • architectures: aarch64/nvidia/grace
  • repositories: eessi.io-2023.06-software

bot: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace

trz42 avatar Mar 28 '25 22:03 trz42

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace from trz42

    • expanded format: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace
  • handling command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace resulted in:

    • no jobs were submitted

eessi-bot[bot] avatar Mar 28 '25 22:03 eessi-bot[bot]

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace from trz42

    • expanded format: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace
  • handling command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace resulted in:

    • no jobs were submitted

eessi-bot[bot] avatar Mar 28 '25 22:03 eessi-bot[bot]

Updates by the bot instance rt-Grace-jr (click for details)
  • account trz42 has NO permission to send commands to the bot

Updates by the bot instance trz42-GH200-jr (click for details)
  • received bot command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace from trz42

    • expanded format: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace
  • handling command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace resulted in:

    • submitted job 13545111, for details & status see https://github.com/EESSI/software-layer/pull/988#issuecomment-2762692932

eessi-bot-trz42[bot] avatar Mar 28 '25 22:03 eessi-bot-trz42[bot]

New job on instance trz42-GH200-jr for CPU micro-architecture aarch64-nvidia-grace for repository eessi.io-2023.06-software in job dir /p/project1/ceasybuilders/bot-trz42/jobs/2025.03/pr_988/13545111

date job status comment
Mar 28 22:10:42 UTC 2025 submitted job id 13545111 awaits release by job manager
Mar 28 22:10:50 UTC 2025 released job awaits launch by Slurm scheduler
Mar 28 22:11:53 UTC 2025 running job 13545111 is running
Mar 28 22:42:45 UTC 2025 finished
:cry: FAILURE (click triangle for details)
Details
:white_check_mark: job output file slurm-13545111.out
:x: found message matching FATAL:
:white_check_mark: no message matching ERROR:
:white_check_mark: no message matching FAILED:
:white_check_mark: no message matching required modules missing:
:x: no message matching No missing installations
:white_check_mark: found message matching .tar.gz created!
Artefacts
No artefacts were created or found.
Mar 28 22:42:45 UTC 2025 test result
:shrug: UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job13545111.test does not exist in job directory, or parsing it failed.

eessi-bot-trz42[bot] avatar Mar 28 '25 22:03 eessi-bot-trz42[bot]

Rebuild... bot: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace

trz42 avatar Mar 28 '25 22:03 trz42

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace from trz42

    • expanded format: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace
  • handling command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace resulted in:

    • no jobs were submitted

eessi-bot[bot] avatar Mar 28 '25 22:03 eessi-bot[bot]

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace from trz42

    • expanded format: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace
  • handling command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace resulted in:

    • no jobs were submitted

eessi-bot[bot] avatar Mar 28 '25 22:03 eessi-bot[bot]

Updates by the bot instance rt-Grace-jr (click for details)
  • account trz42 has NO permission to send commands to the bot

Updates by the bot instance trz42-GH200-jr (click for details)
  • received bot command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace from trz42

    • expanded format: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace
  • handling command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace resulted in:

    • submitted job 13545157, for details & status see https://github.com/EESSI/software-layer/pull/988#issuecomment-2762754015

eessi-bot-trz42[bot] avatar Mar 28 '25 22:03 eessi-bot-trz42[bot]

New job on instance trz42-GH200-jr for CPU micro-architecture aarch64-nvidia-grace for repository eessi.io-2023.06-software in job dir /p/project1/ceasybuilders/bot-trz42/jobs/2025.03/pr_988/13545157

date job status comment
Mar 28 22:42:45 UTC 2025 submitted job id 13545157 awaits release by job manager
Mar 28 22:43:49 UTC 2025 released job awaits launch by Slurm scheduler
Mar 28 22:44:52 UTC 2025 running job 13545157 is running
Mar 28 23:14:38 UTC 2025 finished
:grin: SUCCESS (click triangle for details)
Details
:white_check_mark: job output file slurm-13545157.out
:white_check_mark: no message matching FATAL:
:white_check_mark: no message matching ERROR:
:white_check_mark: no message matching FAILED:
:white_check_mark: no message matching required modules missing:
:white_check_mark: found message(s) matching No missing installations
:white_check_mark: found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-aarch64-nvidia-grace-1743202985.tar.gzsize: 0 MiB (426755 bytes)
entries: 52
modules under 2023.06/software/linux/aarch64/nvidia/grace/modules/all
BWA/0.7.18-GCCcore-13.2.0.lua
software under 2023.06/software/linux/aarch64/nvidia/grace/software
BWA/0.7.18-GCCcore-13.2.0
other under 2023.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Mar 28 23:14:38 UTC 2025 test result
:shrug: UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job13545157.test does not exist in job directory, or parsing it failed.

eessi-bot-trz42[bot] avatar Mar 28 '25 22:03 eessi-bot-trz42[bot]

Rebuild after adding /dev to extra bind paths... bot: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace

trz42 avatar Mar 28 '25 23:03 trz42

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace from trz42

    • expanded format: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace
  • handling command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace resulted in:

    • no jobs were submitted

eessi-bot[bot] avatar Mar 28 '25 23:03 eessi-bot[bot]

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace from trz42

    • expanded format: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace
  • handling command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace resulted in:

    • no jobs were submitted

eessi-bot[bot] avatar Mar 28 '25 23:03 eessi-bot[bot]

Updates by the bot instance rt-Grace-jr (click for details)
  • account trz42 has NO permission to send commands to the bot

Updates by the bot instance trz42-GH200-jr (click for details)
  • received bot command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace from trz42

    • expanded format: build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace
  • handling command build instance:trz42-GH200-jr repository:eessi.io-2023.06-software architecture:aarch64/nvidia/grace resulted in:

    • submitted job 13545169, for details & status see https://github.com/EESSI/software-layer/pull/988#issuecomment-2762816419

eessi-bot-trz42[bot] avatar Mar 28 '25 23:03 eessi-bot-trz42[bot]

New job on instance trz42-GH200-jr for CPU micro-architecture aarch64-nvidia-grace for repository eessi.io-2023.06-software in job dir /p/project1/ceasybuilders/bot-trz42/jobs/2025.03/pr_988/13545169

date job status comment
Mar 28 23:16:59 UTC 2025 submitted job id 13545169 awaits release by job manager
Mar 28 23:17:42 UTC 2025 released job awaits launch by Slurm scheduler
Mar 28 23:18:46 UTC 2025 running job 13545169 is running
Mar 28 23:56:45 UTC 2025 finished
:grin: SUCCESS (click triangle for details)
Details
:white_check_mark: job output file slurm-13545169.out
:white_check_mark: no message matching FATAL:
:white_check_mark: no message matching ERROR:
:white_check_mark: no message matching FAILED:
:white_check_mark: no message matching required modules missing:
:white_check_mark: found message(s) matching No missing installations
:white_check_mark: found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-aarch64-nvidia-grace-1743204932.tar.gzsize: 0 MiB (428313 bytes)
entries: 52
modules under 2023.06/software/linux/aarch64/nvidia/grace/modules/all
BWA/0.7.18-GCCcore-13.2.0.lua
software under 2023.06/software/linux/aarch64/nvidia/grace/software
BWA/0.7.18-GCCcore-13.2.0
other under 2023.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Mar 28 23:56:45 UTC 2025 test result
:shrug: UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job13545169.test does not exist in job directory, or parsing it failed.

eessi-bot-trz42[bot] avatar Mar 28 '25 23:03 eessi-bot-trz42[bot]

@trz42 Can you retarget this pr?

laraPPr avatar Jun 27 '25 14:06 laraPPr

@trz42 Can you retarget this pr?

laraPPr avatar Jun 27 '25 15:06 laraPPr