Test Report : results

Test Suite: run

Results

Duration2.06 sec
Tests4
Failures0

Tests

run

Test case:[dev-tools/cmake] running cmake --system-information
Outcome:Passed
Duration:0.86 sec
Test case:[dev-tools/cmake] run cmake
Outcome:Passed
Duration:0.422 sec
Test case:[dev-tools/cmake] run make on generated Makefile
Outcome:Passed
Duration:0.768 sec
Test case:[dev-tools/cmake] run compiled binary
Outcome:Passed
Duration:0.01 sec

Test Suite: computes_installed.bats

Results

Duration2.833 sec
Tests5
Failures0

Tests

computes_installed.bats

Test case:num_computes is greater than zero
Outcome:Passed
Duration:0.02 sec
Test case:koomie_cf command is available
Outcome:Passed
Duration:0.019 sec
Test case:get uptime from compute nodes produces results
Outcome:Passed
Duration:1.096 sec
Test case:correct number of compute hosts respond
Outcome:Passed
Duration:0.592 sec
Test case:verify boot times are reasonable
Outcome:Passed
Duration:1.106 sec

Test Suite: sms_installed.bats

Results

Duration0.029 sec
Tests2
Failures0

Tests

sms_installed.bats

Test case:Verify hostname matches expectations
Outcome:Passed
Duration:0.012 sec
Test case:Base OS check
Outcome:Passed
Duration:0.017 sec

Test Suite: clustershell

Results

Duration0.871 sec
Tests2
Failures0

Tests

clustershell

Test case:[clush] check for OS-provided RPM
Outcome:Passed
Duration:0.044 sec
Test case:[clush] clush -Sg compute
Outcome:Passed
Duration:0.827 sec

Test Suite: conman

Results

Duration0.063 sec
Tests2
Failures0

Tests

conman

Test case:[conman] check for RPM
Outcome:Passed
Duration:0.043 sec
Test case:[conman] query conmand conman
Outcome:Passed
Duration:0.02 sec

Test Suite: genders

Results

Duration0.059 sec
Tests2
Failures0

Tests

genders

Test case:[genders] check for RPM
Outcome:Passed
Duration:0.043 sec
Test case:[genders] check node attributes
Outcome:Passed
Duration:0.016 sec

Test Suite: nhc

Results

Duration5.373 sec
Tests3
Failures0

Tests

nhc

Test case:[nhc] check for RPM
Outcome:Passed
Duration:0.044 sec
Test case:[nhc] generate config file
Outcome:Passed
Duration:2.602 sec
Test case:[nhc] service failure detection and restart
Outcome:Passed
Duration:2.727 sec

Test Suite: slurm-plugins

Results

Duration1.287 sec
Tests4
Failures0

Tests

slurm-plugins

Test case:[slurm] check for jobcomp_elasticsearch plugin
Outcome:Passed
Duration:0.018 sec
Test case:[slurm] check for job_submit_lua plugin
Outcome:Passed
Duration:0.016 sec
Test case:[slurm] check for --x11 option
Outcome:Passed
Duration:0.024 sec
Test case:[slurm] check for sview rpm availability
Outcome:Passed
Duration:1.229 sec

Test Suite: lmod

Results

Duration0.394 sec
Tests1
Failures0

Tests

lmod

Test case:[lmod] test that the setup function passed
Outcome:Passed
Duration:0.394 sec

Test Suite: spack

Results

Duration75.281 sec
Tests4
Failures0

Tests

spack

Test case:[spack] check for RPM
Outcome:Passed
Duration:0.537 sec
Test case:[spack] add compiler
Outcome:Passed
Duration:48.092 sec
Test case:[spack] test build
Outcome:Passed
Duration:25.521 sec
Test case:[spack] test module refresh
Outcome:Passed
Duration:1.131 sec

Test Suite: build

Results

Duration6.537 sec
Tests1
Failures0

Tests

build

Test case:[/apps/hpcg] build HPCG executable (gnu15/mpich)
Outcome:Passed
Duration:6.537 sec

Test Suite: rm_execution_multi_host

Results

Duration16.987 sec
Tests3
Failures0

Tests

rm_execution_multi_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.009 sec
Test case:[/apps/hpcg] run HPCG on multi nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:16.928 sec
Test case:[/apps/hpcg] log HPCG multi node results (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.05 sec

Test Suite: rm_execution_single_host

Results

Duration12.78 sec
Tests3
Failures0

Tests

rm_execution_single_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.009 sec
Test case:[/apps/hpcg] run HPCG on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:12.721 sec
Test case:[/apps/hpcg] log HPCG single node results (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.05 sec

Test Suite: build

Results

Duration4.218 sec
Tests1
Failures0

Tests

build

Test case:[/apps/hpcg] build HPCG executable (gnu15/openmpi5)
Outcome:Passed
Duration:4.218 sec

Test Suite: rm_execution_multi_host

Results

Duration1.227 sec
Tests3
Failures2

Tests

rm_execution_multi_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.01 sec
Test case:[/apps/hpcg] run HPCG on multi nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:1.192 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution_multi_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.20247
Batch job 14 submitted

Job 14 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./bin/xhpcg.gnu15.openmpi5 32 32 32 10 (family=openmpi5)
[c1:04570] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04575] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04571] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04568] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04572] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04574] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04569] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04573] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04570] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04571] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04568] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04572] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04573] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04569] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04574] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04575] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[c2:03167] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03161] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03168] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03166] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03163] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03164] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03162] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03165] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03167] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03168] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03161] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03164] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03163] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03165] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03166] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03162] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-7: Exited with exit code 14
srun: error: c2: tasks 8-15: Exited with exit code 14
Test case:[/apps/hpcg] log HPCG multi node results (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:0.025 sec
Failed
(in test file rm_execution_multi_host, line 55)
  `mv $run_yaml $wrk_yaml' failed
Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg
ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory
Moving  to HPCG.32x32x32.P16.gnu15.openmpi5.yaml
mv: missing destination file operand after 'HPCG.32x32x32.P16.gnu15.openmpi5.yaml'
Try 'mv --help' for more information.

Test Suite: rm_execution_single_host

Results

Duration2.281 sec
Tests3
Failures2

Tests

rm_execution_single_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.01 sec
Test case:[/apps/hpcg] run HPCG on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.246 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution_single_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT ${EXE} "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.14726
Batch job 13 submitted

Job 13 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./bin/xhpcg.gnu15.openmpi5 32 32 32 10 (family=openmpi5)
[c1:04381] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04376] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04382] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04379] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:04383] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04376] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04381] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04379] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04383] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:04382] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0,3,5-7: Exited with exit code 14
[2026-03-10T17:23:22.506] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.13.0:0]
[2026-03-10T17:23:22.513] error: *** STEP 13.0 ON c1 CANCELLED AT 2026-03-10T17:23:22 DUE to SIGNAL Killed ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: c1: tasks 1-2,4: Killed
Test case:[/apps/hpcg] log HPCG single node results (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:0.025 sec
Failed
(in test file rm_execution_single_host, line 55)
  `mv $run_yaml $wrk_yaml' failed
Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg
ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory
Moving  to HPCG.32x32x32.P8.gnu15.openmpi5.yaml
mv: missing destination file operand after 'HPCG.32x32x32.P8.gnu15.openmpi5.yaml'
Try 'mv --help' for more information.

Test Suite: build

Results

Duration11.564 sec
Tests1
Failures0

Tests

build

Test case:[Apps/miniFE] build MiniFE executable (gnu15/mpich)
Outcome:Passed
Duration:11.564 sec

Test Suite: rm_execution_multi_host

Results

Duration14.87 sec
Tests2
Failures0

Tests

rm_execution_multi_host

Test case:[Apps/miniFE] run miniFE on multi nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:14.826 sec
Test case:[Apps/miniFE] log miniFE multi node results (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.044 sec

Test Suite: rm_execution_single_host

Results

Duration3.315 sec
Tests2
Failures0

Tests

rm_execution_single_host

Test case:[Apps/miniFE] run miniFE on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.27 sec
Test case:[Apps/miniFE] log miniFE single node results (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.045 sec

Test Suite: build

Results

Duration6.516 sec
Tests1
Failures0

Tests

build

Test case:[Apps/miniFE] build MiniFE executable (gnu15/openmpi5)
Outcome:Passed
Duration:6.516 sec

Test Suite: rm_execution_multi_host

Results

Duration2.275 sec
Tests2
Failures1

Tests

rm_execution_multi_host

Test case:[Apps/miniFE] run miniFE on multi nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.232 sec
Failed
(from function `run_mpi_binary' in file ./common-test/functions, line 434,
 in test file rm_execution_multi_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.30949
Batch job 18 submitted

Job 18 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./src/miniFE.x.gnu15.openmpi5 nx=256 ny=256 nz=256 verify_solution=0 (family=openmpi5)
[c1:05388] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05389] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05392] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05393] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05394] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05387] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05390] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05391] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03395] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03396] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03397] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03398] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03399] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03400] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03401] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03394] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05388] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05393] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03395] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05390] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03400] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05391] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03394] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05392] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05394] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05389] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05387] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03397] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03401] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03398] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03399] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03396] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-7: Exited with exit code 14
srun: error: c2: tasks 8-15: Exited with exit code 14
Test case:[Apps/miniFE] log miniFE multi node results (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.043 sec

Test Suite: rm_execution_single_host

Results

Duration2.276 sec
Tests2
Failures1

Tests

rm_execution_single_host

Test case:[Apps/miniFE] run miniFE on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.232 sec
Failed
(from function `run_mpi_binary' in file ./common-test/functions, line 434,
 in test file rm_execution_single_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.636
Batch job 17 submitted

Job 17 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./src/miniFE.x.gnu15.openmpi5 nx=100 ny=100 nz=100 verify_solution=1 (family=openmpi5)
[c1:05204] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05203] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05198] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05199] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05201] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05200] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05202] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:05205] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05200] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05204] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05201] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05202] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05205] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05199] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05203] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:05198] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-7: Exited with exit code 14
Test case:[Apps/miniFE] log miniFE single node results (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.044 sec

Test Suite: os_distribution

Results

Duration0.008 sec
Tests1
Failures0

Tests

os_distribution

Test case:[BOS] OS distribution matches (local)
Outcome:Passed
Duration:0.008 sec

Test Suite: computes

Results

Duration4.366 sec
Tests4
Failures0

Tests

computes

Test case:[BOS] OS distribution matches (2 active computes)
Outcome:Skipped
Duration:0.0 sec
Skipped
disable BOS_RELEASE check
Test case:[BOS] consistent kernel (2 active computes)
Outcome:Passed
Duration:1.096 sec
Test case:[BOS] increased locked memory limits
Outcome:Passed
Duration:2.18 sec
Test case:[BOS] syslog forwarding
Outcome:Passed
Duration:1.09 sec

Test Suite: debugger

Results

Duration0.014 sec
Tests2
Failures0

Tests

debugger

Test case:[Compilers] debugger man page (gnu15)
Outcome:Passed
Duration:0.007 sec
Test case:[Compilers] debugger availability (gnu15)
Outcome:Passed
Duration:0.007 sec

Test Suite: man_pages

Results

Duration0.131 sec
Tests3
Failures0

Tests

man_pages

Test case:[Compilers] C compiler man/help page (gnu15)
Outcome:Passed
Duration:0.048 sec
Test case:[Compilers] C++ compiler man/help page (gnu15)
Outcome:Passed
Duration:0.044 sec
Test case:[Compilers] Fortran compiler man/help page (gnu15)
Outcome:Passed
Duration:0.039 sec

Test Suite: rm_execution

Results

Duration0.755 sec
Tests6
Failures0

Tests

rm_execution

Test case:[Compilers] C binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.114 sec
Test case:[Compilers] C++ binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.119 sec
Test case:[Compilers] Fortran binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.138 sec
Test case:[Compilers] C openmp binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.117 sec
Test case:[Compilers] C++ openmp binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.118 sec
Test case:[Compilers] Fortran openmp binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.149 sec

Test Suite: version_match

Results

Duration0.433 sec
Tests3
Failures0

Tests

version_match

Test case:[Compilers] compiler module loaded (gnu15)
Outcome:Passed
Duration:0.138 sec
Test case:[Compilers] compiler module version available (gnu15)
Outcome:Passed
Duration:0.132 sec
Test case:[Compilers] C, C++, and Fortran versions match module (gnu15)
Outcome:Passed
Duration:0.163 sec

Test Suite: EasyBuild

Results

Duration542.233 sec
Tests3
Failures0

Tests

EasyBuild

Test case:[EasyBuild] check for RPM
Outcome:Passed
Duration:0.206 sec
Test case:[EasyBuild] test executable
Outcome:Passed
Duration:2.193 sec
Test case:[EasyBuild] test install of tmux
Outcome:Passed
Duration:539.834 sec

Test Suite: rm_execution

Results

Duration0.358 sec
Tests2
Failures0

Tests

rm_execution

Test case:[dev-tools/hwloc] lstopo runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.202 sec
Test case:[dev-tools/hwloc] hwloc_hello runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.156 sec

Test Suite: test_module

Results

Duration0.218 sec
Tests7
Failures0

Tests

test_module

Test case:[HWLOC] Verify HWLOC module is loaded and matches rpm version (gnu15/)
Outcome:Passed
Duration:0.164 sec
Test case:[HWLOC] Verify module HWLOC_DIR is defined and exists (gnu15/)
Outcome:Passed
Duration:0.011 sec
Test case:[HWLOC] Verify module HWLOC_LIB is defined and exists (gnu15/)
Outcome:Passed
Duration:0.008 sec
Test case:[HWLOC] Verify dynamic library available in HWLOC_LIB (gnu15/)
Outcome:Passed
Duration:0.009 sec
Test case:[HWLOC] Verify static library is not present in HWLOC_LIB (gnu15/)
Outcome:Passed
Duration:0.008 sec
Test case:[HWLOC] Verify module HWLOC_INC is defined and exists (gnu15/)
Outcome:Passed
Duration:0.009 sec
Test case:[HWLOC] Verify header file is present in HWLOC_INC (gnu15/)
Outcome:Passed
Duration:0.009 sec

Test Suite: hipy

Results

Duration7.482 sec
Tests1
Failures0

Tests

hipy

Test case:[dev-tools/py3-mpi4py] python hello world (slurm/gnu15/mpich)
Outcome:Passed
Duration:7.482 sec

Test Suite: test_module

Results

Duration0.029 sec
Tests3
Failures0

Tests

test_module

Test case:[mpi4py] Verify module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[mpi4py] Verify module MPI4PY_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.011 sec
Test case:[mpi4py] Verify PYTHONPATH is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec

Test Suite: hipy

Results

Duration2.247 sec
Tests1
Failures1

Tests

hipy

Test case:[dev-tools/py3-mpi4py] python hello world (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.247 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file hipy, line 35)
  `run_mpi_binary "${_python} helloworld.py" $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.736
Batch job 28 submitted

Job 28 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- python3 helloworld.py 8 (family=openmpi5)
[c1:06101] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06102] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06100] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06099] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03932] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03929] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03931] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:03930] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06101] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06102] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06099] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06100] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03930] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03932] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03931] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:03929] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14

Test Suite: test_module

Results

Duration0.029 sec
Tests3
Failures0

Tests

test_module

Test case:[mpi4py] Verify module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[mpi4py] Verify module MPI4PY_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.011 sec
Test case:[mpi4py] Verify PYTHONPATH is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec

Test Suite: MM

Results

Duration1.295 sec
Tests1
Failures0

Tests

MM

Test case:[dev-tools/py3-numpy] Numpy Matrix Multiply
Outcome:Passed
Duration:1.295 sec

Test Suite: test_module

Results

Duration0.038 sec
Tests4
Failures0

Tests

test_module

Test case:[Numpy] Verify NUMPY modules can be loaded and match rpm version (gnu15)
Outcome:Passed
Duration:0.007 sec
Test case:[Numpy] Verify module NUMPY_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.011 sec
Test case:[Numpy] Verify module NUMPY_BIN is defined and exists
Outcome:Passed
Duration:0.011 sec
Test case:[Numpy] Verify PYTHONPATH is defined and exists (gnu15)
Outcome:Passed
Duration:0.009 sec

Test Suite: rm_execution

Results

Duration1.501 sec
Tests2
Failures0

Tests

rm_execution

Test case:[Valgrind] Callgrind execution under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.362 sec
Test case:[Valgrind] Memcheck execution under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.139 sec

Test Suite: test_module

Results

Duration1.629 sec
Tests8
Failures0

Tests

test_module

Test case:[Valgrind] Verify valgrind module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.162 sec
Test case:[Valgrind] Verify module VALGRIND_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[Valgrind] Verify availability of valgrind binary (gnu15)
Outcome:Passed
Duration:0.016 sec
Test case:[Valgrind] Verify availability of man page (gnu15)
Outcome:Passed
Duration:0.048 sec
Test case:[Valgrind] Verify module VALGRIND_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[Valgrind] Verify header file is present in VALGRIND_INC (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[Valgrind] Callgrind compile/test (gnu15)
Outcome:Passed
Duration:0.354 sec
Test case:[Valgrind] Memcheck compile/test (gnu15)
Outcome:Passed
Duration:1.023 sec

Test Suite: rm_execution

Results

Duration12.327 sec
Tests1
Failures0

Tests

rm_execution

Test case:[R] Running Rscript bench.R under resource manager (slurm/gnu15)
Outcome:Passed
Duration:12.327 sec

Test Suite: test_module

Results

Duration13.08 sec
Tests7
Failures0

Tests

test_module

Test case:[R] Verify R module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.218 sec
Test case:[R] Verify R_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.01 sec
Test case:[R] Verify availability of R executable -> R (gnu15)
Outcome:Passed
Duration:0.016 sec
Test case:[R] Verify availability of R executable -> Rscript (gnu15)
Outcome:Passed
Duration:0.017 sec
Test case:[R] Verify module R_SHARE is defined and exists (gnu15)
Outcome:Passed
Duration:0.011 sec
Test case:[R] Running bench.R test
Outcome:Passed
Duration:12.305 sec
Test case:[R] Verify ability to compile C code for R and execute
Outcome:Passed
Duration:0.503 sec

Test Suite: rm_execution

Results

Duration4.447 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/adios2] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec
Test case:[libs/adios2] MPI F90 binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.224 sec

Test Suite: test_module

Results

Duration0.331 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/adios2] Verify ADIOS2 module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.205 sec
Test case:[libs/adios2] Verify module ADIOS2_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/adios2] Verify module ADIOS2_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/adios2] Verify module ADIOS2_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/adios2] Verify dynamic library available in ADIOS2_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.025 sec
Test case:[libs/adios2] Verify module ADIOS2_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/adios2] Verify header file is present in ADIOS2_INC (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration4.494 sec
Tests2
Failures2

Tests

rm_execution

Test case:[libs/adios2] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.247 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 28)
  `run_mpi_binary -t "${TIMEOUT}" ./arrays_write "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.8314
Batch job 34 submitted

Job 34 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./arrays_write  (family=openmpi5)
[c2:04287] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:04288] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:04289] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:04290] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06815] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06816] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06818] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06817] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:04287] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:04289] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:04288] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:04290] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06816] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06815] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06817] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06818] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 4-7: Exited with exit code 14
srun: error: c1: tasks 0-3: Exited with exit code 14
Test case:[libs/adios2] MPI F90 binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.247 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 37)
  `run_mpi_binary -t "${TIMEOUT}" ./scalars_write "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.28063
Batch job 35 submitted

Job 35 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./scalars_write  (family=openmpi5)
[c1:06996] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06998] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06999] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:06997] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:04376] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:04379] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:04377] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:04378] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06996] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06997] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06998] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:06999] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:04376] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:04379] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:04378] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:04377] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14

Test Suite: test_module

Results

Duration0.34 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/adios2] Verify ADIOS2 module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.21 sec
Test case:[libs/adios2] Verify module ADIOS2_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/adios2] Verify module ADIOS2_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/adios2] Verify module ADIOS2_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/adios2] Verify dynamic library available in ADIOS2_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.027 sec
Test case:[libs/adios2] Verify module ADIOS2_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/adios2] Verify header file is present in ADIOS2_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration0.528 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Accumulators] min-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.129 sec
Test case:[Boost/Accumulators] max-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.138 sec
Test case:[Boost/Accumulators] skewness-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.145 sec
Test case:[Boost/Accumulators] variance-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.116 sec

Test Suite: rm_execution

Results

Duration0.569 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Accumulators] min-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.118 sec
Test case:[Boost/Accumulators] max-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.156 sec
Test case:[Boost/Accumulators] skewness-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.159 sec
Test case:[Boost/Accumulators] variance-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.136 sec

Test Suite: test_module

Results

Duration2.632 sec
Tests8
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.208 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.011 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Test interactive build/exec of f_test.cpp (gnu15/mpich)
Outcome:Passed
Duration:2.37 sec

Test Suite: test_module

Results

Duration2.665 sec
Tests8
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.245 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Test interactive build/exec of f_test.cpp (gnu15/openmpi5)
Outcome:Passed
Duration:2.366 sec

Test Suite: rm_execution

Results

Duration0.523 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Multi Array] access-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.169 sec
Test case:[Boost/Multi Array] iterators-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.118 sec
Test case:[Boost/Multi Array] resize-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.119 sec
Test case:[Boost/Multi Array] idxgen1-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.117 sec

Test Suite: rm_execution

Results

Duration0.473 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Multi Array] access-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.124 sec
Test case:[Boost/Multi Array] iterators-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.117 sec
Test case:[Boost/Multi Array] resize-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.122 sec
Test case:[Boost/Multi Array] idxgen1-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.11 sec

Test Suite: rm_execution

Results

Duration1.323 sec
Tests1
Failures0

Tests

rm_execution

Test case:[Boost/ublas] bench1_ublas binary under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.323 sec

Test Suite: rm_execution

Results

Duration1.324 sec
Tests1
Failures0

Tests

rm_execution

Test case:[Boost/ublas] bench1_ublas binary under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:1.324 sec

Test Suite: rm_execution

Results

Duration1.172 sec
Tests11
Failures0

Tests

rm_execution

Test case:[Boost/Program Options] cmdline_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.131 sec
Test case:[Boost/Program Options] exception_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.122 sec
Test case:[Boost/Program Options] options_description_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.117 sec
Test case:[Boost/Program Options] parsers_test on master host(gnu15/mpich)
Outcome:Passed
Duration:0.014 sec
Test case:[Boost/Program Options] parsers_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.125 sec
Test case:[Boost/Program Options] positional_options_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.126 sec
Test case:[Boost/Program Options] required_test on master host(gnu15/mpich)
Outcome:Passed
Duration:0.015 sec
Test case:[Boost/Program Options] required_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.151 sec
Test case:[Boost/Program Options] unicode_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.13 sec
Test case:[Boost/Program Options] unrecognized_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.123 sec
Test case:[Boost/Program Options] variable_map_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.118 sec

Test Suite: rm_execution

Results

Duration1.172 sec
Tests11
Failures0

Tests

rm_execution

Test case:[Boost/Program Options] cmdline_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.133 sec
Test case:[Boost/Program Options] exception_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.12 sec
Test case:[Boost/Program Options] options_description_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.122 sec
Test case:[Boost/Program Options] parsers_test on master host(gnu15/openmpi5)
Outcome:Passed
Duration:0.016 sec
Test case:[Boost/Program Options] parsers_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.148 sec
Test case:[Boost/Program Options] positional_options_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.123 sec
Test case:[Boost/Program Options] required_test on master host(gnu15/openmpi5)
Outcome:Passed
Duration:0.013 sec
Test case:[Boost/Program Options] required_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.12 sec
Test case:[Boost/Program Options] unicode_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.122 sec
Test case:[Boost/Program Options] unrecognized_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.124 sec
Test case:[Boost/Program Options] variable_map_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.131 sec

Test Suite: rm_execution

Results

Duration1.814 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Random] test_piecewise_linear under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.135 sec
Test case:[Boost/Random] test_discrete under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.368 sec
Test case:[Boost/Random] test_random_device under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.185 sec
Test case:[Boost/Random] test_random_number_generator under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.126 sec

Test Suite: rm_execution

Results

Duration1.747 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Random] test_piecewise_linear under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:1.1 sec
Test case:[Boost/Random] test_discrete under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.387 sec
Test case:[Boost/Random] test_random_device under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.135 sec
Test case:[Boost/Random] test_random_number_generator under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.125 sec

Test Suite: rm_execution

Results

Duration0.149 sec
Tests1
Failures0

Tests

rm_execution

Test case:[Boost] regress under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.149 sec

Test Suite: rm_execution

Results

Duration0.771 sec
Tests3
Failures0

Tests

rm_execution

Test case:[Boost] bad_expression_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.444 sec
Test case:[Boost] named_subexpressions_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.134 sec
Test case:[Boost] recursion_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.193 sec

Test Suite: rm_execution

Results

Duration0.775 sec
Tests3
Failures0

Tests

rm_execution

Test case:[Boost] bad_expression_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.467 sec
Test case:[Boost] named_subexpressions_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.128 sec
Test case:[Boost] recursion_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.18 sec

Test Suite: test_module

Results

Duration0.261 sec
Tests7
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.205 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.012 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec

Test Suite: test_module

Results

Duration0.272 sec
Tests7
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.215 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.011 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec

Test Suite: rm_execution

Results

Duration57.561 sec
Tests7
Failures0

Tests

rm_execution

Test case:[Boost/MPI] all_gather_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:6.42 sec
Test case:[Boost/MPI] all_reduce_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:7.472 sec
Test case:[Boost/MPI] all_to_all_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:7.469 sec
Test case:[Boost/MPI] groups_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:8.52 sec
Test case:[Boost/MPI] broadcast_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:8.517 sec
Test case:[Boost/MPI] ring_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:9.588 sec
Test case:[Boost/MPI] pointer_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:9.575 sec

Test Suite: rm_execution

Results

Duration51.548 sec
Tests7
Failures6

Tests

rm_execution

Test case:[Boost/MPI] all_gather_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.295 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 24)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.608
Batch job 95 submitted

Job 95 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./all_gather_test atest (family=openmpi5)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
[c1:11594] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11595] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11589] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11591] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11593] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11596] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
[c2:06485] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06486] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06481] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06484] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06480] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06483] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06479] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06482] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11589] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11594] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11596] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11591] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11593] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11595] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06485] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06479] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06480] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06481] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06484] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06486] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06483] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06482] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0,2,4-7: Exited with exit code 14
srun: error: c2: tasks 8-15: Exited with exit code 14
[2026-03-10T17:37:11.530] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.95.0:0]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:37:11.539] error: *** STEP 95.0 ON c1 CANCELLED AT 2026-03-10T17:37:11 DUE to SIGNAL Killed ***
srun: error: c1: tasks 1,3: Killed
Test case:[Boost/MPI] all_reduce_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.34 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 33)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.13406
Batch job 96 submitted

Job 96 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./all_reduce_test atest (family=openmpi5)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
[c1:11787] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11790] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11784] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11788] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11791] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11785] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
[c2:06583] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06584] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06582] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06588] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06586] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06587] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06585] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06589] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11791] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11787] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11785] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11788] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11790] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11784] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06583] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06588] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06582] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06587] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06586] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06584] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06589] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06585] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1,3-4,6-7: Exited with exit code 14
srun: error: c2: tasks 8-15: Exited with exit code 14
[2026-03-10T17:37:13.281] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.96.0:3]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:37:13.288] error: *** STEP 96.0 ON c1 CANCELLED AT 2026-03-10T17:37:13 DUE to SIGNAL Killed ***
srun: error: c1: tasks 2,5: Killed
Test case:[Boost/MPI] all_to_all_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:5.395 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 42)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.27740
Batch job 97 submitted

Job 97 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./all_to_all_test atest (family=openmpi5)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
[c1:11981] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11983] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11978] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11984] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11982] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11980] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11979] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:11977] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06683] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06681] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06685] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06687] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06688] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06684] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06682] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06686] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11983] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06683] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06681] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06688] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11978] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11984] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11979] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06687] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11980] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11982] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11981] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:11977] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06686] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06684] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06685] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06682] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-7: Exited with exit code 14
srun: error: c2: tasks 8-15: Exited with exit code 14
Test case:[Boost/MPI] groups_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:6.442 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 51)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.15508
Batch job 98 submitted

Job 98 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./groups_test atest (family=openmpi5)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
[c1:12171] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12173] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12167] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12169] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12170] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12168] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12172] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06782] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06778] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06779] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06781] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06783] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06784] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06785] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06780] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12171] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06782] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12167] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12168] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06778] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12173] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06779] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12170] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12169] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12172] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06785] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06780] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06781] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06784] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06783] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 1-7: Exited with exit code 14
srun: error: c2: tasks 8-15: Exited with exit code 14
[2026-03-10T17:37:15.765] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.98.0:5]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:37:15.773] error: *** STEP 98.0 ON c1 CANCELLED AT 2026-03-10T17:37:15 DUE to SIGNAL Killed ***
srun: error: c1: task 0: Killed
Test case:[Boost/MPI] broadcast_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:7.495 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 60)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.7604
Batch job 99 submitted

Job 99 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./broadcast_test atest (family=openmpi5)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
[c2:06882] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06881] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06877] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06884] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06878] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06879] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06883] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:06880] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06881] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
[c1:12363] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06882] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[c1:12366] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12362] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12367] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12360] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12364] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12365] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12361] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06883] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06877] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06884] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06880] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06878] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:06879] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12363] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12367] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12361] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12364] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12365] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12362] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12366] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12360] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 8-15: Exited with exit code 14
srun: error: c1: tasks 0-7: Exited with exit code 14
Test case:[Boost/MPI] ring_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:16.033 sec
Test case:[Boost/MPI] pointer_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:8.548 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 79)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.1206
Batch job 101 submitted

Job 101 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pointer_test atest (family=openmpi5)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
[c2:07077] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12740] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:07071] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:07072] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:07076] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12741] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:07074] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:07075] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12742] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12743] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12744] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:07073] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:07078] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12739] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12738] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:12745] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12740] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07077] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12738] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07071] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12742] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12744] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12739] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07074] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12745] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07072] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07075] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12741] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07073] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07076] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07078] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Running 1 test case...
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:12743] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-7: Exited with exit code 14
srun: error: c2: tasks 8-15: Exited with exit code 14

Test Suite: test_module

Results

Duration0.228 sec
Tests7
Failures0

Tests

test_module

Test case:[FFTW] Verify FFTW module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.168 sec
Test case:[FFTW] Verify module FFTW_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.012 sec
Test case:[FFTW] Verify module FFTW_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify dynamic library available in FFTW_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[FFTW] Verify static library is not present in FFTW_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[FFTW] Verify module FFTW_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify header file is present in FFTW_INC (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec

Test Suite: test_module

Results

Duration0.252 sec
Tests7
Failures0

Tests

test_module

Test case:[FFTW] Verify FFTW module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.192 sec
Test case:[FFTW] Verify module FFTW_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.012 sec
Test case:[FFTW] Verify module FFTW_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify dynamic library available in FFTW_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[FFTW] Verify static library is not present in FFTW_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify module FFTW_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify header file is present in FFTW_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec

Test Suite: rm_execution

Results

Duration4.479 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/FFTW] Serial C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.143 sec
Test case:[libs/FFTW] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.21 sec
Test case:[libs/FFTW] Serial Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.126 sec

Test Suite: rm_execution

Results

Duration4.311 sec
Tests3
Failures1

Tests

rm_execution

Test case:[libs/FFTW] Serial C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.182 sec
Test case:[libs/FFTW] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.232 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 41)
  `run_mpi_binary -t "${TIMEOUT}" ./C_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.30914
Batch job 108 submitted

Job 108 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./C_mpi_test 8 (family=openmpi5)
[c1:13745] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:13742] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:13744] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:13743] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:13745] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[c2:07468] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:13744] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:13742] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[c2:07466] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:07465] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:07467] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:13743] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07467] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07468] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:07465] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-3: Exited with exit code 14
srun: error: c2: tasks 4,6-7: Exited with exit code 14
[2026-03-10T17:38:30.108] error:  mpi/pmix_v4: _errhandler: c2 [1]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.108.0:6]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: c2: task 5: Killed
Test case:[libs/FFTW] Serial Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:1.897 sec

Test Suite: rm_execution

Results

Duration32.461 sec
Tests50
Failures0

Tests

rm_execution

Test case:[libs/GSL] run test_gsl_histogram (gnu15)
Outcome:Passed
Duration:0.028 sec
Test case:[libs/GSL] run block under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.195 sec
Test case:[libs/GSL] run bspline under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.488 sec
Test case:[libs/GSL] run cblas under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.162 sec
Test case:[libs/GSL] run cdf under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.124 sec
Test case:[libs/GSL] run cheb under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.117 sec
Test case:[libs/GSL] run combination under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.162 sec
Test case:[libs/GSL] run complex under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.153 sec
Test case:[libs/GSL] run const under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.13 sec
Test case:[libs/GSL] run deriv under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.121 sec
Test case:[libs/GSL] run dht under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.171 sec
Test case:[libs/GSL] run diff under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.158 sec
Test case:[libs/GSL] run eigen under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.296 sec
Test case:[libs/GSL] run err under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.114 sec
Test case:[libs/GSL] run fft under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.441 sec
Test case:[libs/GSL] run fit under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.115 sec
Test case:[libs/GSL] run histogram under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.167 sec
Test case:[libs/GSL] run ieee-utils under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.113 sec
Test case:[libs/GSL] run integration under resource manager (slurm/gnu15)
Outcome:Passed
Duration:2.85 sec
Test case:[libs/GSL] run interpolation under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.116 sec
Test case:[libs/GSL] run linalg under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.201 sec
Test case:[libs/GSL] run matrix under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.661 sec
Test case:[libs/GSL] run min under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.118 sec
Test case:[libs/GSL] run monte under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.396 sec
Test case:[libs/GSL] run multifit under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.763 sec
Test case:[libs/GSL] run multilarge under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.145 sec
Test case:[libs/GSL] run multimin under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.163 sec
Test case:[libs/GSL] run multiroots under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.123 sec
Test case:[libs/GSL] run multiset under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.128 sec
Test case:[libs/GSL] run ntuple under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.117 sec
Test case:[libs/GSL] run ode-initval under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.407 sec
Test case:[libs/GSL] run ode-initval2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:5.032 sec
Test case:[libs/GSL] run permutation under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.121 sec
Test case:[libs/GSL] run poly under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.113 sec
Test case:[libs/GSL] run qrng under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.116 sec
Test case:[libs/GSL] run randist under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.916 sec
Test case:[libs/GSL] run rng under resource manager (slurm/gnu15)
Outcome:Passed
Duration:2.138 sec
Test case:[libs/GSL] run roots under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.111 sec
Test case:[libs/GSL] run rstat under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.397 sec
Test case:[libs/GSL] run siman under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.188 sec
Test case:[libs/GSL] run sort under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.202 sec
Test case:[libs/GSL] run spblas under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.312 sec
Test case:[libs/GSL] run specfunc under resource manager (slurm/gnu15)
Outcome:Passed
Duration:2.26 sec
Test case:[libs/GSL] run splinalg under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.315 sec
Test case:[libs/GSL] run spmatrix under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.137 sec
Test case:[libs/GSL] run statistics under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.151 sec
Test case:[libs/GSL] run sum under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.12 sec
Test case:[libs/GSL] run sys under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.121 sec
Test case:[libs/GSL] run vector under resource manager (slurm/gnu15)
Outcome:Passed
Duration:4.536 sec
Test case:[libs/GSL] run wavelet under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.732 sec

Test Suite: test_module

Results

Duration0.218 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/GSL] Verify GSL module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.162 sec
Test case:[libs/GSL] Verify module GSL_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.011 sec
Test case:[libs/GSL] Verify module GSL_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/GSL] Verify dynamic library available in GSL_LIB (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/GSL] Verify static library is not present in GSL_LIB (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/GSL] Verify module GSL_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/GSL] Verify header file is present in GSL_INC (gnu15)
Outcome:Passed
Duration:0.009 sec

Test Suite: rm_execution

Results

Duration27.339 sec
Tests12
Failures0

Tests

rm_execution

Test case:[libs/HYPRE] 2 PE structured test binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.218 sec
Test case:[libs/HYPRE] 2 PE PCG with SMG preconditioner binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.22 sec
Test case:[libs/HYPRE] 2 PE Semi-Structured PCG binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.325 sec
Test case:[libs/HYPRE] 2 PE Three-part stencil binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.222 sec
Test case:[libs/HYPRE] 2 PE FORTRAN PCG with PFMG preconditioner binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.274 sec
Test case:[libs/HYPRE] 2 PE -Delta u = 1 binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.271 sec
Test case:[libs/HYPRE] 2 PE convection-reaction-diffusion binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.218 sec
Test case:[libs/HYPRE] 2 PE FORTRAN 2-D Laplacian binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.219 sec
Test case:[libs/HYPRE] 2 PE Semi-Structured convection binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE biharmonic binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE C++ Finite Element Interface binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
C++ example depends on non-installed header
Test case:[libs/HYPRE] 2 PE 2-D Laplacian eigenvalue binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:5.372 sec

Test Suite: test_module

Results

Duration2.466 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/HYPRE] Verify HYPRE module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.189 sec
Test case:[libs/HYPRE] Verify module HYPRE_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/HYPRE] Verify module HYPRE_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/HYPRE] Verify module HYPRE_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/HYPRE] Verify dynamic library available in HYPRE_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/HYPRE] Verify static library is not present in HYPRE_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/HYPRE] Verify module HYPRE_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/HYPRE] Verify header file is present in HYPRE_INC (gnu15/mpich)
Outcome:Passed
Duration:0.007 sec
Test case:[libs/HYPRE] Sample job (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.217 sec

Test Suite: rm_execution

Results

Duration23.336 sec
Tests12
Failures9

Tests

rm_execution

Test case:[libs/HYPRE] 2 PE structured test binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:1.19 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 23)
  `run_mpi_binary ./ex1 $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.16511
Batch job 172 submitted

Job 172 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex1 8 (family=openmpi5)
[c1:19749] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:19750] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:19750] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:19749] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[c2:08595] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:08596] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08596] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08595] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/HYPRE] 2 PE PCG with SMG preconditioner binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.297 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 32)
  `run_mpi_binary ./ex2 $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.10672
Batch job 173 submitted

Job 173 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex2 8 (family=openmpi5)
[c1:19927] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:19926] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:08680] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:08681] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:19926] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:19927] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08680] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08681] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/HYPRE] 2 PE Semi-Structured PCG binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.243 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 41)
  `run_mpi_binary ./ex6 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.7928
Batch job 174 submitted

Job 174 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex6  (family=openmpi5)
[c1:20103] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:20104] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:08766] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:08765] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20103] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20104] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08766] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08765] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/HYPRE] 2 PE Three-part stencil binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.243 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 50)
  `run_mpi_binary ./ex8 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.13092
Batch job 175 submitted

Job 175 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex8  (family=openmpi5)
[c1:20281] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:20280] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:08850] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:08851] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20280] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20281] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08850] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08851] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/HYPRE] 2 PE FORTRAN PCG with PFMG preconditioner binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.293 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 59)
  `run_mpi_binary ./ex12f "" 1 2' failed
job script = /tmp/job.ohpc-test.622
Batch job 176 submitted

Job 176 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex12f  (family=openmpi5)
[c1:20461] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:20460] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20460] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20461] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/HYPRE] 2 PE -Delta u = 1 binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.295 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 68)
  `run_mpi_binary ./ex3 "-n 33 -solver 0 -v 1 1" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.19986
Batch job 177 submitted

Job 177 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex3 -n 33 -solver 0 -v 1 1 (family=openmpi5)
[c2:08936] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:08937] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:20637] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:20638] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08936] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:08937] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20637] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20638] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/HYPRE] 2 PE convection-reaction-diffusion binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.292 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 77)
  `run_mpi_binary ./ex4 "-n 33 -solver 10  -K 3 -B 0 -C 1 -U0 2 -F 4" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.26833
Batch job 178 submitted

Job 178 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex4 -n 33 -solver 10 -K 3 -B 0 -C 1 -U0 2 -F 4 (family=openmpi5)
[c2:09021] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:09022] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:20814] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:20815] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09021] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09022] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20815] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20814] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/HYPRE] 2 PE FORTRAN 2-D Laplacian binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.24 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 86)
  `run_mpi_binary ./ex5f "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.1261
Batch job 179 submitted

Job 179 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex5f  (family=openmpi5)
[c1:20991] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:20992] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:09107] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:09106] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20991] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:20992] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09106] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09107] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/HYPRE] 2 PE Semi-Structured convection binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE biharmonic binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE C++ Finite Element Interface binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
C++ example depends on non-installed header
Test case:[libs/HYPRE] 2 PE 2-D Laplacian eigenvalue binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.243 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 125)
  `run_mpi_binary ./ex11 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.29784
Batch job 180 submitted

Job 180 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex11  (family=openmpi5)
[c2:09191] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:09192] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:21169] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:21168] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09191] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09192] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:21168] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:21169] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14

Test Suite: test_module

Results

Duration7.744 sec
Tests9
Failures1

Tests

test_module

Test case:[libs/HYPRE] Verify HYPRE module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.193 sec
Test case:[libs/HYPRE] Verify module HYPRE_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/HYPRE] Verify module HYPRE_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/HYPRE] Verify module HYPRE_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/HYPRE] Verify dynamic library available in HYPRE_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.007 sec
Test case:[libs/HYPRE] Verify static library is not present in HYPRE_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.007 sec
Test case:[libs/HYPRE] Verify module HYPRE_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.007 sec
Test case:[libs/HYPRE] Verify header file is present in HYPRE_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.007 sec
Test case:[libs/HYPRE] Sample job (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:7.497 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_module, line 131)
  `run_mpi_binary ./ex1 "atest" 1 1' failed
job script = /tmp/job.ohpc-test.31313
Batch job 171 submitted

Job 171 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex1 atest (family=openmpi5)
[c1:19574] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:19574] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 0: Exited with exit code 14

Test Suite: test_metis

Results

Duration3.524 sec
Tests4
Failures0

Tests

test_metis

Test case:[libs/Metis] Graph partition (gnu15)
Outcome:Passed
Duration:0.346 sec
Test case:[libs/Metis] Fill-reducing ordering (gnu15)
Outcome:Passed
Duration:2.742 sec
Test case:[libs/Metis] Mesh to graph conversion (gnu15)
Outcome:Passed
Duration:0.05 sec
Test case:[libs/Metis] C API mesh partitioning (slurm/gnu15)
Outcome:Passed
Duration:0.386 sec

Test Suite: test_module

Results

Duration0.26 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/Metis] Verify METIS module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.171 sec
Test case:[libs/Metis] Verify module METIS_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/Metis] Verify module METIS_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/Metis] Verify availability of m2gmetis binary (gnu15)
Outcome:Passed
Duration:0.029 sec
Test case:[libs/Metis] Verify module METIS_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/Metis] Verify dynamic library available in METIS_LIB (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/Metis] Verify static library is not present in METIS_LIB (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/Metis] Verify module METIS_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/Metis] Verify header file is present in METIS_INC (gnu15)
Outcome:Passed
Duration:0.008 sec

Test Suite: rm_execution

Results

Duration13.216 sec
Tests5
Failures0

Tests

rm_execution

Test case:[libs/Mumps] C (double precision) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec
Test case:[libs/Mumps] Fortran (single precision) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec
Test case:[libs/Mumps] Fortran (double precision) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.274 sec
Test case:[libs/Mumps] Fortran (complex) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.327 sec
Test case:[libs/Mumps] Fortran (double complex) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.169 sec

Test Suite: test_module

Results

Duration0.194 sec
Tests2
Failures0

Tests

test_module

Test case:[libs/MUMPS] Verify MUMPS module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.185 sec
Test case:[libs/MUMPS] Verify MUMPS_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.009 sec

Test Suite: rm_execution

Results

Duration13.338 sec
Tests5
Failures5

Tests

rm_execution

Test case:[libs/Mumps] C (double precision) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.309 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 26)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.27144
Batch job 188 submitted

Job 188 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./C_double null (family=openmpi5)
[c1:22331] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:09798] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:09797] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:22331] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09797] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09798] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
[2026-03-10T17:44:03.524] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.188.0:1]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:44:03.532] error: *** STEP 188.0 ON c1 CANCELLED AT 2026-03-10T17:44:03 DUE to SIGNAL Killed ***
srun: error: c1: task 0: Killed
Test case:[libs/Mumps] Fortran (single precision) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.242 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 36)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.9216
Batch job 189 submitted

Job 189 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./F_single null (family=openmpi5)
[c2:09884] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:09885] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:22512] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:22513] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09884] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09885] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:22512] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:22513] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/Mumps] Fortran (double precision) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:1.193 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 46)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.7328
Batch job 190 submitted

Job 190 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./F_double null (family=openmpi5)
[c2:09970] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:09969] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:22690] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:22689] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09970] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:09969] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:22689] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:22690] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/Mumps] Fortran (complex) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.296 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 56)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.11357
Batch job 191 submitted

Job 191 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./F_complex null (family=openmpi5)
[c1:22866] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:22867] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:10054] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:10055] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:22867] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:22866] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:10054] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:10055] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/Mumps] Fortran (double complex) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.298 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 66)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.28994
Batch job 192 submitted

Job 192 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./F_doublecomplex null (family=openmpi5)
[c2:10143] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:10142] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:23043] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:23043] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:10142] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:10143] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 0: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
[2026-03-10T17:44:14.465] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.192.0:0]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:44:14.473] error: *** STEP 192.0 ON c1 CANCELLED AT 2026-03-10T17:44:14 DUE to SIGNAL Killed ***
srun: error: c1: task 1: Killed

Test Suite: test_module

Results

Duration0.202 sec
Tests2
Failures0

Tests

test_module

Test case:[libs/MUMPS] Verify MUMPS module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.192 sec
Test case:[libs/MUMPS] Verify MUMPS_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.01 sec

Test Suite: test_C_module

Results

Duration0.396 sec
Tests9
Failures0

Tests

test_C_module

Test case:[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.201 sec
Test case:[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify availability of nc-config binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.049 sec
Test case:[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_CXX_module

Results

Duration0.407 sec
Tests9
Failures0

Tests

test_CXX_module

Test case:[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.212 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.048 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_Fortran_module

Results

Duration0.403 sec
Tests9
Failures0

Tests

test_Fortran_module

Test case:[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.208 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.048 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec

Test Suite: test_netcdf

Results

Duration3.764 sec
Tests8
Failures0

Tests

test_netcdf

Test case:[libs/NetCDF] ncdump availability (gnu15/openmpi5)
Outcome:Passed
Duration:0.136 sec
Test case:[libs/NetCDF] nctest availability (gnu15/openmpi5)
Outcome:Passed
Duration:0.363 sec
Test case:[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu15/openmpi5)
Outcome:Passed
Duration:0.083 sec
Test case:[libs/NetCDF] verify nc4 available for Fortran interface (gnu15/openmpi5)
Outcome:Passed
Duration:0.062 sec
Test case:[libs/NetCDF] verify nc4 available for C++ interface (gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
option no longer supported
Test case:[libs/NetCDF] C write/read (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.942 sec
Test case:[libs/NetCDF] Fortran write/read (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:1.119 sec
Test case:[libs/NetCDF] C++ write/read (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:1.059 sec

Test Suite: test_C_module

Results

Duration0.383 sec
Tests9
Failures0

Tests

test_C_module

Test case:[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.19 sec
Test case:[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify availability of nc-config binary (gnu15)
Outcome:Passed
Duration:0.048 sec
Test case:[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu15)
Outcome:Passed
Duration:0.022 sec

Test Suite: test_CXX_module

Results

Duration0.383 sec
Tests9
Failures0

Tests

test_CXX_module

Test case:[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.195 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu15)
Outcome:Passed
Duration:0.046 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu15)
Outcome:Passed
Duration:0.02 sec

Test Suite: test_Fortran_module

Results

Duration0.396 sec
Tests9
Failures0

Tests

test_Fortran_module

Test case:[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.203 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu15)
Outcome:Passed
Duration:0.048 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu15)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_netcdf

Results

Duration2.391 sec
Tests8
Failures0

Tests

test_netcdf

Test case:[libs/NetCDF] ncdump availability (gnu15)
Outcome:Passed
Duration:0.088 sec
Test case:[libs/NetCDF] nctest availability (gnu15)
Outcome:Passed
Duration:0.284 sec
Test case:[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu15)
Outcome:Passed
Duration:0.081 sec
Test case:[libs/NetCDF] verify nc4 available for Fortran interface (gnu15)
Outcome:Passed
Duration:0.062 sec
Test case:[libs/NetCDF] verify nc4 available for C++ interface (gnu15)
Outcome:Skipped
Duration:0.0 sec
Skipped
option no longer supported
Test case:[libs/NetCDF] C write/read (slurm/gnu15)
Outcome:Passed
Duration:0.544 sec
Test case:[libs/NetCDF] Fortran write/read (slurm/gnu15)
Outcome:Passed
Duration:0.698 sec
Test case:[libs/NetCDF] C++ write/read (slurm/gnu15)
Outcome:Passed
Duration:0.634 sec

Test Suite: test_C_module

Results

Duration0.39 sec
Tests9
Failures0

Tests

test_C_module

Test case:[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.194 sec
Test case:[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify availability of nc-config binary (gnu15/mpich)
Outcome:Passed
Duration:0.049 sec
Test case:[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec

Test Suite: test_CXX_module

Results

Duration0.406 sec
Tests9
Failures0

Tests

test_CXX_module

Test case:[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.217 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu15/mpich)
Outcome:Passed
Duration:0.047 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_Fortran_module

Results

Duration0.399 sec
Tests9
Failures0

Tests

test_Fortran_module

Test case:[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.206 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu15/mpich)
Outcome:Passed
Duration:0.047 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_netcdf

Results

Duration2.778 sec
Tests8
Failures0

Tests

test_netcdf

Test case:[libs/NetCDF] ncdump availability (gnu15/mpich)
Outcome:Passed
Duration:0.094 sec
Test case:[libs/NetCDF] nctest availability (gnu15/mpich)
Outcome:Passed
Duration:0.3 sec
Test case:[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu15/mpich)
Outcome:Passed
Duration:0.077 sec
Test case:[libs/NetCDF] verify nc4 available for Fortran interface (gnu15/mpich)
Outcome:Passed
Duration:0.062 sec
Test case:[libs/NetCDF] verify nc4 available for C++ interface (gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
option no longer supported
Test case:[libs/NetCDF] C write/read (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.814 sec
Test case:[libs/NetCDF] Fortran write/read (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.524 sec
Test case:[libs/NetCDF] C++ write/read (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.907 sec

Test Suite: test_pnetcdf

Results

Duration2.377 sec
Tests2
Failures0

Tests

test_pnetcdf

Test case:[libs/NetCDF] C parallel I/O (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.377 sec
Test case:[libs/NetCDF] Fortran parallel I/O (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
Skipping fortran test bindings with mpich

Test Suite: test_pnetcdf

Results

Duration9.806 sec
Tests2
Failures2

Tests

test_pnetcdf

Test case:[libs/NetCDF] C parallel I/O (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.378 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_pnetcdf, line 47)
  `run_mpi_binary ./C_parallel "atest" 2 4' failed
job script = /tmp/job.ohpc-test.18033
Batch job 214 submitted

Job 214 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./C_parallel atest (family=openmpi5)
[c1:24506] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:24505] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:11230] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:11231] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:24505] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:24506] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:11230] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:11231] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/NetCDF] Fortran parallel I/O (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:5.428 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_pnetcdf, line 83)
  `run_mpi_binary -t "00:02:00" ./F90_parallel "atest" 2 4' failed
job script = /tmp/job.ohpc-test.30762
Batch job 215 submitted

Job 215 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./F90_parallel atest (family=openmpi5)
[c1:24682] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:24683] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:11315] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:11316] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:24683] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:24682] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:11315] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:11316] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14

Test Suite: rm_execution

Results

Duration3.843 sec
Tests11
Failures0

Tests

rm_execution

Test case:[libs/OpenBLAS/dblat1] dblat1 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.121 sec
Test case:[libs/OpenBLAS/xccblat1] xccblat1 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.122 sec
Test case:[libs/OpenBLAS/xzcblat1] xzcblat1 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.171 sec
Test case:[libs/OpenBLAS/xscblat2] xscblat2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.328 sec
Test case:[libs/OpenBLAS/xdcblat2] xdcblat2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.378 sec
Test case:[libs/OpenBLAS/xccblat2] xccblat2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.767 sec
Test case:[libs/OpenBLAS/xzcblat2] xzcblat2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.663 sec
Test case:[libs/OpenBLAS/xscblat3] xscblat3 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.186 sec
Test case:[libs/OpenBLAS/xdcblat3] xdcblat3 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.194 sec
Test case:[libs/OpenBLAS/xccblat3] xccblat3 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.223 sec
Test case:[libs/OpenBLAS/xzcblat3] xzcblat3 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.69 sec

Test Suite: test_lapack

Results

Duration0.286 sec
Tests1
Failures0

Tests

test_lapack

Test case:[libs/OpenBLAS/eigen] run lapack eigen-value solver (gnu15)
Outcome:Passed
Duration:0.286 sec

Test Suite: test_module

Results

Duration0.282 sec
Tests5
Failures0

Tests

test_module

Test case:[libs/OpenBLAS] Verify OPENBLAS module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.2 sec
Test case:[libs/OpenBLAS] Verify OPENBLAS_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/OpenBLAS] Verify module OPENBLAS_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/OpenBLAS] Verify dynamic library available in OPENBLAS_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/OpenBLAS] Verify static library is not present in OPENBLAS_LIB (gnu15)
Outcome:Passed
Duration:0.019 sec

Test Suite: rm_execution

Results

Duration5.619 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/PETSc] C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.169 sec
Test case:[libs/PETSc] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec
Test case:[libs/PETSc] MPI F90 binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.227 sec

Test Suite: test_module

Results

Duration2.583 sec
Tests8
Failures0

Tests

test_module

Test case:[libs/PETSc] Verify PETSC module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.222 sec
Test case:[libs/PETSc] Verify module PETSC_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/PETSc] Verify module PETSC_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Verify dynamic library available in PETSC_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PETSc] Verify static library is not present in PETSC_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Verify module PETSC_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Verify header file is present in PETSC_INC (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PETSc] Sample job (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.238 sec

Test Suite: rm_execution

Results

Duration10.948 sec
Tests3
Failures3

Tests

rm_execution

Test case:[libs/PETSc] C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.248 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 27)
  `run_mpi_binary ./C_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.23464
Batch job 238 submitted

Job 238 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./C_test 8 (family=openmpi5)
[c1:26496] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:26495] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:26494] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:26497] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12337] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12339] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26496] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26494] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26495] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26497] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12339] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12337] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-3: Exited with exit code 14
srun: error: c2: tasks 4,6: Exited with exit code 14
[2026-03-10T17:47:32.526] error:  mpi/pmix_v4: _errhandler: c2 [1]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.238.0:6]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: c2: tasks 5,7: Killed
Test case:[libs/PETSc] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.351 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 36)
  `run_mpi_binary ./C_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.3940
Batch job 239 submitted

Job 239 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./C_mpi_test 8 (family=openmpi5)
[c1:26682] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:26683] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:26684] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12434] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12432] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12435] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12433] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26684] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26682] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26683] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12435] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12434] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12432] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12433] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 1-3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14
[2026-03-10T17:47:34.236] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.239.0:3]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:47:34.244] error: *** STEP 239.0 ON c1 CANCELLED AT 2026-03-10T17:47:34 DUE to SIGNAL Killed ***
srun: error: c1: task 0: Killed
Test case:[libs/PETSc] MPI F90 binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.349 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 45)
  `run_mpi_binary ./F90_test 2 "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.15391
Batch job 240 submitted

Job 240 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./F90_test 2 (family=openmpi5)
[c2:12524] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12525] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12523] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12526] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:26869] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:26871] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:26870] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:26868] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12524] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12525] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12523] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12526] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26869] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26871] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26870] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26868] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 4-7: Exited with exit code 14
srun: error: c1: tasks 0-3: Exited with exit code 14

Test Suite: test_module

Results

Duration2.61 sec
Tests8
Failures1

Tests

test_module

Test case:[libs/PETSc] Verify PETSC module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.223 sec
Test case:[libs/PETSc] Verify module PETSC_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/PETSc] Verify module PETSC_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PETSc] Verify dynamic library available in PETSC_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PETSc] Verify static library is not present in PETSC_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Verify module PETSC_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Verify header file is present in PETSC_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/PETSc] Sample job (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.26 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file test_module, line 132)
  `run_mpi_binary ./C_test "atest" 1 1' failed
job script = /tmp/job.ohpc-test.27616
Batch job 237 submitted

Job 237 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./C_test atest (family=openmpi5)
[c1:26319] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:26319] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 0: Exited with exit code 14

Test Suite: rm_execution

Results

Duration4.441 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/PHDF5] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.222 sec
Test case:[libs/PHDF5] Parallel Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.219 sec

Test Suite: test_module

Results

Duration0.31 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/PHDF5] Verify HDF5 module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.191 sec
Test case:[libs/PHDF5] Verify HDF5_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/PHDF5] Verify module HDF5_LIB is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify dynamic library available in HDF5_LIB (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify static library is not present in HDF5_LIB (gnu15-mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PHDF5] Verify module HDF5_INC is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PHDF5] Verify header file is present in HDF5_INC (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec

Test Suite: rm_execution

Results

Duration5.544 sec
Tests2
Failures2

Tests

rm_execution

Test case:[libs/PHDF5] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.249 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 43)
  `run_mpi_binary -t "${TIMEOUT}" ./C_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.7066
Batch job 243 submitted

Job 243 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./C_mpi_test /home/ohpc-test/phdf5_test.K8KXy4/phdf_test.out (family=openmpi5)
[c2:12802] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:27425] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12800] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12801] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12799] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:27426] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:27427] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:27424] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12802] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12801] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12800] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12799] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:27425] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:27424] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:27426] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:27427] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 4-7: Exited with exit code 14
srun: error: c1: tasks 0-3: Exited with exit code 14
Test case:[libs/PHDF5] Parallel Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.295 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 52)
  `run_mpi_binary -t "${TIMEOUT}" ./F_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.588
Batch job 244 submitted

Job 244 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./F_mpi_test /home/ohpc-test/phdf5_test.K8KXy4/phdf_test.out (family=openmpi5)
[c1:27606] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:27605] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:27607] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12891] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12888] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12890] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:12889] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:27605] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:27607] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:27606] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12889] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12891] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12888] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:12890] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-2: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14
[2026-03-10T17:47:58.202] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.244.0:0]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:47:58.210] error: *** STEP 244.0 ON c1 CANCELLED AT 2026-03-10T17:47:58 DUE to SIGNAL Killed ***
srun: error: c1: task 3: Killed

Test Suite: test_module

Results

Duration0.323 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/PHDF5] Verify HDF5 module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.202 sec
Test case:[libs/PHDF5] Verify HDF5_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PHDF5] Verify module HDF5_LIB is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PHDF5] Verify dynamic library available in HDF5_LIB (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PHDF5] Verify static library is not present in HDF5_LIB (gnu15-openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PHDF5] Verify module HDF5_INC is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PHDF5] Verify header file is present in HDF5_INC (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration1.952 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/PLASMA/C_test] C_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.8 sec
Test case:[libs/PLASMA/F90_test] F90_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.152 sec

Test Suite: test_module

Results

Duration0.309 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/PLASMA] Verify PLASMA module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.19 sec
Test case:[libs/PLASMA] Verify PLASMA_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PLASMA] Verify module PLASMA_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.017 sec
Test case:[libs/PLASMA] Verify dynamic library available in PLASMA_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PLASMA] Verify static library is not present in PLASMA_LIB (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PLASMA] Verify module PLASMA_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PLASMA] Verify header file is present in PLASMA_INC (gnu15)
Outcome:Passed
Duration:0.019 sec

Test Suite: test_module

Results

Duration0.226 sec
Tests5
Failures0

Tests

test_module

Test case:[PNETCDF] Verify PNETCDF module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.19 sec
Test case:[PNETCDF] Verify module PNETCDF_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.01 sec
Test case:[PNETCDF] Verify module PNETCDF_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[PNETCDF] Verify module PNETCDF_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[PNETCDF] Verify header file is present in PNETCDF_INC (gnu15)
Outcome:Passed
Duration:0.009 sec

Test Suite: rm_execution

Results

Duration10.976 sec
Tests4
Failures0

Tests

rm_execution

Test case:[libs/PNETCDF] Parallel Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.219 sec
Test case:[libs/PNETCDF] Parallel Fortran binary 2 runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.221 sec
Test case:[libs/PNETCDF] Parallel Fortran binary 3 runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.267 sec
Test case:[libs/PNETCDF] Parallel Fortran binary 4 runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.269 sec

Test Suite: rm_execution

Results

Duration11.083 sec
Tests4
Failures4

Tests

rm_execution

Test case:[libs/PNETCDF] Parallel Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.297 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 22)
  `run_mpi_binary ./f90tst_parallel f90tst_parallel.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.11878
Batch job 251 submitted

Job 251 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./f90tst_parallel f90tst_parallel.nc (family=openmpi5)
[c1:28603] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:28603] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 0: Exited with exit code 14
[2026-03-10T17:50:06.533] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.251.0:0]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:50:06.541] error: *** STEP 251.0 ON c1 CANCELLED AT 2026-03-10T17:50:06 DUE to SIGNAL Killed ***
srun: error: c1: task 1: Killed
Test case:[libs/PNETCDF] Parallel Fortran binary 2 runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.243 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 31)
  `run_mpi_binary ./f90tst_parallel2 f90tst_parallel2.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.25933
Batch job 252 submitted

Job 252 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./f90tst_parallel2 f90tst_parallel2.nc (family=openmpi5)
[c1:28785] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:28784] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:28784] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:28785] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/PNETCDF] Parallel Fortran binary 3 runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:1.196 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 40)
  `run_mpi_binary ./f90tst_parallel3 f90tst_parallel3.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.25226
Batch job 253 submitted

Job 253 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./f90tst_parallel3 f90tst_parallel3.nc (family=openmpi5)
[c1:28961] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:28960] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:28960] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:28961] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/PNETCDF] Parallel Fortran binary 4 runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.347 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 49)
  `run_mpi_binary ./f90tst_parallel4 f90tst_parallel4.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.29434
Batch job 254 submitted

Job 254 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./f90tst_parallel4 f90tst_parallel4.nc (family=openmpi5)
[c1:29136] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:29137] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:29136] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:29137] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14

Test Suite: rm_execution

Results

Duration6.664 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/PTScotch] dgraph_redist binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.219 sec
Test case:[libs/PTScotch] strat_par binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.221 sec
Test case:[libs/PTScotch] dgord binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.224 sec

Test Suite: test_module

Results

Duration0.386 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/PTScotch] Verify PTSCOTCH module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.215 sec
Test case:[libs/PTScotch] Verify PTSCOTCH_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_LIB is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PTScotch] Verify dynamic library available in PTSCOTCH_LIB (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify static library is not present in PTSCOTCH_LIB (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_INC is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify header file is present in PTSCOTCH_INC (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_BIN is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify availability of dgscat binary (gnu15-mpich)
Outcome:Passed
Duration:0.032 sec

Test Suite: rm_execution

Results

Duration16.163 sec
Tests3
Failures2

Tests

rm_execution

Test case:[libs/PTScotch] dgraph_redist binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.242 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 27)
  `run_mpi_binary -t "${TIMEOUT}" ./dgraph_redist "bump.grf" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.29191
Batch job 258 submitted

Job 258 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./dgraph_redist bump.grf (family=openmpi5)
[c2:13401] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:13402] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:29909] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:29908] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:13402] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:13401] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:29908] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:29909] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/PTScotch] strat_par binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:7.473 sec
Test case:[libs/PTScotch] dgord binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:6.448 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 45)
  `run_mpi_binary -t "${TIMEOUT}" dgord "bump.grf /dev/null -vt" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.23141
Batch job 260 submitted

Job 260 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- dgord bump.grf /dev/null -vt (family=openmpi5)
[c2:13569] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:30260] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:13569] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:30260] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: task 2: Exited with exit code 14
srun: error: c1: task 0: Exited with exit code 14
[2026-03-10T17:50:48.215] error:  mpi/pmix_v4: _errhandler: c2 [1]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.260.0:2]
[2026-03-10T17:50:48.217] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.260.0:0]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:50:48.222] error: *** STEP 260.0 ON c1 CANCELLED AT 2026-03-10T17:50:48 DUE to SIGNAL Killed ***
srun: error: c1: task 1: Killed
srun: error: c2: task 3: Killed

Test Suite: test_module

Results

Duration0.383 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/PTScotch] Verify PTSCOTCH module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.201 sec
Test case:[libs/PTScotch] Verify PTSCOTCH_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_LIB is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify dynamic library available in PTSCOTCH_LIB (gnu15-openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/PTScotch] Verify static library is not present in PTSCOTCH_LIB (gnu15-openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_INC is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify header file is present in PTSCOTCH_INC (gnu15-openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_BIN is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify availability of dgscat binary (gnu15-openmpi5)
Outcome:Passed
Duration:0.033 sec

Test Suite: rm_execution

Results

Duration19.4 sec
Tests4
Failures0

Tests

rm_execution

Test case:[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.329 sec
Test case:[libs/ScaLAPACK/PDSCAEX] DPCGESV under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.322 sec
Test case:[libs/ScaLAPACK/PSSCAEX] SPCGESV under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:5.373 sec
Test case:[libs/ScaLAPACK/PZSCAEX] ZPCGESV under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:5.376 sec

Test Suite: test_module

Results

Duration0.259 sec
Tests5
Failures0

Tests

test_module

Test case:[libs/ScaLAPACK] Verify SCALAPACK module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.183 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/ScaLAPACK] Verify dynamic library available in SCALAPACK_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/ScaLAPACK] Verify static library is not present in SCALAPACK_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec

Test Suite: rm_execution

Results

Duration16.33 sec
Tests4
Failures4

Tests

rm_execution

Test case:[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.299 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 35)
  `run_mpi_binary ./pcscaex "${ARGS}" "${NODES}" "${TASKS}"' failed
run_mpi_binary ./pcscaex 0 2 8
SHELL=/bin/bash
BATS_RUN_LOGS_DIRECTORY=
BATS_TEST_TMPDIR=/tmp/bats-run-iayYzv/test/1
SUDO_GID=0
HISTCONTROL=ignoredups
OMPI_MCA_rmaps_base_oversubscribe=1
UCX_INC=/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/include
PARALLEL_JOBSLOT0=0
PKG_CONFIG_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/lib/pkgconfig:/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/lib/pkgconfig:/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9/lib/pkgconfig
HOSTNAME=ohpc-lenovo-sms
HISTSIZE=1048576
FPATH=/opt/ohpc/admin/lmod/lmod/init/ksh_funcs
BATS_LIBDIR=lib
BATS_SUITE_TMPDIR=/tmp/bats-run-iayYzv/suite
BATS_ROOT_PID=320551
BATS_WARNING_FILE=/tmp/bats-run-iayYzv/warnings.log
BATS_FILE_TMPDIR=/tmp/bats-run-iayYzv/file/0
_ModuleTable007_=dWxlZmlsZXMiLAp9Cg==
_ModuleTable002_=dGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAiZ251MTUiLAp3ViA9ICIwMDAwMDAwMTUuMDAwMDAwMDAyLip6ZmluYWwiLAp9LApod2xvYyA9IHsKZm4gPSAiL29wdC9vaHBjL3B1Yi9tb2R1bGVmaWxlcy9od2xvYy8yLjEyLjEiLApmdWxsTmFtZSA9ICJod2xvYy8yLjEyLjEiLApsb2FkT3JkZXIgPSAzLApwcm9wVCA9IHt9LApyZWZfY291bnQgPSAxLApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImh3bG9jIiwKd1YgPSAiMDAwMDAwMDAyLjAwMDAwMDAxMi4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCm9wZW5ibGFzID0gewpmbiA9ICIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUvb3BlbmJsYXMvMC4zLjMxIiwKZnVsbE5hbWUg
BATS_SEMAPHORE_DIR=/tmp/bats-run-iayYzv/semaphores
__LMOD_REF_COUNT_MODULEPATH=/opt/ohpc/pub/moduledeps/gnu15-openmpi5:1;/opt/ohpc/pub/moduledeps/gnu15:1;/opt/ohpc/pub/modulefiles:1
_ModuleTable005_=aXZlIiwKdXNlck5hbWUgPSAicHJ1biIsCndWID0gIjAwMDAwMDAwMi4wMDAwMDAwMDIuKnpmaW5hbCIsCn0sCnNjYWxhcGFjayA9IHsKZGVwVCA9IHsKZGVwQSA9IHsKewpzbiA9ICJvcGVuYmxhcyIsCnZlcnNpb24gPSB7CmtpbmQgPSAiYm9vbCIsCnZhbHVlID0gdHJ1ZSwKfSwKfSwKfSwKfSwKZm4gPSAiL29wdC9vaHBjL3B1Yi9tb2R1bGVkZXBzL2dudTE1LW9wZW5tcGk1L3NjYWxhcGFjay8yLjIuMiIsCmZ1bGxOYW1lID0gInNjYWxhcGFjay8yLjIuMiIsCmxvYWRPcmRlciA9IDcsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAic2NhbGFwYWNrIiwKd1YgPSAiMDAwMDAwMDAyLjAwMDAwMDAwMi4wMDAwMDAwMDIu
TEST_LOGS=test_module.log rm_execution.log
SSHPASS=
BATS_END_CODE_QUOTE='
SUDO_COMMAND=/bin/bash -c /tmp/user_integration_tests
SUDO_USER=root
BATS_TEST_SOURCE=/home/ohpc-test/tests/libs/scalapack/tests/rm_execution
BATS_TEST_NAME=test_-5b-24TESTNAME-2fPCSCAEX-5d_CPCGESV_under_resource_manager_-28-24RESOURCE-5fMANAGER-2f-24LMOD-5fFAMILY-5fCOMPILER-2f-24LMOD-5fFAMILY-5fMPI-29
GPG_TTY=not a tty
LMOD_DIR=/opt/ohpc/admin/lmod/lmod/libexec
LMOD_FAMILY_OPENBLAS_VERSION=0.3.31
OHPC_MPI_LAUNCHERS=pmix
PWD=/home/ohpc-test/tests/libs/scalapack/tests
LOGNAME=ohpc-test
HWLOC_INC=/opt/ohpc/pub/libs/hwloc/include
MODULESHOME=/opt/ohpc/admin/lmod/lmod
PARALLEL_TMP=/tmp/parU9WWV
MANPATH=/opt/ohpc/pub/libs/hwloc/man:/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9/share/man:/opt/ohpc/pub/compiler/gcc/15.2.0/share/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:/opt/ohpc/admin/lmod/lmod/share/man::
OPENBLAS_INC=/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/include
BATS_JUNIT_FORMAT=1
CI_CLUSTER=lenovo
LMOD_PREPEND_BLOCK=normal
BATS_TMPDIR=/tmp
__LMOD_REF_COUNT_INCLUDE=/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/include:1;/opt/ohpc/pub/compiler/gcc/15.2.0/include:1
BATS_USE_MKTEMP=1
BATS_TEST_DESCRIPTION=[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu15/openmpi5)
UCX_LIB=/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/lib
__LMOD_REF_COUNT_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/bin:1;/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/bin:1;/opt/ohpc/pub/libs/hwloc/bin:1;/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9/bin:1;/opt/ohpc/pub/compiler/gcc/15.2.0/bin:1;/opt/ohpc/pub/utils/prun/2.2:1;/home/ohpc-test/.local/bin:1;/home/ohpc-test/bin:1;/sbin:1;/bin:1;/usr/sbin:1;/usr/bin:1;/usr/local/sbin:1
HOME=/home/ohpc-test
FI_PROVIDER=sockets
ARGS=0
PARALLEL_RSYNC_OPTS=-rlDzR
_ModuleTable_Sz_=7
LANG=en_US.UTF-8
BATS_TEST_FILENAME=/home/ohpc-test/tests/libs/scalapack/tests/rm_execution
__LMOD_REF_COUNT_PKG_CONFIG_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/lib/pkgconfig:1;/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/lib/pkgconfig:1;/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9/lib/pkgconfig:1
SCALAPACK_LIB=/opt/ohpc/pub/libs/gnu15/openmpi5/scalapack/2.2.2/lib
srcdir=.
PARALLEL_PID=320612
BATS_RUN_TMPDIR=/tmp/bats-run-iayYzv
LMOD_SETTARG_FULL_SUPPORT=no
BATS_JUNIT_CLASS=scalapack
BATS_VERSION=1.11.1
TMPDIR=/tmp
https_proxy=http://10.241.58.130:3128
OPENBLAS_DIR=/opt/ohpc/pub/libs/gnu15/openblas/0.3.31
LMOD_VERSION=9.0.5
BATS_GUARANTEED_MINIMUM_VERSION=0.0.0
TESTNAME=libs/ScaLAPACK
SINGULARITY_TMPDIR=/var/tmp
MFLAGS=-k --no-print-directory
_ModuleTable003_=PSAib3BlbmJsYXMvMC4zLjMxIiwKbG9hZE9yZGVyID0gNiwKcHJvcFQgPSB7fSwKcmVmX2NvdW50ID0gMSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJvcGVuYmxhcyIsCndWID0gIjAwMDAwMDAwMC4wMDAwMDAwMDMuMDAwMDAwMDMxLip6ZmluYWwiLAp9LApvcGVubXBpNSA9IHsKYWN0aW9uQSA9IHsKW1twcmVwZW5kX3BhdGgoIk1PRFVMRVBBVEgiLCIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUtb3Blbm1waTUiKV1dLAp9LApkZXBUID0gewpkZXBBID0gewp7CnNuID0gImh3bG9jIiwKdmVyc2lvbiA9IHsKa2luZCA9ICJib29sIiwKdmFsdWUgPSB0cnVlLAp9LAp9LAp7CnNuID0gInVjeCIsCnZlcnNpb24gPSB7CmtpbmQg
PARALLEL_SEQ=1
AUTOMAKE_JUNIT_FILE=1
LMOD_FAMILY_OPENBLAS=openblas
BATS_CWD=/home/ohpc-test/tests/libs/scalapack/tests
MAKEFLAGS=k --no-print-directory -- TEST_LOGS=test_module.log\ rm_execution.log
LMOD_PKG=/opt/ohpc/admin/lmod/lmod
TERM=unknown
BATS_LINE_REFERENCE_FORMAT=comma_line
BATS_SEMAPHORE_NUMBER_OF_SLOTS=4
LESSOPEN=||/usr/bin/lesspipe.sh %s
PARALLEL_SSHLOGIN=:
USER=ohpc-test
BATS_BEGIN_CODE_QUOTE=`
NODES=2
LOADEDMODULES=prun/2.2:gnu15/15.2.0:hwloc/2.12.1:ucx/1.20.0:openmpi5/5.0.9:openblas/0.3.31:scalapack/2.2.2
HWLOC_DIR=/opt/ohpc/pub/libs/hwloc
BATS_TEMPDIR_CLEANUP=1
INCLUDE=/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/include:/opt/ohpc/pub/compiler/gcc/15.2.0/include
LMOD_ROOT=/opt/ohpc/admin/lmod
LMOD_SETTARG_CMD=:
TASKS=8
SHLVL=10
BASH_ENV=/opt/ohpc/admin/lmod/lmod/init/bash
_ModuleTable006_=KnpmaW5hbCIsCn0sCnVjeCA9IHsKZm4gPSAiL29wdC9vaHBjL3B1Yi9tb2R1bGVmaWxlcy91Y3gvMS4yMC4wIiwKZnVsbE5hbWUgPSAidWN4LzEuMjAuMCIsCmxvYWRPcmRlciA9IDQsCnByb3BUID0ge30sCnJlZl9jb3VudCA9IDEsCnN0YWNrRGVwdGggPSAxLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAidWN4IiwKd1YgPSAiMDAwMDAwMDAxLjAwMDAwMDAyMC4qemZpbmFsIiwKfSwKfSwKbXBhdGhBID0gewoiL29wdC9vaHBjL3B1Yi9tb2R1bGVkZXBzL2dudTE1LW9wZW5tcGk1IiwgIi9vcHQvb2hwYy9wdWIvbW9kdWxlZGVwcy9nbnUxNSIsICIvb3B0L29ocGMvcHViL21vZHVsZWZpbGVzIiwKfSwKc3lzdGVtQmFzZU1QQVRIID0gIi9vcHQvb2hwYy9wdWIvbW9k
BATS_REPORT_FILENAME=rm_execution.log.xml
MAKELEVEL=8
LMOD_FULL_SETTARG_SUPPORT=no
MPI_DIR=/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9
UCX_BIN=/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/bin
__LMOD_REF_COUNT_MANPATH=/opt/ohpc/pub/libs/hwloc/man:1;/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9/share/man:1;/opt/ohpc/pub/compiler/gcc/15.2.0/share/man:1;/usr/local/share/man:1;/usr/share/man/overrides:1;/usr/share/man/en:1;/usr/share/man:1;/opt/ohpc/admin/lmod/lmod/share/man:1;:1
http_proxy=http://10.241.58.130:3128
PARALLEL_SSHHOST=:
HWLOC_LIB=/opt/ohpc/pub/libs/hwloc/lib
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpNUEkgPSAib3Blbm1waTUiLApjb21waWxlciA9ICJnbnUxNSIsCm9wZW5ibGFzID0gIm9wZW5ibGFzIiwKfSwKbVQgPSB7CmdudTE1ID0gewphY3Rpb25BID0gewpbW3ByZXBlbmRfcGF0aCgiTU9EVUxFUEFUSCIsIi9vcHQvb2hwYy9wdWIvbW9kdWxlZGVwcy9nbnUxNSIpXV0sCn0sCmZuID0gIi9vcHQvb2hwYy9wdWIvbW9kdWxlZmlsZXMvZ251MTUvMTUuMi4wLmx1YSIsCmZ1bGxOYW1lID0gImdudTE1LzE1LjIuMCIsCmxvYWRPcmRlciA9IDIsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApz
LD_LIBRARY_PATH=/opt/ohpc/pub/libs/gnu15/openmpi5/scalapack/2.2.2/lib:/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/lib:/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/lib:/opt/ohpc/pub/libs/hwloc/lib:/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9/lib:/opt/ohpc/pub/compiler/gcc/15.2.0/lib64
LMOD_FAMILY_COMPILER=gnu15
LMOD_COLORIZE=no
_ModuleTable004_=PSAiYm9vbCIsCnZhbHVlID0gdHJ1ZSwKfSwKfSwKfSwKfSwKZm4gPSAiL29wdC9vaHBjL3B1Yi9tb2R1bGVkZXBzL2dudTE1L29wZW5tcGk1LzUuMC45IiwKZnVsbE5hbWUgPSAib3Blbm1waTUvNS4wLjkiLApsb2FkT3JkZXIgPSA1LApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gIm9wZW5tcGk1IiwKd1YgPSAiMDAwMDAwMDA1LjAwMDAwMDAwMC4wMDAwMDAwMDkuKnpmaW5hbCIsCn0sCnBydW4gPSB7CmZuID0gIi9vcHQvb2hwYy9wdWIvbW9kdWxlZmlsZXMvcHJ1bi8yLjIiLApmdWxsTmFtZSA9ICJwcnVuLzIuMiIsCmxvYWRPcmRlciA9IDEsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0
LMOD_FAMILY_MPI_VERSION=5.0.9
RESOURCE_MANAGER=slurm
BATS_TEST_RETRIES=0
OMP_NUM_THREADS=2
LMOD_FAMILY_MPI=openmpi5
DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima:
UCX_WARN_UNUSED_ENV_VARS=N
UCX_DIR=/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0
BATS_RUNLOG_FILE=/dev/null
LMOD_FAMILY_COMPILER_VERSION=15.2.0
PARALLEL_REMOTE_TMPDIR=/tmp
BATS_TEST_NUMBER=1
BATS_ENABLE_TIMING=-T
OMPI_MCA_mca_base_component_show_load_errors=0
PATH=/usr/libexec/bats-core:/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/bin:/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/bin:/opt/ohpc/pub/libs/hwloc/bin:/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9/bin:/opt/ohpc/pub/compiler/gcc/15.2.0/bin:/opt/ohpc/pub/utils/prun/2.2:/home/ohpc-test/.local/bin:/home/ohpc-test/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin
HWLOC_BIN=/opt/ohpc/pub/libs/hwloc/bin
MODULEPATH=/opt/ohpc/pub/moduledeps/gnu15-openmpi5:/opt/ohpc/pub/moduledeps/gnu15:/opt/ohpc/pub/modulefiles
SCALAPACK_DIR=/opt/ohpc/pub/libs/gnu15/openmpi5/scalapack/2.2.2
_LMFILES_=/opt/ohpc/pub/modulefiles/prun/2.2:/opt/ohpc/pub/modulefiles/gnu15/15.2.0.lua:/opt/ohpc/pub/modulefiles/hwloc/2.12.1:/opt/ohpc/pub/modulefiles/ucx/1.20.0:/opt/ohpc/pub/moduledeps/gnu15/openmpi5/5.0.9:/opt/ohpc/pub/moduledeps/gnu15/openblas/0.3.31:/opt/ohpc/pub/moduledeps/gnu15-openmpi5/scalapack/2.2.2
SUDO_UID=0
LMOD_CMD=/opt/ohpc/admin/lmod/lmod/libexec/lmod
MAIL=/var/spool/mail/ohpc-test
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/opt/ohpc/pub/libs/gnu15/openmpi5/scalapack/2.2.2/lib:1;/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/lib:1;/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/lib:1;/opt/ohpc/pub/libs/hwloc/lib:1;/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9/lib:1;/opt/ohpc/pub/compiler/gcc/15.2.0/lib64:1
BATS_JUNIT_GROUP=UserLevelTests
BATS_LIB_PATH=/usr/lib/bats
OPENBLAS_LIB=/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/lib
PARALLEL_JOBSLOT=1
BATS_LIBEXEC=/usr/libexec/bats-core
BATS_SUITE_TEST_NUMBER=1
BATS_ROOT=/usr
OLDPWD=/home/ohpc-test/tests/libs/scalapack/tests
VERBOSE=1
BATS_SAVED_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.31/bin:/opt/ohpc/pub/mpi/ucx-ohpc/1.20.0/bin:/opt/ohpc/pub/libs/hwloc/bin:/opt/ohpc/pub/mpi/openmpi5-gnu15/5.0.9/bin:/opt/ohpc/pub/compiler/gcc/15.2.0/bin:/opt/ohpc/pub/utils/prun/2.2:/home/ohpc-test/.local/bin:/home/ohpc-test/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin
BASH_FUNC_ml%%=() {  eval "$($LMOD_DIR/ml_cmd "$@")"
}
BASH_FUNC_bats_semaphore_acquire_while_locked%%=() {  if [[ $(bats_semaphore_get_free_slot_count) -gt 0 ]]; then
 local slot=0;
 while [[ -e "$BATS_SEMAPHORE_DIR/slot-$slot" ]]; do
 ((++slot));
 done;
 if [[ $slot -lt $BATS_SEMAPHORE_NUMBER_OF_SLOTS ]]; then
 touch "$BATS_SEMAPHORE_DIR/slot-$slot" && printf "%d\n" "$slot" && return 0;
 fi;
 fi;
 return 1
}
BASH_FUNC_bats_readlinkf%%=() {  readlink -f "$1"
}
BASH_FUNC_bats_semaphore_get_free_slot_count%%=() {  until used_slots=$(find "$BATS_SEMAPHORE_DIR" -name 'slot-*' 2> /dev/null | wc -l); do
 :;
 done;
 echo $((BATS_SEMAPHORE_NUMBER_OF_SLOTS - used_slots))
}
BASH_FUNC_module%%=() {  if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
 case "$-" in
 *v*x*)
 __lmod_sh_dbg='vx'
 ;;
 *v*)
 __lmod_sh_dbg='v'
 ;;
 *x*)
 __lmod_sh_dbg='x'
 ;;
 esac;
 fi;
 if [ -n "${__lmod_sh_dbg:-}" ]; then
 set +$__lmod_sh_dbg;
 echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output" 1>&2;
 fi;
 eval "$($LMOD_CMD shell "$@")" && eval "$(${LMOD_SETTARG_CMD:-:} -s sh)";
 __lmod_my_status=$?;
 if [ -n "${__lmod_sh_dbg:-}" ]; then
 echo "Shell debugging restarted" 1>&2;
 set -$__lmod_sh_dbg;
 fi;
 unset __lmod_sh_dbg;
 return $__lmod_my_status
}
_=/bin/printenv
job script = /tmp/job.ohpc-test.24443
Batch job 266 submitted

Job 266 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pcscaex 0 (family=openmpi5)
[c1:31277] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31278] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14039] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14038] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31277] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31278] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14039] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: task 7: Exited with exit code 14
[2026-03-10T17:51:51.540] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.266.0:0]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:51:51.547] error: *** STEP 266.0 ON c1 CANCELLED AT 2026-03-10T17:51:51 DUE to SIGNAL Killed ***
srun: error: c1: tasks 2-3: Killed
srun: error: c2: tasks 4-6: Killed
Test case:[libs/ScaLAPACK/PDSCAEX] DPCGESV under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.292 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 44)
  `run_mpi_binary ./pdscaex "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.31119
Batch job 267 submitted

Job 267 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pdscaex 0 (family=openmpi5)
[c2:14133] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14134] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14131] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14132] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14134] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14133] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14132] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[c1:31463] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31462] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31464] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31465] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14131] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31463] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31465] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31464] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31462] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 4-7: Exited with exit code 14
srun: error: c1: tasks 0-3: Exited with exit code 14
Test case:[libs/ScaLAPACK/PSSCAEX] SPCGESV under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.342 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 53)
  `run_mpi_binary ./psscaex "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.12813
Batch job 268 submitted

Job 268 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./psscaex 0 (family=openmpi5)
[c2:14220] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14223] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14221] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14222] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31643] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31644] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31645] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31646] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14220] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14221] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14222] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14223] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31645] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31646] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31643] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31644] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 4-7: Exited with exit code 14
srun: error: c1: tasks 0-3: Exited with exit code 14
Test case:[libs/ScaLAPACK/PZSCAEX] ZPCGESV under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:5.397 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 62)
  `run_mpi_binary ./pzscaex "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.15209
Batch job 269 submitted

Job 269 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pzscaex 0 (family=openmpi5)
[c1:31826] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31827] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31824] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:31825] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14310] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14311] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14312] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14309] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31826] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31827] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31825] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:31824] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14309] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14312] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14311] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14310] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14

Test Suite: test_module

Results

Duration0.292 sec
Tests5
Failures0

Tests

test_module

Test case:[libs/ScaLAPACK] Verify SCALAPACK module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.209 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/ScaLAPACK] Verify dynamic library available in SCALAPACK_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/ScaLAPACK] Verify static library is not present in SCALAPACK_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec

Test Suite: rm_execution

Results

Duration0.529 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/Scotch] graph_map binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.282 sec
Test case:[libs/Scotch] strat_seq binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.125 sec
Test case:[libs/Scotch] gout binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.122 sec

Test Suite: test_module

Results

Duration0.355 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/Scotch] Verify SCOTCH module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.189 sec
Test case:[libs/Scotch] Verify SCOTCH_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/Scotch] Verify module SCOTCH_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/Scotch] Verify dynamic library available in SCOTCH_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/Scotch] Verify static library is not present in SCOTCH_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/Scotch] Verify module SCOTCH_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/Scotch] Verify header file is present in SCOTCH_INC (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/Scotch] Verify module SCOTCH_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/Scotch] Verify availability of gout binary (gnu15)
Outcome:Passed
Duration:0.029 sec

Test Suite: rm_execution

Results

Duration9.941 sec
Tests4
Failures0

Tests

rm_execution

Test case:[libs/SLEPc] F90 SVD test binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.22 sec
Test case:[libs/SLEPc] C SVD of the Lauchli matrix binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.17 sec
Test case:[libs/SLEPc] F90 quadratic eigensystem with PEP object binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.326 sec
Test case:[libs/SLEPc] C nonsymmetric eignenproblem binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.225 sec

Test Suite: test_module

Results

Duration0.247 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/slepc] Verify slepc module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.197 sec
Test case:[libs/slepc] Verify module SLEPC_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/slepc] Verify module SLEPC_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/slepc] Verify dynamic library available in SLEPC_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/slepc] Verify static library is not present in SLEPC_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/slepc] Verify module SLEPC_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/slepc] Verify header file is present in SLEPC_INC (gnu15/mpich)
Outcome:Passed
Duration:0.007 sec

Test Suite: rm_execution

Results

Duration8.99 sec
Tests4
Failures4

Tests

rm_execution

Test case:[libs/SLEPc] F90 SVD test binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.251 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 23)
  `run_mpi_binary ./test4f "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.27867
Batch job 277 submitted

Job 277 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./test4f  (family=openmpi5)
[c1:34375] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14744] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14743] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:34375] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14743] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14744] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 0: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
[2026-03-10T17:52:57.552] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.277.0:0]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:52:57.560] error: *** STEP 277.0 ON c1 CANCELLED AT 2026-03-10T17:52:57 DUE to SIGNAL Killed ***
srun: error: c1: task 1: Killed
Test case:[libs/SLEPc] C SVD of the Lauchli matrix binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.298 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 32)
  `run_mpi_binary ./ex15 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.23315
Batch job 278 submitted

Job 278 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex15  (family=openmpi5)
[c1:34557] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14830] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14831] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:34557] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14830] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14831] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 0: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
[2026-03-10T17:53:00.519] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.278.0:0]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:53:00.527] error: *** STEP 278.0 ON c1 CANCELLED AT 2026-03-10T17:53:00 DUE to SIGNAL Killed ***
srun: error: c1: task 1: Killed
Test case:[libs/SLEPc] F90 quadratic eigensystem with PEP object binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:1.195 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 41)
  `run_mpi_binary ./ex16f90 " -pep_nev 4 -terse " 1 1' failed
job script = /tmp/job.ohpc-test.28331
Batch job 279 submitted

Job 279 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex16f90 -pep_nev 4 -terse (family=openmpi5)
[c1:34738] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:34738] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 0: Exited with exit code 14
Test case:[libs/SLEPc] C nonsymmetric eignenproblem binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.246 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 50)
  `run_mpi_binary ./ex29 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.14677
Batch job 280 submitted

Job 280 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./ex29  (family=openmpi5)
[c2:14917] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:14918] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:34913] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:34914] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14917] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:14918] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:34914] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:34913] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14

Test Suite: test_module

Results

Duration0.257 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/slepc] Verify slepc module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.206 sec
Test case:[libs/slepc] Verify module SLEPC_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.011 sec
Test case:[libs/slepc] Verify module SLEPC_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/slepc] Verify dynamic library available in SLEPC_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.007 sec
Test case:[libs/slepc] Verify static library is not present in SLEPC_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.007 sec
Test case:[libs/slepc] Verify module SLEPC_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/slepc] Verify header file is present in SLEPC_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec

Test Suite: rm_execution

Results

Duration0.315 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/SuperLU] C test runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.157 sec
Test case:[libs/SuperLU] F77 test runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.158 sec

Test Suite: test_module

Results

Duration0.317 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/SuperLU] Verify SUPERLU module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.194 sec
Test case:[libs/SuperLU] Verify SUPERLU_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/SuperLU] Verify module SUPERLU_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/SuperLU] Verify dynamic library available in SUPERLU_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/SuperLU] Verify static library is not present in SUPERLU_LIB (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/SuperLU] Verify module SUPERLU_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/SuperLU] Verify header file is present in SUPERLU_INC (gnu15)
Outcome:Passed
Duration:0.022 sec

Test Suite: rm_execution

Results

Duration43.357 sec
Tests9
Failures9

Tests

rm_execution

Test case:[libs/SuperLU_DIST] PDGSSVX with full (default) options (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.247 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 39)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.3095
Batch job 292 submitted

Job 292 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pddrive -r 2 -c 2 g20.rua (family=openmpi5)
[c2:15886] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:15887] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:36805] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:36804] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:15886] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:15887] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:36804] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:36805] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/SuperLU_DIST] pdgssvx_ABglobal with full (default) options (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.244 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 48)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.22901
Batch job 293 submitted

Job 293 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pddrive_ABglobal -r 1 -c 1 g20.rua (family=openmpi5)
[c1:36982] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:36981] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:15972] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:15971] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:36981] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:36982] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:15971] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:15972] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/SuperLU_DIST] vary RHS (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.291 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 57)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive1 "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.7154
Batch job 294 submitted

Job 294 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pddrive1 -r 2 -c 2 g20.rua (family=openmpi5)
[c2:16057] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16056] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:37161] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:37162] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16056] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16057] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37161] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37162] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/SuperLU_DIST] vary RHS ABglobal (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.351 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 66)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive1_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.6558
Batch job 295 submitted

Job 295 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pddrive1_ABglobal -r 1 -c 1 g20.rua (family=openmpi5)
[c1:37338] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:37339] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16142] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16141] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37338] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37339] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16141] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16142] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/SuperLU_DIST] reuse permutation vector (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.354 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 75)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive2 "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.24911
Batch job 296 submitted

Job 296 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pddrive2 -r 2 -c 2 g20.rua (family=openmpi5)
[c2:16227] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:37515] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:37516] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16226] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16226] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16227] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37515] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37516] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/SuperLU_DIST] reuse permutation vector ABglobal (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:5.397 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 84)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive2_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.20213
Batch job 297 submitted

Job 297 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pddrive2_ABglobal -r 1 -c 1 g20.rua (family=openmpi5)
[c1:37692] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:37693] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16312] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16311] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37693] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37692] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16312] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16311] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/SuperLU_DIST] reuse symbolic factorization (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:6.458 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 93)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive3 "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.29998
Batch job 298 submitted

Job 298 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pddrive3 -r 2 -c 2 g20.rua (family=openmpi5)
[c1:37870] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:37869] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16396] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16397] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16396] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37870] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16397] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:37869] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 2-3: Exited with exit code 14
srun: error: c1: tasks 0-1: Exited with exit code 14
Test case:[libs/SuperLU_DIST] reuse symbolic factorization ABglobal (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:6.456 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 102)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive3_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.25766
Batch job 299 submitted

Job 299 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pddrive3_ABglobal -r 1 -c 1 g20.rua (family=openmpi5)
[c1:38047] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:38046] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16481] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16482] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:38046] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:38047] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16482] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16481] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/SuperLU_DIST] multi-grid (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:8.559 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 115)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive4 "g20.rua" "${NODES}" 10' failed
job script = /tmp/job.ohpc-test.25092
Batch job 300 submitted

Job 300 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./pddrive4 g20.rua (family=openmpi5)
[c1:38225] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:38226] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:38224] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:38223] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16570] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16566] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16567] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16569] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:16568] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:38225] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:38224] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:38223] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:38226] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16568] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16570] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16569] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16566] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:16567] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-3: Exited with exit code 14
srun: error: c2: tasks 5-9: Exited with exit code 14
[2026-03-10T17:54:01.856] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.300.0:2]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:54:01.864] error: *** STEP 300.0 ON c1 CANCELLED AT 2026-03-10T17:54:01 DUE to SIGNAL Killed ***
srun: error: c1: task 4: Killed

Test Suite: test_module

Results

Duration0.342 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.215 sec
Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST_DIR is defined and directory exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/SuperLU_DIST] Verify dynamic library available in SUPERLU_DIST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/SuperLU_DIST] Verify static library is not present in SUPERLU_DIST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/SuperLU_DIST] Verify header file is present in SUPERLU_DIST_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec

Test Suite: rm_execution

Results

Duration29.472 sec
Tests9
Failures0

Tests

rm_execution

Test case:[libs/SuperLU_DIST] PDGSSVX with full (default) options (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.172 sec
Test case:[libs/SuperLU_DIST] pdgssvx_ABglobal with full (default) options (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.228 sec
Test case:[libs/SuperLU_DIST] vary RHS (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.224 sec
Test case:[libs/SuperLU_DIST] vary RHS ABglobal (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.274 sec
Test case:[libs/SuperLU_DIST] reuse permutation vector (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.275 sec
Test case:[libs/SuperLU_DIST] reuse permutation vector ABglobal (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.27 sec
Test case:[libs/SuperLU_DIST] reuse symbolic factorization (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.324 sec
Test case:[libs/SuperLU_DIST] reuse symbolic factorization ABglobal (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.323 sec
Test case:[libs/SuperLU_DIST] multi-grid (slurm/gnu15/mpich)
Outcome:Passed
Duration:5.382 sec

Test Suite: test_module

Results

Duration0.351 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.226 sec
Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST_DIR is defined and directory exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/SuperLU_DIST] Verify dynamic library available in SUPERLU_DIST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/SuperLU_DIST] Verify static library is not present in SUPERLU_DIST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/SuperLU_DIST] Verify header file is present in SUPERLU_DIST_INC (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration18.604 sec
Tests6
Failures0

Tests

rm_execution

Test case:[libs/Trilinos] Kokkos-MemorySpace runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.226 sec
Test case:[libs/Trilinos] Tpetra-InitMPI runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.226 sec
Test case:[libs/Trilinos] Tpetra-DataRedistribution runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.274 sec
Test case:[libs/Trilinos] Teuchos-ParameterList runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.273 sec
Test case:[libs/Trilinos] Teuchos-LAPACK runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.276 sec
Test case:[libs/Trilinos] Teuchos-BLAS runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.329 sec

Test Suite: rm_execution

Results

Duration23.872 sec
Tests6
Failures2

Tests

rm_execution

Test case:[libs/Trilinos] Kokkos-MemorySpace runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.215 sec
Test case:[libs/Trilinos] Tpetra-InitMPI runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.241 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 39)
  `run_mpi_binary -t "${TIMEOUT}" "${EXE}" "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.15271
Batch job 308 submitted

Job 308 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./lesson_tpetra_init null (family=openmpi5)
[c1:39669] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:39670] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:17277] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:17276] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:39669] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:39670] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:17276] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:17277] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[libs/Trilinos] Tpetra-DataRedistribution runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.345 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 49)
  `run_mpi_binary -t "${TIMEOUT}" "${EXE}" "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.15369
Batch job 309 submitted

Job 309 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./lesson_tpetra_dataredist null (family=openmpi5)
[c1:39847] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:39847] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[c2:17361] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:17362] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:17361] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:17362] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
[2026-03-10T17:55:10.059] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.309.0:1]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:55:10.067] error: *** STEP 309.0 ON c1 CANCELLED AT 2026-03-10T17:55:10 DUE to SIGNAL Killed ***
srun: error: c1: task 0: Killed
Test case:[libs/Trilinos] Teuchos-ParameterList runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.323 sec
Test case:[libs/Trilinos] Teuchos-LAPACK runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:5.377 sec
Test case:[libs/Trilinos] Teuchos-BLAS runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:5.371 sec

Test Suite: lmod_installed

Results

Duration0.036 sec
Tests1
Failures0

Tests

lmod_installed

Test case:[modules] Check if lmod RPM installed
Outcome:Passed
Duration:0.036 sec

Test Suite: interactive_commands

Results

Duration5.179 sec
Tests8
Failures0

Tests

interactive_commands

Test case:[modules] module purge
Outcome:Passed
Duration:0.137 sec
Test case:[modules] module list
Outcome:Passed
Duration:0.251 sec
Test case:[modules] module help
Outcome:Passed
Duration:0.259 sec
Test case:[modules] module load/unload
Outcome:Passed
Duration:2.382 sec
Test case:[modules] module whatis
Outcome:Passed
Duration:0.257 sec
Test case:[modules] module swap
Outcome:Passed
Duration:0.552 sec
Test case:[modules] path updated
Outcome:Passed
Duration:0.791 sec
Test case:[modules] module depends-on
Outcome:Passed
Duration:0.55 sec

Test Suite: rm_execution

Results

Duration1.946 sec
Tests4
Failures0

Tests

rm_execution

Test case:[modules] env variable passes through ()
Outcome:Passed
Duration:0.404 sec
Test case:[modules] loaded module passes through ()
Outcome:Passed
Duration:0.57 sec
Test case:[modules] module commands available in RMS job ()
Outcome:Passed
Duration:0.578 sec
Test case:[modules] module load propagates thru RMS ()
Outcome:Passed
Duration:0.394 sec

Test Suite: build

Results

Duration1.018 sec
Tests3
Failures0

Tests

build

Test case:[MPI] build/execute C binary (gnu15/mpich)
Outcome:Passed
Duration:0.305 sec
Test case:[MPI] build/execute C++ binary (gnu15/mpich)
Outcome:Passed
Duration:0.437 sec
Test case:[MPI] build/execute F90 binary (gnu15/mpich)
Outcome:Passed
Duration:0.276 sec

Test Suite: man_page_check

Results

Duration0.091 sec
Tests1
Failures0

Tests

man_page_check

Test case:[MPI] mpicc man page availible (gnu15/mpich)
Outcome:Passed
Duration:0.091 sec

Test Suite: rm_execution_multi_host

Results

Duration6.476 sec
Tests3
Failures0

Tests

rm_execution_multi_host

Test case:[MPI] C binary runs on two nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.158 sec
Test case:[MPI] C++ binary runs on two nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.158 sec
Test case:[MPI] F90 binary runs on two nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.16 sec

Test Suite: rm_execution_single_host

Results

Duration6.479 sec
Tests3
Failures0

Tests

rm_execution_single_host

Test case:[MPI] C binary runs on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.123 sec
Test case:[MPI] C++ binary runs on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.197 sec
Test case:[MPI] F90 binary runs on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.159 sec

Test Suite: version_match

Results

Duration0.446 sec
Tests3
Failures0

Tests

version_match

Test case:[MPI] MPI module loaded (gnu15/mpich)
Outcome:Passed
Duration:0.146 sec
Test case:[MPI] MPI module version available (gnu15/mpich)
Outcome:Passed
Duration:0.127 sec
Test case:[MPI] mpicc, mpicxx, and mpif90 versions match module (gnu15/mpich)
Outcome:Passed
Duration:0.173 sec

Test Suite: build

Results

Duration2.671 sec
Tests3
Failures0

Tests

build

Test case:[MPI] build/execute C binary (gnu15/openmpi5)
Outcome:Passed
Duration:1.296 sec
Test case:[MPI] build/execute C++ binary (gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
C++ not available with openmpi5
Test case:[MPI] build/execute F90 binary (gnu15/openmpi5)
Outcome:Passed
Duration:1.375 sec

Test Suite: man_page_check

Results

Duration0.086 sec
Tests1
Failures0

Tests

man_page_check

Test case:[MPI] mpicc man page availible (gnu15/openmpi5)
Outcome:Passed
Duration:0.086 sec

Test Suite: rm_execution_multi_host

Results

Duration5.807 sec
Tests3
Failures2

Tests

rm_execution_multi_host

Test case:[MPI] C binary runs on two nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.387 sec
Failed
(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file rm_execution_multi_host, line 24)
  `assert_success' failed

-- command failed --
status : 1
output (59 lines):
  job script = /tmp/job.ohpc-test.28548
  Batch job 321 submitted

  Job 321 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = srun --mpi=pmix -- ./C_test 8 (family=openmpi5)
  [c1:41992] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c1:41993] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c1:41994] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c1:41995] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c2:17988] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c2:17986] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c2:17985] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c2:17987] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:41992] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:41995] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:41993] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:41994] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c2:17988] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c2:17986] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c2:17987] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c2:17985] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  srun: error: c1: tasks 0-3: Exited with exit code 14
  srun: error: c2: tasks 4-7: Exited with exit code 14
--
Test case:[MPI] C++ binary runs on two nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
C++ not available with openmpi5
Test case:[MPI] F90 binary runs on two nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.42 sec
Failed
(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file rm_execution_multi_host, line 45)
  `assert_success' failed

-- command failed --
status : 1
output (50 lines):
  job script = /tmp/job.ohpc-test.10310
  Batch job 322 submitted

  Job 322 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = srun --mpi=pmix -- ./F90_test 8 (family=openmpi5)
  [c1:42177] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c1:42176] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c1:42179] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c1:42178] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c2:18074] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c2:18076] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:42178] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:42177] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c2:18074] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c2:18076] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:42176] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:42179] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  srun: error: c2: tasks 4,6: Exited with exit code 14
  srun: error: c1: tasks 0-3: Exited with exit code 14
  [2026-03-10T17:55:58.487] error:  mpi/pmix_v4: _errhandler: c2 [1]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.322.0:4]
  srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
  srun: error: c2: tasks 5,7: Killed
--

Test Suite: rm_execution_single_host

Results

Duration4.654 sec
Tests3
Failures2

Tests

rm_execution_single_host

Test case:[MPI] C binary runs on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.321 sec
Failed
(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file rm_execution_single_host, line 24)
  `assert_success' failed

-- command failed --
status : 1
output (22 lines):
  job script = /tmp/job.ohpc-test.13408
  Batch job 319 submitted

  Job 319 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = srun --mpi=pmix -- ./C_test 2 (family=openmpi5)
  [c1:41639] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c1:41640] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:41639] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:41640] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  srun: error: c1: tasks 0-1: Exited with exit code 14
--
Test case:[MPI] C++ binary runs on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
C++ not available with openmpi5
Test case:[MPI] F90 binary runs on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.333 sec
Failed
(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file rm_execution_single_host, line 45)
  `assert_success' failed

-- command failed --
status : 1
output (22 lines):
  job script = /tmp/job.ohpc-test.20588
  Batch job 320 submitted

  Job 320 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = srun --mpi=pmix -- ./F90_test 2 (family=openmpi5)
  [c1:41816] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c1:41815] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:41815] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:41816] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  srun: error: c1: tasks 0-1: Exited with exit code 14
--

Test Suite: version_match

Results

Duration0.468 sec
Tests3
Failures0

Tests

version_match

Test case:[MPI] MPI module loaded (gnu15/openmpi5)
Outcome:Passed
Duration:0.126 sec
Test case:[MPI] MPI module version available (gnu15/openmpi5)
Outcome:Passed
Duration:0.131 sec
Test case:[MPI] mpicc, mpicxx, and mpif90 versions match module (gnu15/openmpi5)
Outcome:Passed
Duration:0.211 sec

Test Suite: conman

Results

Duration0.131 sec
Tests3
Failures0

Tests

conman

Test case:[ConMan] Verify conman binary available
Outcome:Passed
Duration:0.025 sec
Test case:[ConMan] Verify rpm version matches binary
Outcome:Passed
Duration:0.061 sec
Test case:[ConMan] Verify man page availability
Outcome:Passed
Duration:0.045 sec

Test Suite: warewulf-ipmi

Results

Duration0.0 sec
Tests1
Failures0

Tests

warewulf-ipmi

Test case:[warewulf-ipmi] ipmitool lanplus protocol
Outcome:Skipped
Duration:0.0 sec
Skipped
Warewulf not installed, skipping warewulf-ipmitool check

Test Suite: test_module

Results

Duration1.868 sec
Tests6
Failures0

Tests

test_module

Test case:[perf-tools/Dimemas] Verify dimemas module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.191 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[perf-tools/Dimemas] Verify availability of prv2dim binary (gnu15/mpich)
Outcome:Passed
Duration:0.028 sec
Test case:[perf-tools/Dimemas] Verify availability of Dimemas binary (gnu15/mpich)
Outcome:Passed
Duration:0.026 sec
Test case:[perf-tools/Dimemas] Run Dimemas simulation (gnu15/mpich)
Outcome:Passed
Duration:1.585 sec

Test Suite: test_module

Results

Duration1.888 sec
Tests6
Failures0

Tests

test_module

Test case:[perf-tools/Dimemas] Verify dimemas module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.204 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[perf-tools/Dimemas] Verify availability of prv2dim binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.027 sec
Test case:[perf-tools/Dimemas] Verify availability of Dimemas binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.029 sec
Test case:[perf-tools/Dimemas] Run Dimemas simulation (gnu15/openmpi5)
Outcome:Passed
Duration:1.588 sec

Test Suite: test_imb_mpi1

Results

Duration2.22 sec
Tests1
Failures0

Tests

test_imb_mpi1

Test case:[Libs/IMB] run IMB-MPI1 on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.22 sec

Test Suite: test_imb_mpi2

Results

Duration5.491 sec
Tests2
Failures0

Tests

test_imb_mpi2

Test case:[Libs/IMB] run IMB-EXT on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.22 sec
Test case:[Libs/IMB] run IMB-IO on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.271 sec

Test Suite: test_imb_mpi3

Results

Duration2.221 sec
Tests2
Failures0

Tests

test_imb_mpi3

Test case:[Libs/IMB] run IMB-NBC on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[Libs/IMB] run IMB-RMA on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.221 sec

Test Suite: test_module

Results

Duration0.202 sec
Tests3
Failures0

Tests

test_module

Test case:[IMB] Verify IMB module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.179 sec
Test case:[IMB] Verify IMB_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[IMB] Verify executables are present in IMB_DIR/bin (gnu15-mpich)
Outcome:Passed
Duration:0.014 sec

Test Suite: test_imb_mpi1

Results

Duration1.192 sec
Tests1
Failures1

Tests

test_imb_mpi1

Test case:[Libs/IMB] run IMB-MPI1 on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:1.192 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_imb_mpi1, line 38)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.3278
Batch job 327 submitted

Job 327 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- /opt/ohpc/pub/libs/gnu15/openmpi5/imb/2021.3/bin/IMB-MPI1 -off_cache -1 -time 1.5 -npmin 100 -msglog 1:4 PingPong Sendrecv Bcast Allreduce (family=openmpi5)
[c1:43101] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43104] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43098] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43103] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43100] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43102] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43099] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43097] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18544] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18545] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18543] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18546] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18539] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18541] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18540] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18542] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43101] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43103] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43100] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43102] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43104] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[c1:43098] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43099] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43097] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18543] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18544] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18540] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18546] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18541] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18545] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18539] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18542] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-7: Exited with exit code 14
srun: error: c2: tasks 8-15: Exited with exit code 14

Test Suite: test_imb_mpi2

Results

Duration4.491 sec
Tests2
Failures2

Tests

test_imb_mpi2

Test case:[Libs/IMB] run IMB-EXT on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:1.195 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_imb_mpi2, line 41)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.10224
Batch job 328 submitted

Job 328 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- /opt/ohpc/pub/libs/gnu15/openmpi5/imb/2021.3/bin/IMB-EXT -npmin 100 -msglog 1:4 Window (family=openmpi5)
[c1:43286] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43287] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18637] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18636] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18637] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18636] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43286] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43287] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14
Test case:[Libs/IMB] run IMB-IO on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.296 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_imb_mpi2, line 53)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.29783
Batch job 329 submitted

Job 329 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- /opt/ohpc/pub/libs/gnu15/openmpi5/imb/2021.3/bin/IMB-IO -npmin 100 Open_Close (family=openmpi5)
[c1:43463] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43464] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18722] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18721] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43463] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43464] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18721] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18722] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1: Exited with exit code 14
srun: error: c2: tasks 2-3: Exited with exit code 14

Test Suite: test_imb_mpi3

Results

Duration3.298 sec
Tests2
Failures1

Tests

test_imb_mpi3

Test case:[Libs/IMB] run IMB-NBC on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[Libs/IMB] run IMB-RMA on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:3.298 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_imb_mpi3, line 56)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.30513
Batch job 330 submitted

Job 330 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- /opt/ohpc/pub/libs/gnu15/openmpi5/imb/2021.3/bin/IMB-RMA -npmin 100 -msglog 1:4 Unidir_put Unidir_get (family=openmpi5)
[c2:18806] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18807] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18808] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:18809] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43643] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43641] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43640] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:43642] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18806] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18807] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18808] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:18809] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43641] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43643] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43642] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:43640] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 4-7: Exited with exit code 14
srun: error: c1: tasks 0-3: Exited with exit code 14

Test Suite: test_module

Results

Duration0.212 sec
Tests3
Failures0

Tests

test_module

Test case:[IMB] Verify IMB module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.188 sec
Test case:[IMB] Verify IMB_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[IMB] Verify executables are present in IMB_DIR/bin (gnu15-openmpi5)
Outcome:Passed
Duration:0.014 sec

Test Suite: rm_execution

Results

Duration11.799 sec
Tests2
Failures0

Tests

rm_execution

Test case:[OMB] run osu_bw on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.22 sec
Test case:[OMB] run osu_latency on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:9.579 sec

Test Suite: test_module

Results

Duration0.366 sec
Tests7
Failures0

Tests

test_module

Test case:[OMB] Verify OMB module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.19 sec
Test case:[OMB] Verify OMB_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[OMB] Verify osu_bw binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[OMB] Verify osu_latency binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.032 sec
Test case:[OMB] Verify osu_allgather binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.033 sec
Test case:[OMB] Verify osu_get_bw binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[OMB] Verify osu_put_latency binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.03 sec

Test Suite: rm_execution

Results

Duration3.442 sec
Tests2
Failures2

Tests

rm_execution

Test case:[OMB] run osu_bw on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:1.194 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 23)
  `run_mpi_binary -t "$TIMEOUT" "$EXE" "-m 512" 2 2' failed
job script = /tmp/job.ohpc-test.26311
Batch job 333 submitted

Job 333 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- osu_bw -m 512 (family=openmpi5)
[c1:44173] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:44173] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 0: Exited with exit code 14
[c2:19060] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:19060] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: task 1: Exited with exit code 14
Test case:[OMB] run osu_latency on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.248 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 36)
  `run_mpi_binary -t "$TIMEOUT" "$EXE" "-m $MESSAGE_SIZE" 2 2' failed
job script = /tmp/job.ohpc-test.31201
Batch job 334 submitted

Job 334 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- osu_latency -m 0 (family=openmpi5)
[c2:19143] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:44348] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:19143] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: task 1: Exited with exit code 14
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:44348] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: task 0: Exited with exit code 14

Test Suite: test_module

Results

Duration0.381 sec
Tests7
Failures0

Tests

test_module

Test case:[OMB] Verify OMB module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.202 sec
Test case:[OMB] Verify OMB_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[OMB] Verify osu_bw binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.031 sec
Test case:[OMB] Verify osu_latency binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[OMB] Verify osu_allgather binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.03 sec
Test case:[OMB] Verify osu_get_bw binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.033 sec
Test case:[OMB] Verify osu_put_latency binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.032 sec

Test Suite: rm_execution

Results

Duration15.246 sec
Tests3
Failures0

Tests

rm_execution

Test case:[perf-tools/Scalasca] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:9.12 sec
Test case:[perf-tools/Scalasca] MPI C++ binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.85 sec
Test case:[perf-tools/Scalasca] Serial C OpenMP binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.276 sec

Test Suite: test_module

Results

Duration0.357 sec
Tests11
Failures0

Tests

test_module

Test case:[perf-tools/Scalasca] Verify scalasca module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.196 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[perf-tools/Scalasca] Verify availability of scalasca binary (gnu15/mpich)
Outcome:Passed
Duration:0.023 sec
Test case:[perf-tools/Scalasca] Verify availability of scan binary (gnu15/mpich)
Outcome:Passed
Duration:0.016 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.mpi binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.omp binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.hyb binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.ser binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of square binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of user guide for scalasca (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec

Test Suite: rm_execution

Results

Duration8.925 sec
Tests3
Failures2

Tests

rm_execution

Test case:[perf-tools/Scalasca] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:4.356 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 24)
  `run_mpi_binary -s 1 ./mpi/C_mpi_test $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.14923
Batch job 357 submitted

Job 357 failed...
Reason=NonZeroExitCode

S=C=A=N: Scalasca 2.6.2 trace collection and analysis
S=C=A=N: ./scorep_C_mpi_test_8x2_trace experiment archive
S=C=A=N: Tue Mar 10 17:59:52 2026: Collect start
/opt/ohpc/pub/utils/prun/2.2/prun ./mpi/C_mpi_test 8
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./mpi/C_mpi_test 8 (family=openmpi5)
[c1:48069] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:48070] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:48071] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:48068] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:21067] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:21069] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:21066] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:21068] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:48069] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:48068] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:48070] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:48071] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:21067] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:21068] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:21069] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:21066] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14
S=C=A=N: Tue Mar 10 17:59:55 2026: Collect done (status=0) 3s
Abort: missing experiment archive ./scorep_C_mpi_test_8x2_trace
Test case:[perf-tools/Scalasca] MPI C++ binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:2.249 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 38)
  `run_mpi_binary -s 1 ./mpi/CXX_mpi_test $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.18436
Batch job 358 submitted

Job 358 failed...
Reason=NonZeroExitCode

S=C=A=N: Scalasca 2.6.2 trace collection and analysis
S=C=A=N: ./scorep_CXX_mpi_test_8x2_trace experiment archive
S=C=A=N: Tue Mar 10 17:59:56 2026: Collect start
/opt/ohpc/pub/utils/prun/2.2/prun ./mpi/CXX_mpi_test 8
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./mpi/CXX_mpi_test 8 (family=openmpi5)
[c2:21158] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:21155] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:21156] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:21157] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:48375] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:48373] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:48374] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:48376] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:21158] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:21156] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:21155] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:21157] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:48375] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:48374] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:48373] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:48376] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c2: tasks 4-7: Exited with exit code 14
srun: error: c1: tasks 0-3: Exited with exit code 14
S=C=A=N: Tue Mar 10 17:59:57 2026: Collect done (status=0) 1s
Abort: missing experiment archive ./scorep_CXX_mpi_test_8x2_trace
Test case:[perf-tools/Scalasca] Serial C OpenMP binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.32 sec

Test Suite: test_module

Results

Duration0.359 sec
Tests11
Failures0

Tests

test_module

Test case:[perf-tools/Scalasca] Verify scalasca module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.206 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[perf-tools/Scalasca] Verify availability of scalasca binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scan binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.016 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.mpi binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.omp binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.016 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.hyb binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.ser binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of square binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[perf-tools/Scalasca] Verify availability of user guide for scalasca (gnu15/openmpi5)
Outcome:Passed
Duration:0.017 sec

Test Suite: instrumenter_test

Results

Duration2.301 sec
Tests9
Failures0

Tests

instrumenter_test

Test case:[perf-tools/Score-P] MPI C binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.187 sec
Test case:[perf-tools/Score-P] MPI C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.18 sec
Test case:[perf-tools/Score-P] MPI Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.178 sec
Test case:[perf-tools/Score-P] Serial C OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.201 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.203 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.203 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.381 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.38 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.388 sec

Test Suite: rm_execution

Results

Duration70.757 sec
Tests9
Failures0

Tests

rm_execution

Test case:[perf-tools/Score-P] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.358 sec
Test case:[perf-tools/Score-P] MPI C++ binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:7.68 sec
Test case:[perf-tools/Score-P] MPI Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:8.078 sec
Test case:[perf-tools/Score-P] Serial C OpenMP binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:7.527 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:7.492 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:7.652 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:8.752 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:9.794 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:12.424 sec

Test Suite: test_module

Results

Duration0.697 sec
Tests14
Failures0

Tests

test_module

Test case:[perf-tools/Score-P] Verify scorep module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.204 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[perf-tools/Score-P] Verify availability of scorep binary (gnu15/mpich)
Outcome:Passed
Duration:0.03 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-backend-info binary (gnu15/mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-config binary (gnu15/mpich)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-info binary (gnu15/mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-preload-init binary (gnu15/mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-score binary (gnu15/mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-wrapper binary (gnu15/mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep wrapper binaries (gnu15/mpich)
Outcome:Passed
Duration:0.131 sec
Test case:[perf-tools/Score-P] Verify availability of user guide for scorep (gnu15/mpich)
Outcome:Passed
Duration:0.034 sec
Test case:[perf-tools/Score-P] Verify availability of OPEN_ISSUES for scorep (gnu15/mpich)
Outcome:Passed
Duration:0.035 sec
Test case:[perf-tools/Score-P] Verify availability of COPYING for scorep (gnu15/mpich)
Outcome:Passed
Duration:0.034 sec

Test Suite: instrumenter_test

Results

Duration2.389 sec
Tests9
Failures0

Tests

instrumenter_test

Test case:[perf-tools/Score-P] MPI C binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.198 sec
Test case:[perf-tools/Score-P] MPI C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.185 sec
Test case:[perf-tools/Score-P] MPI Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.199 sec
Test case:[perf-tools/Score-P] Serial C OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.197 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.203 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.204 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.389 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.403 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.411 sec

Test Suite: rm_execution

Results

Duration99.134 sec
Tests9
Failures6

Tests

rm_execution

Test case:[perf-tools/Score-P] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:8.577 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 59)
  `run_mpi_binary ./mpi/main_mpi_c "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.10445
Batch job 344 submitted

Job 344 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./mpi/main_mpi_c 8 (family=openmpi5)
[c1:45812] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:45813] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:45813] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:45812] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 2-3: Exited with exit code 14
[2026-03-10T17:58:37.662] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.344.0:3]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:58:37.670] error: *** STEP 344.0 ON c1 CANCELLED AT 2026-03-10T17:58:37 DUE to SIGNAL Killed ***
srun: error: c2: tasks 4,6-7: Killed
srun: error: c1: tasks 0-1: Killed
srun: error: c2: task 5: Killed
Test case:[perf-tools/Score-P] MPI C++ binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:8.577 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 73)
  `run_mpi_binary ./mpi/main_mpi_cxx "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.1714
Batch job 345 submitted

Job 345 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./mpi/main_mpi_cxx 8 (family=openmpi5)
[c1:45996] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:45998] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:45997] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:45999] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:19972] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:19970] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:19971] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:19973] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:45996] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:45998] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:45999] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:45997] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:19972] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:19973] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:19970] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:19971] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14
Test case:[perf-tools/Score-P] MPI Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:10.685 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 87)
  `run_mpi_binary ./mpi/main_mpi_fort "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.28973
Batch job 346 submitted

Job 346 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./mpi/main_mpi_fort 8 (family=openmpi5)
[c1:46180] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46177] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46178] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20063] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20060] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20062] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20061] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46180] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46178] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46177] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20063] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20061] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20060] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20062] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1,3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14
[2026-03-10T17:58:45.224] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.346.0:3]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:58:45.232] error: *** STEP 346.0 ON c1 CANCELLED AT 2026-03-10T17:58:45 DUE to SIGNAL Killed ***
srun: error: c1: task 2: Killed
Test case:[perf-tools/Score-P] Serial C OpenMP binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:10.278 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:10.186 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:10.354 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:12.788 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 143)
  `run_mpi_binary ./mpi/main_hyb_c "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.32666
Batch job 350 submitted

Job 350 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./mpi/main_hyb_c 8 (family=openmpi5)
[c2:20311] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46446] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20312] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46444] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20313] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46443] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20314] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46446] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20311] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46443] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20314] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20313] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46444] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20312] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-1,3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14
[2026-03-10T17:58:47.439] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.350.0:3]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-03-10T17:58:47.447] error: *** STEP 350.0 ON c1 CANCELLED AT 2026-03-10T17:58:47 DUE to SIGNAL Killed ***
srun: error: c1: task 2: Killed
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:12.789 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 157)
  `run_mpi_binary ./mpi/main_hyb_cxx "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.19972
Batch job 351 submitted

Job 351 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./mpi/main_hyb_cxx 8 (family=openmpi5)
[c1:46631] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46629] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46630] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46632] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20405] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20402] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20403] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20404] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46631] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46632] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46629] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46630] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20405] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20404] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20403] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20402] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0-3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:14.9 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 171)
  `run_mpi_binary ./mpi/main_hyb_fort "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.7642
Batch job 352 submitted

Job 352 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = srun --mpi=pmix -- ./mpi/main_hyb_fort 8 (family=openmpi5)
[c2:20494] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20492] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20493] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c2:20491] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46810] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
[c1:46813] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46813] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20492] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20494] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c1:46810] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20491] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[c2:20493] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: c1: tasks 0,3: Exited with exit code 14
srun: error: c2: tasks 4-7: Exited with exit code 14
[2026-03-10T17:58:49.869] error:  mpi/pmix_v4: _errhandler: c1 [0]: pmixp_client_v2.c:219: Error handler invoked: status = -61, source = [slurm.pmix.352.0:3]
[2026-03-10T17:58:49.877] error: *** STEP 352.0 ON c1 CANCELLED AT 2026-03-10T17:58:49 DUE to SIGNAL Killed ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: c1: tasks 1-2: Killed

Test Suite: test_module

Results

Duration0.764 sec
Tests14
Failures0

Tests

test_module

Test case:[perf-tools/Score-P] Verify scorep module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.23 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[perf-tools/Score-P] Verify availability of scorep binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-backend-info binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-config binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-info binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.028 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-preload-init binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.033 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-score binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-wrapper binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.033 sec
Test case:[perf-tools/Score-P] Verify availability of scorep wrapper binaries (gnu15/openmpi5)
Outcome:Passed
Duration:0.168 sec
Test case:[perf-tools/Score-P] Verify availability of user guide for scorep (gnu15/openmpi5)
Outcome:Passed
Duration:0.034 sec
Test case:[perf-tools/Score-P] Verify availability of OPEN_ISSUES for scorep (gnu15/openmpi5)
Outcome:Passed
Duration:0.035 sec
Test case:[perf-tools/Score-P] Verify availability of COPYING for scorep (gnu15/openmpi5)
Outcome:Passed
Duration:0.034 sec

Test Suite: test_harness

Results

Duration3.302 sec
Tests3
Failures0

Tests

test_harness

Test case:[rms-harness] Verify zero exit code from MPI job runs OK (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.123 sec
Test case:[rms-harness] Verify non-zero exit code from MPI job detected as failure (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.179 sec
Test case:[rms-harness] Verify long-running MPI job terminates with timeout parameter (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
disabled timeout test

Test Suite: test_harness

Results

Duration3.494 sec
Tests3
Failures1

Tests

test_harness

Test case:[rms-harness] Verify zero exit code from MPI job runs OK (slurm/gnu15/openmpi5)
Outcome:Failed
Duration:1.31 sec
Failed
(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file test_harness, line 29)
  `assert_success' failed

-- command failed --
status : 1
output (23 lines):
  job script = /tmp/job.ohpc-test.17074
  Batch job 363 submitted

  Job 363 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = srun --mpi=pmix -- ./mpi_exit 0 (family=openmpi5)
  [c2:21498] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  [c1:48986] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c1:48986] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  *** An error occurred in MPI_Init
  *** on a NULL communicator
  *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
  ***    and MPI will try to terminate your MPI job as well)
  [c2:21498] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
  srun: error: c1: task 0: Exited with exit code 14
  srun: error: c2: task 1: Exited with exit code 14
--
Test case:[rms-harness] Verify non-zero exit code from MPI job detected as failure (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.184 sec
Test case:[rms-harness] Verify long-running MPI job terminates with timeout parameter (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
disabled timeout test

Test Suite: run

Results

Duration3.663 sec
Tests5
Failures0

Tests

run

Test case:[charliecloud] check for RPM
Outcome:Passed
Duration:0.283 sec
Test case:[charliecloud] build image
Outcome:Skipped
Duration:0.0 sec
Skipped
skip experimental charliecloud image builder
Test case:[charliecloud] build alpine image from Docker (using singularity)
Outcome:Passed
Duration:2.635 sec
Test case:[charliecloud] exec image locally
Outcome:Passed
Duration:0.295 sec
Test case:[charliecloud] exec image via slurm
Outcome:Passed
Duration:0.45 sec

Test Suite: run

Results

Duration8.776 sec
Tests4
Failures0

Tests

run

Test case:[singularity] check for RPM
Outcome:Passed
Duration:0.021 sec
Test case:[singularity] pull down ubuntu docker image
Outcome:Passed
Duration:8.012 sec
Test case:[singularity] exec image
Outcome:Passed
Duration:0.323 sec
Test case:[singularity] exec image via slurm
Outcome:Passed
Duration:0.42 sec

Test Suite: completion

Results

Duration0.083 sec
Tests1
Failures0

Tests

completion

Test case:[singularity] check for bash completion
Outcome:Passed
Duration:0.083 sec

Test Suite: ntp

Results

Duration0.361 sec
Tests3
Failures0

Tests

ntp

Test case:[ntp] check for chronyc binary
Outcome:Passed
Duration:0.018 sec
Test case:[ntp] verify local time in sync on SMS
Outcome:Passed
Duration:0.152 sec
Test case:[ntp] verify local time in sync on compute
Outcome:Passed
Duration:0.191 sec

Test Suite: mem_limits

Results

Duration0.309 sec
Tests2
Failures0

Tests

mem_limits

Test case:[memlock] check increased soft limit
Outcome:Passed
Duration:0.164 sec
Test case:[memlock] check increased hard limit
Outcome:Passed
Duration:0.145 sec

Test Suite: pdsh

Results

Duration2.58 sec
Tests4
Failures0

Tests

pdsh

Test case:[pdsh] check for RPM (slurm)
Outcome:Passed
Duration:0.046 sec
Test case:[pdsh] run a shell command on c[1-2] (slurm)
Outcome:Passed
Duration:2.049 sec
Test case:[pdsh] check for pdsh-mod-slurm RPM (slurm)
Outcome:Passed
Duration:0.046 sec
Test case:[pdsh] run a shell command on -P normal (slurm)
Outcome:Passed
Duration:0.439 sec

Test Suite: ompi_info

Results

Duration0.506 sec
Tests1
Failures0

Tests

ompi_info

Test case:[openmpi] check for no output to stderr with ompi_info
Outcome:Passed
Duration:0.506 sec

Test Suite: magpie

Results

Duration0.968 sec
Tests3
Failures0

Tests

magpie

Test case:[magpie] check for RPM
Outcome:Passed
Duration:0.057 sec
Test case:[magpie] Verify MAGPIE module is loaded and matches rpm version
Outcome:Passed
Duration:0.548 sec
Test case:[magpie] Verify module MAGPIE_DIR is defined and exists
Outcome:Passed
Duration:0.363 sec

Test Suite: munge

Results

Duration1.101 sec
Tests4
Failures0

Tests

munge

Test case:[munge] check for OS provdied RPM
Outcome:Passed
Duration:0.043 sec
Test case:[munge] Generate a credential
Outcome:Passed
Duration:0.018 sec
Test case:[munge] Decode credential locally
Outcome:Passed
Duration:0.017 sec
Test case:[munge] Run benchmark
Outcome:Passed
Duration:1.023 sec

Test Suite: sacct

Results

Duration0.0 sec
Tests1
Failures0

Tests

sacct

Test case:[slurm] check for working sacct
Outcome:Skipped
Duration:0.0 sec
Skipped
Temporarily skip sacct check

Test Suite: sinfo

Results

Duration0.077 sec
Tests1
Failures0

Tests

sinfo

Test case:[slurm] Verify SLURM RPM version matches sinfo binary
Outcome:Passed
Duration:0.077 sec

Test Suite: oom

Results

Duration0.0 sec
Tests1
Failures0

Tests

oom

Test case:[oom] Test job OOM condition (gnu15/slurm)
Outcome:Skipped
Duration:0.0 sec
Skipped
disable to invistigate disk fillup

Generated by junit2html