Test Report : results

Test Suite: run

Results

Duration2.147 sec
Tests4
Failures0

Tests

run

Test case:[dev-tools/cmake] running cmake --system-information
Outcome:Passed
Duration:0.893 sec
Test case:[dev-tools/cmake] run cmake
Outcome:Passed
Duration:0.406 sec
Test case:[dev-tools/cmake] run make on generated Makefile
Outcome:Passed
Duration:0.836 sec
Test case:[dev-tools/cmake] run compiled binary
Outcome:Passed
Duration:0.012 sec

Test Suite: computes_installed.bats

Results

Duration2.841 sec
Tests5
Failures0

Tests

computes_installed.bats

Test case:num_computes is greater than zero
Outcome:Passed
Duration:0.02 sec
Test case:koomie_cf command is available
Outcome:Passed
Duration:0.02 sec
Test case:get uptime from compute nodes produces results
Outcome:Passed
Duration:1.106 sec
Test case:correct number of compute hosts respond
Outcome:Passed
Duration:0.59 sec
Test case:verify boot times are reasonable
Outcome:Passed
Duration:1.105 sec

Test Suite: sms_installed.bats

Results

Duration0.029 sec
Tests2
Failures0

Tests

sms_installed.bats

Test case:Verify hostname matches expectations
Outcome:Passed
Duration:0.013 sec
Test case:Base OS check
Outcome:Passed
Duration:0.016 sec

Test Suite: clustershell

Results

Duration1.124 sec
Tests2
Failures0

Tests

clustershell

Test case:[clush] check for OS-provided RPM
Outcome:Passed
Duration:0.047 sec
Test case:[clush] clush -Sg compute
Outcome:Passed
Duration:1.077 sec

Test Suite: conman

Results

Duration0.064 sec
Tests2
Failures0

Tests

conman

Test case:[conman] check for RPM
Outcome:Passed
Duration:0.045 sec
Test case:[conman] query conmand conman
Outcome:Passed
Duration:0.019 sec

Test Suite: genders

Results

Duration0.059 sec
Tests2
Failures0

Tests

genders

Test case:[genders] check for RPM
Outcome:Passed
Duration:0.043 sec
Test case:[genders] check node attributes
Outcome:Passed
Duration:0.016 sec

Test Suite: nhc

Results

Duration4.348 sec
Tests3
Failures0

Tests

nhc

Test case:[nhc] check for RPM
Outcome:Passed
Duration:0.045 sec
Test case:[nhc] generate config file
Outcome:Passed
Duration:1.691 sec
Test case:[nhc] service failure detection and restart
Outcome:Passed
Duration:2.612 sec

Test Suite: slurm-plugins

Results

Duration1.316 sec
Tests4
Failures0

Tests

slurm-plugins

Test case:[slurm] check for jobcomp_elasticsearch plugin
Outcome:Passed
Duration:0.017 sec
Test case:[slurm] check for job_submit_lua plugin
Outcome:Passed
Duration:0.018 sec
Test case:[slurm] check for --x11 option
Outcome:Passed
Duration:0.024 sec
Test case:[slurm] check for sview rpm availability
Outcome:Passed
Duration:1.257 sec

Test Suite: lmod

Results

Duration0.399 sec
Tests1
Failures0

Tests

lmod

Test case:[lmod] test that the setup function passed
Outcome:Passed
Duration:0.399 sec

Test Suite: spack

Results

Duration70.507 sec
Tests4
Failures0

Tests

spack

Test case:[spack] check for RPM
Outcome:Passed
Duration:0.576 sec
Test case:[spack] add compiler
Outcome:Passed
Duration:48.231 sec
Test case:[spack] test build
Outcome:Passed
Duration:20.734 sec
Test case:[spack] test module refresh
Outcome:Passed
Duration:0.966 sec

Test Suite: build

Results

Duration6.491 sec
Tests1
Failures0

Tests

build

Test case:[/apps/hpcg] build HPCG executable (gnu15/mpich)
Outcome:Passed
Duration:6.491 sec

Test Suite: rm_execution_multi_host

Results

Duration18.045 sec
Tests3
Failures0

Tests

rm_execution_multi_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.01 sec
Test case:[/apps/hpcg] run HPCG on multi nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:17.985 sec
Test case:[/apps/hpcg] log HPCG multi node results (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.05 sec

Test Suite: rm_execution_single_host

Results

Duration13.831 sec
Tests3
Failures0

Tests

rm_execution_single_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.009 sec
Test case:[/apps/hpcg] run HPCG on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:13.774 sec
Test case:[/apps/hpcg] log HPCG single node results (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.048 sec

Test Suite: build

Results

Duration6.575 sec
Tests1
Failures0

Tests

build

Test case:[/apps/hpcg] build HPCG executable (gnu15/mvapich2)
Outcome:Passed
Duration:6.575 sec

Test Suite: rm_execution_multi_host

Results

Duration2.281 sec
Tests3
Failures2

Tests

rm_execution_multi_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.011 sec
Test case:[/apps/hpcg] run HPCG on multi nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.245 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution_multi_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.14887
Batch job 14 submitted

Job 14 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./bin/xhpcg.gnu15.mvapich2 32 32 32 10 (family=mvapich2)
[1776234727.429983] [c1:5191 :0]        ib_iface.c:1011 UCX  ERROR iface=0x192345b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234727.430056] [c1:5194 :0]        ib_iface.c:1011 UCX  ERROR iface=0x2d7555b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234727.430108] [c1:5196 :0]        ib_iface.c:1011 UCX  ERROR iface=0xf6ef5b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234727.430315] [c1:5190 :0]        ib_iface.c:1011 UCX  ERROR iface=0x12f6f5b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234727.430321] [c1:5197 :0]        ib_iface.c:1011 UCX  ERROR iface=0x340835b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234727.430433] [c1:5193 :0]        ib_iface.c:1011 UCX  ERROR iface=0x1e7835b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(1010952463) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffcd92535cc, argv=0x7ffcd92535c0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(876734735) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fffe0c0236c, argv=0x7fffe0c02360) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(205646095) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc140a5d2c, argv=0x7ffc140a5d20) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(742517007) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe4cecd88c, argv=0x7ffe4cecd880) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc9d7c24ec, argv=0x7ffc9d7c24e0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(71428367) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd1c0bc42c, argv=0x7ffd1c0bc420) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776234727.435793] [c2:3593 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3eef63a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234727.436186] [c2:3594 :0]        ib_iface.c:1011 UCX  ERROR iface=0x2d3803a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234727.436202] [c2:3591 :0]        ib_iface.c:1011 UCX  ERROR iface=0x194413a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234727.436310] [c2:3588 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3e2033a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234727.436388] [c2:3590 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3667d3a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(742517007) on node 13: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fffaf77876c, argv=0x7fffaf778760) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(608299279) on node 8: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd042ed3ac, argv=0x7ffd042ed3a0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339863823) on node 14: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffcb0af494c, argv=0x7ffcb0af4940) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(1010952463) on node 11: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdef86d36c, argv=0x7ffdef86d360) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(71428367) on node 10: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffeeb9649cc, argv=0x7ffeeb9649c0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776234727.438496] [c2:3592 :0]        ib_iface.c:1011 UCX  ERROR iface=0x329933a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 3592 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
Test case:[/apps/hpcg] log HPCG multi node results (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:0.025 sec
Failed
(in test file rm_execution_multi_host, line 55)
  `mv $run_yaml $wrk_yaml' failed
Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg
ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory
Moving  to HPCG.32x32x32.P16.gnu15.mvapich2.yaml
mv: missing destination file operand after 'HPCG.32x32x32.P16.gnu15.mvapich2.yaml'
Try 'mv --help' for more information.

Test Suite: rm_execution_single_host

Results

Duration2.284 sec
Tests3
Failures2

Tests

rm_execution_single_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.009 sec
Test case:[/apps/hpcg] run HPCG on single node under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.25 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution_single_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT ${EXE} "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.31957
Batch job 13 submitted

Job 13 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./bin/xhpcg.gnu15.mvapich2 32 32 32 10 (family=mvapich2)
[1776234724.416413] [c1:5005 :0]        ib_iface.c:1011 UCX  ERROR iface=0x1380deb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234724.416411] [c1:5008 :0]        ib_iface.c:1011 UCX  ERROR iface=0x28d60eb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234724.416415] [c1:5009 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3f7cdeb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234724.416579] [c1:5003 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3117aeb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234724.417005] [c1:5007 :0]        ib_iface.c:1011 UCX  ERROR iface=0x26e99eb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234724.417211] [c1:5006 :0]        ib_iface.c:1011 UCX  ERROR iface=0x22201eb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734735) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff4594966c, argv=0x7fff45949660) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(608299279) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff4f4e9dac, argv=0x7fff4f4e9da0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339863823) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc91cd17ac, argv=0x7ffc91cd17a0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(742517007) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fffcddce70c, argv=0x7fffcddce700) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(474081551) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fffbf03a7ec, argv=0x7fffbf03a7e0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339863823) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdeabd454c, argv=0x7ffdeabd4540) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[/apps/hpcg] log HPCG single node results (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:0.025 sec
Failed
(in test file rm_execution_single_host, line 55)
  `mv $run_yaml $wrk_yaml' failed
Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg
ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory
Moving  to HPCG.32x32x32.P8.gnu15.mvapich2.yaml
mv: missing destination file operand after 'HPCG.32x32x32.P8.gnu15.mvapich2.yaml'
Try 'mv --help' for more information.

Test Suite: build

Results

Duration4.209 sec
Tests1
Failures0

Tests

build

Test case:[/apps/hpcg] build HPCG executable (gnu15/openmpi5)
Outcome:Passed
Duration:4.209 sec

Test Suite: rm_execution_multi_host

Results

Duration17.007 sec
Tests3
Failures0

Tests

rm_execution_multi_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.01 sec
Test case:[/apps/hpcg] run HPCG on multi nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:16.947 sec
Test case:[/apps/hpcg] log HPCG multi node results (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.05 sec

Test Suite: rm_execution_single_host

Results

Duration14.902 sec
Tests3
Failures0

Tests

rm_execution_single_host

Test case:[/apps/hpcg] check if resource manager is defined
Outcome:Passed
Duration:0.01 sec
Test case:[/apps/hpcg] run HPCG on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:14.842 sec
Test case:[/apps/hpcg] log HPCG single node results (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.05 sec

Test Suite: build

Results

Duration11.333 sec
Tests1
Failures0

Tests

build

Test case:[Apps/miniFE] build MiniFE executable (gnu15/mpich)
Outcome:Passed
Duration:11.333 sec

Test Suite: rm_execution_multi_host

Results

Duration14.874 sec
Tests2
Failures0

Tests

rm_execution_multi_host

Test case:[Apps/miniFE] run miniFE on multi nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:14.83 sec
Test case:[Apps/miniFE] log miniFE multi node results (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.044 sec

Test Suite: rm_execution_single_host

Results

Duration2.258 sec
Tests2
Failures0

Tests

rm_execution_single_host

Test case:[Apps/miniFE] run miniFE on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.213 sec
Test case:[Apps/miniFE] log miniFE single node results (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.045 sec

Test Suite: build

Results

Duration11.725 sec
Tests1
Failures0

Tests

build

Test case:[Apps/miniFE] build MiniFE executable (gnu15/mvapich2)
Outcome:Passed
Duration:11.725 sec

Test Suite: rm_execution_multi_host

Results

Duration2.287 sec
Tests2
Failures1

Tests

rm_execution_multi_host

Test case:[Apps/miniFE] run miniFE on multi nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.244 sec
Failed
(from function `run_mpi_binary' in file ./common-test/functions, line 434,
 in test file rm_execution_multi_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.25487
Batch job 20 submitted

Job 20 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./src/miniFE.x.gnu15.mvapich2 nx=256 ny=256 nz=256 verify_solution=0 (family=mvapich2)
[1776234825.432335] [c2:3948 :0]        ib_iface.c:1011 UCX  ERROR iface=0x1cea1ad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234825.432340] [c2:3950 :0]        ib_iface.c:1011 UCX  ERROR iface=0xb664ad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234825.433393] [c2:3946 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3a09ead0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234825.430679] [c1:6447 :0]        ib_iface.c:1011 UCX  ERROR iface=0x375ee600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234825.430718] [c1:6448 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3d0e5600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(1010952463) on node 12: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc27de094c, argv=0x7ffc27de0940) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776234825.430752] [c1:6451 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3aaf2600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234825.430689] [c1:6453 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3c7b6600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428367) on node 14: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd670976dc, argv=0x7ffd670976d0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776234825.430815] [c1:6454 :0]        ib_iface.c:1011 UCX  ERROR iface=0x236fa600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234825.434039] [c2:3949 :0]        ib_iface.c:1011 UCX  ERROR iface=0x754ad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234825.434062] [c2:3945 :0]        ib_iface.c:1011 UCX  ERROR iface=0x34b24ad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234825.431232] [c1:6452 :0]        ib_iface.c:1011 UCX  ERROR iface=0x33f5e600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234825.434095] [c2:3944 :0]        ib_iface.c:1011 UCX  ERROR iface=0xbfacad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(1010952463) on node 10: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffddda5a78c, argv=0x7ffddda5a780) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdb35aa14c, argv=0x7ffdb35aa140) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe73d7230c, argv=0x7ffe73d72300) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(1010952463) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd3c33a3fc, argv=0x7ffd3c33a3f0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(205646095) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdab91dafc, argv=0x7ffdab91daf0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(608299279) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffda642bd0c, argv=0x7ffda642bd00) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339863823) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff9be163ec, argv=0x7fff9be163e0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(71428367) on node 9: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc78a834ac, argv=0x7ffc78a834a0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 3944 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
Test case:[Apps/miniFE] log miniFE multi node results (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.043 sec

Test Suite: rm_execution_single_host

Results

Duration1.234 sec
Tests2
Failures1

Tests

rm_execution_single_host

Test case:[Apps/miniFE] run miniFE on single node under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.19 sec
Failed
(from function `run_mpi_binary' in file ./common-test/functions, line 434,
 in test file rm_execution_single_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.17307
Batch job 19 submitted

Job 19 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./src/miniFE.x.gnu15.mvapich2 nx=100 ny=100 nz=100 verify_solution=1 (family=mvapich2)
[1776234823.386964] [c1:6260 :0]        ib_iface.c:1011 UCX  ERROR iface=0x34026ed0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234823.398042] [c1:6264 :0]        ib_iface.c:1011 UCX  ERROR iface=0x658aed0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428367) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffcf4971b3c, argv=0x7ffcf4971b30) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(742517007) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc63d4ee2c, argv=0x7ffc63d4ee20) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 6261 RUNNING AT c1
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
Test case:[Apps/miniFE] log miniFE single node results (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.044 sec

Test Suite: build

Results

Duration8.432 sec
Tests1
Failures0

Tests

build

Test case:[Apps/miniFE] build MiniFE executable (gnu15/openmpi5)
Outcome:Passed
Duration:8.432 sec

Test Suite: rm_execution_multi_host

Results

Duration14.888 sec
Tests2
Failures0

Tests

rm_execution_multi_host

Test case:[Apps/miniFE] run miniFE on multi nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:14.844 sec
Test case:[Apps/miniFE] log miniFE multi node results (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.044 sec

Test Suite: rm_execution_single_host

Results

Duration4.373 sec
Tests2
Failures0

Tests

rm_execution_single_host

Test case:[Apps/miniFE] run miniFE on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.328 sec
Test case:[Apps/miniFE] log miniFE single node results (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.045 sec

Test Suite: os_distribution

Results

Duration0.008 sec
Tests1
Failures0

Tests

os_distribution

Test case:[BOS] OS distribution matches (local)
Outcome:Passed
Duration:0.008 sec

Test Suite: computes

Results

Duration4.363 sec
Tests4
Failures0

Tests

computes

Test case:[BOS] OS distribution matches (2 active computes)
Outcome:Skipped
Duration:0.0 sec
Skipped
disable BOS_RELEASE check
Test case:[BOS] consistent kernel (2 active computes)
Outcome:Passed
Duration:1.093 sec
Test case:[BOS] increased locked memory limits
Outcome:Passed
Duration:2.177 sec
Test case:[BOS] syslog forwarding
Outcome:Passed
Duration:1.093 sec

Test Suite: debugger

Results

Duration0.014 sec
Tests2
Failures0

Tests

debugger

Test case:[Compilers] debugger man page (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[Compilers] debugger availability (gnu15)
Outcome:Passed
Duration:0.006 sec

Test Suite: man_pages

Results

Duration0.117 sec
Tests3
Failures0

Tests

man_pages

Test case:[Compilers] C compiler man/help page (gnu15)
Outcome:Passed
Duration:0.047 sec
Test case:[Compilers] C++ compiler man/help page (gnu15)
Outcome:Passed
Duration:0.033 sec
Test case:[Compilers] Fortran compiler man/help page (gnu15)
Outcome:Passed
Duration:0.037 sec

Test Suite: rm_execution

Results

Duration0.718 sec
Tests6
Failures0

Tests

rm_execution

Test case:[Compilers] C binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.112 sec
Test case:[Compilers] C++ binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.11 sec
Test case:[Compilers] Fortran binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.144 sec
Test case:[Compilers] C openmp binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.113 sec
Test case:[Compilers] C++ openmp binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.123 sec
Test case:[Compilers] Fortran openmp binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.116 sec

Test Suite: version_match

Results

Duration0.423 sec
Tests3
Failures0

Tests

version_match

Test case:[Compilers] compiler module loaded (gnu15)
Outcome:Passed
Duration:0.118 sec
Test case:[Compilers] compiler module version available (gnu15)
Outcome:Passed
Duration:0.125 sec
Test case:[Compilers] C, C++, and Fortran versions match module (gnu15)
Outcome:Passed
Duration:0.18 sec

Test Suite: EasyBuild

Results

Duration540.074 sec
Tests3
Failures0

Tests

EasyBuild

Test case:[EasyBuild] check for RPM
Outcome:Passed
Duration:0.21 sec
Test case:[EasyBuild] test executable
Outcome:Passed
Duration:2.246 sec
Test case:[EasyBuild] test install of tmux
Outcome:Passed
Duration:537.618 sec

Test Suite: rm_execution

Results

Duration0.327 sec
Tests2
Failures0

Tests

rm_execution

Test case:[dev-tools/hwloc] lstopo runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.185 sec
Test case:[dev-tools/hwloc] hwloc_hello runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.142 sec

Test Suite: test_module

Results

Duration0.227 sec
Tests7
Failures0

Tests

test_module

Test case:[HWLOC] Verify HWLOC module is loaded and matches rpm version (gnu15/)
Outcome:Passed
Duration:0.169 sec
Test case:[HWLOC] Verify module HWLOC_DIR is defined and exists (gnu15/)
Outcome:Passed
Duration:0.012 sec
Test case:[HWLOC] Verify module HWLOC_LIB is defined and exists (gnu15/)
Outcome:Passed
Duration:0.01 sec
Test case:[HWLOC] Verify dynamic library available in HWLOC_LIB (gnu15/)
Outcome:Passed
Duration:0.009 sec
Test case:[HWLOC] Verify static library is not present in HWLOC_LIB (gnu15/)
Outcome:Passed
Duration:0.01 sec
Test case:[HWLOC] Verify module HWLOC_INC is defined and exists (gnu15/)
Outcome:Passed
Duration:0.009 sec
Test case:[HWLOC] Verify header file is present in HWLOC_INC (gnu15/)
Outcome:Passed
Duration:0.008 sec

Test Suite: hipy

Results

Duration1.172 sec
Tests1
Failures0

Tests

hipy

Test case:[dev-tools/py3-mpi4py] python hello world (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.172 sec

Test Suite: test_module

Results

Duration0.027 sec
Tests3
Failures0

Tests

test_module

Test case:[mpi4py] Verify module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.007 sec
Test case:[mpi4py] Verify module MPI4PY_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.012 sec
Test case:[mpi4py] Verify PYTHONPATH is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec

Test Suite: hipy

Results

Duration2.248 sec
Tests1
Failures1

Tests

hipy

Test case:[dev-tools/py3-mpi4py] python hello world (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.248 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file hipy, line 35)
  `run_mpi_binary "${_python} helloworld.py" $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.18082
Batch job 32 submitted

Job 32 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm python3 helloworld.py 8 (family=mvapich2)
[1776234890.398008] [c2:4512 :0]        ib_iface.c:1011 UCX  ERROR iface=0x55b4740613e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408399) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffee7df2a58) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776234890.401553] [c1:7671 :0]        ib_iface.c:1011 UCX  ERROR iface=0x55ba9a2e85e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234890.401798] [c1:7670 :0]        ib_iface.c:1011 UCX  ERROR iface=0x557067e6f5e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299535) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7fffab7671d8) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408399) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffef166c2f8) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 4511 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c1: task 0: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion

Test Suite: test_module

Results

Duration0.027 sec
Tests3
Failures0

Tests

test_module

Test case:[mpi4py] Verify module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.008 sec
Test case:[mpi4py] Verify module MPI4PY_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.011 sec
Test case:[mpi4py] Verify PYTHONPATH is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.008 sec

Test Suite: hipy

Results

Duration3.278 sec
Tests1
Failures0

Tests

hipy

Test case:[dev-tools/py3-mpi4py] python hello world (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.278 sec

Test Suite: test_module

Results

Duration0.027 sec
Tests3
Failures0

Tests

test_module

Test case:[mpi4py] Verify module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[mpi4py] Verify module MPI4PY_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.011 sec
Test case:[mpi4py] Verify PYTHONPATH is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec

Test Suite: rm_execution

Results

Duration1.626 sec
Tests1
Failures0

Tests

rm_execution

Test case:[dev-tools/py3-numpy] Matrix multiply under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.626 sec

Test Suite: test_module

Results

Duration0.272 sec
Tests4
Failures0

Tests

test_module

Test case:[dev-tools/py3-numpy] Verify py3-numpy module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.211 sec
Test case:[dev-tools/py3-numpy] Verify NUMPY_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[dev-tools/py3-numpy] Verify NUMPY_BIN is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[dev-tools/py3-numpy] Verify PYTHONPATH is defined (gnu15)
Outcome:Passed
Duration:0.019 sec

Test Suite: rm_execution

Results

Duration1.554 sec
Tests2
Failures0

Tests

rm_execution

Test case:[dev-tools/Valgrind] Callgrind execution under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.384 sec
Test case:[dev-tools/Valgrind] Memcheck execution under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.17 sec

Test Suite: test_module

Results

Duration1.428 sec
Tests8
Failures0

Tests

test_module

Test case:[dev-tools/Valgrind] Verify valgrind module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.189 sec
Test case:[dev-tools/Valgrind] Verify VALGRIND_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.016 sec
Test case:[dev-tools/Valgrind] Verify availability of valgrind binary (gnu15)
Outcome:Passed
Duration:0.028 sec
Test case:[dev-tools/Valgrind] Verify availability of man page (gnu15)
Outcome:Passed
Duration:0.053 sec
Test case:[dev-tools/Valgrind] Verify module VALGRIND_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[dev-tools/Valgrind] Verify header file is present in VALGRIND_INC (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[dev-tools/Valgrind] Callgrind test (gnu15)
Outcome:Passed
Duration:0.255 sec
Test case:[dev-tools/Valgrind] Memcheck test (gnu15)
Outcome:Passed
Duration:0.847 sec

Test Suite: rm_execution

Results

Duration12.655 sec
Tests1
Failures0

Tests

rm_execution

Test case:[R] Running Rscript bench.R under resource manager (slurm/gnu15)
Outcome:Passed
Duration:12.655 sec

Test Suite: test_module

Results

Duration12.926 sec
Tests7
Failures0

Tests

test_module

Test case:[R] Verify R module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.218 sec
Test case:[R] Verify R_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[R] Verify availability of R executable -> R (gnu15)
Outcome:Passed
Duration:0.016 sec
Test case:[R] Verify availability of R executable -> Rscript (gnu15)
Outcome:Passed
Duration:0.017 sec
Test case:[R] Verify module R_SHARE is defined and exists (gnu15)
Outcome:Passed
Duration:0.011 sec
Test case:[R] Running bench.R test
Outcome:Passed
Duration:12.098 sec
Test case:[R] Verify ability to compile C code for R and execute
Outcome:Passed
Duration:0.557 sec

Test Suite: rm_execution

Results

Duration4.441 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/adios2] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.216 sec
Test case:[libs/adios2] MPI F90 binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.225 sec

Test Suite: test_module

Results

Duration0.345 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/adios2] Verify ADIOS2 module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.217 sec
Test case:[libs/adios2] Verify module ADIOS2_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/adios2] Verify module ADIOS2_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/adios2] Verify module ADIOS2_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/adios2] Verify dynamic library available in ADIOS2_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.025 sec
Test case:[libs/adios2] Verify module ADIOS2_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/adios2] Verify header file is present in ADIOS2_INC (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec

Test Suite: rm_execution

Results

Duration6.564 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/adios2] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.23 sec
Test case:[libs/adios2] MPI F90 binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.334 sec

Test Suite: test_module

Results

Duration0.349 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/adios2] Verify ADIOS2 module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.218 sec
Test case:[libs/adios2] Verify module ADIOS2_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/adios2] Verify module ADIOS2_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/adios2] Verify module ADIOS2_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/adios2] Verify dynamic library available in ADIOS2_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.027 sec
Test case:[libs/adios2] Verify module ADIOS2_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/adios2] Verify header file is present in ADIOS2_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration4.495 sec
Tests2
Failures2

Tests

rm_execution

Test case:[libs/adios2] MPI C binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.246 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 28)
  `run_mpi_binary -t "${TIMEOUT}" ./arrays_write "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.20850
Batch job 40 submitted

Job 40 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./arrays_write  (family=mvapich2)
[1776234947.509003] [c1:8647 :0]        ib_iface.c:1011 UCX  ERROR iface=0x384c44c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff0314791c, argv=0x7fff03147910) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776234947.512290] [c1:8645 :0]        ib_iface.c:1011 UCX  ERROR iface=0x1bc4d4c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234947.519486] [c2:4957 :0]        ib_iface.c:1011 UCX  ERROR iface=0x243142d0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(205646095) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffee710cebc, argv=0x7ffee710ceb0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 4956 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c1: task 0: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion
Test case:[libs/adios2] MPI F90 binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.249 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 37)
  `run_mpi_binary -t "${TIMEOUT}" ./scalars_write "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.11040
Batch job 41 submitted

Job 41 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./scalars_write  (family=mvapich2)
[1776234948.129391] [c1:8827 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3fcd9a50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776234948.130233] [c1:8825 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3b626a50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776234948.143896] [c2:5041 :0]        ib_iface.c:1011 UCX  ERROR iface=0x1dc7840: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_module

Results

Duration0.33 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/adios2] Verify ADIOS2 module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.206 sec
Test case:[libs/adios2] Verify module ADIOS2_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/adios2] Verify module ADIOS2_BIN is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/adios2] Verify module ADIOS2_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/adios2] Verify dynamic library available in ADIOS2_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.026 sec
Test case:[libs/adios2] Verify module ADIOS2_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/adios2] Verify header file is present in ADIOS2_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec

Test Suite: rm_execution

Results

Duration0.529 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Accumulators] min-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.133 sec
Test case:[Boost/Accumulators] max-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.136 sec
Test case:[Boost/Accumulators] skewness-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.132 sec
Test case:[Boost/Accumulators] variance-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.128 sec

Test Suite: rm_execution

Results

Duration0.534 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Accumulators] min-test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.129 sec
Test case:[Boost/Accumulators] max-test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.145 sec
Test case:[Boost/Accumulators] skewness-test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.131 sec
Test case:[Boost/Accumulators] variance-test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.129 sec

Test Suite: rm_execution

Results

Duration0.554 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Accumulators] min-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.124 sec
Test case:[Boost/Accumulators] max-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.159 sec
Test case:[Boost/Accumulators] skewness-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.126 sec
Test case:[Boost/Accumulators] variance-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.145 sec

Test Suite: test_module

Results

Duration2.637 sec
Tests8
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.209 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.011 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Test interactive build/exec of f_test.cpp (gnu15/mpich)
Outcome:Passed
Duration:2.372 sec

Test Suite: test_module

Results

Duration2.639 sec
Tests8
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.219 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.011 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Test interactive build/exec of f_test.cpp (gnu15/mvapich2)
Outcome:Passed
Duration:2.364 sec

Test Suite: test_module

Results

Duration2.644 sec
Tests8
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.214 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.011 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Test interactive build/exec of f_test.cpp (gnu15/openmpi5)
Outcome:Passed
Duration:2.373 sec

Test Suite: rm_execution

Results

Duration0.5 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Multi Array] access-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.139 sec
Test case:[Boost/Multi Array] iterators-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.118 sec
Test case:[Boost/Multi Array] resize-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.124 sec
Test case:[Boost/Multi Array] idxgen1-test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.119 sec

Test Suite: rm_execution

Results

Duration0.575 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Multi Array] access-test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.153 sec
Test case:[Boost/Multi Array] iterators-test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.125 sec
Test case:[Boost/Multi Array] resize-test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.16 sec
Test case:[Boost/Multi Array] idxgen1-test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.137 sec

Test Suite: rm_execution

Results

Duration0.488 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Multi Array] access-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.118 sec
Test case:[Boost/Multi Array] iterators-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.116 sec
Test case:[Boost/Multi Array] resize-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.126 sec
Test case:[Boost/Multi Array] idxgen1-test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.128 sec

Test Suite: rm_execution

Results

Duration1.365 sec
Tests1
Failures0

Tests

rm_execution

Test case:[Boost/ublas] bench1_ublas binary under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.365 sec

Test Suite: rm_execution

Results

Duration1.348 sec
Tests1
Failures0

Tests

rm_execution

Test case:[Boost/ublas] bench1_ublas binary under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:1.348 sec

Test Suite: rm_execution

Results

Duration1.359 sec
Tests1
Failures0

Tests

rm_execution

Test case:[Boost/ublas] bench1_ublas binary under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:1.359 sec

Test Suite: rm_execution

Results

Duration1.188 sec
Tests11
Failures0

Tests

rm_execution

Test case:[Boost/Program Options] cmdline_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.155 sec
Test case:[Boost/Program Options] exception_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.135 sec
Test case:[Boost/Program Options] options_description_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.114 sec
Test case:[Boost/Program Options] parsers_test on master host(gnu15/openmpi5)
Outcome:Passed
Duration:0.016 sec
Test case:[Boost/Program Options] parsers_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.132 sec
Test case:[Boost/Program Options] positional_options_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.116 sec
Test case:[Boost/Program Options] required_test on master host(gnu15/openmpi5)
Outcome:Passed
Duration:0.015 sec
Test case:[Boost/Program Options] required_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.145 sec
Test case:[Boost/Program Options] unicode_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.118 sec
Test case:[Boost/Program Options] unrecognized_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.119 sec
Test case:[Boost/Program Options] variable_map_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.123 sec

Test Suite: rm_execution

Results

Duration1.185 sec
Tests11
Failures0

Tests

rm_execution

Test case:[Boost/Program Options] cmdline_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.147 sec
Test case:[Boost/Program Options] exception_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.125 sec
Test case:[Boost/Program Options] options_description_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.119 sec
Test case:[Boost/Program Options] parsers_test on master host(gnu15/mpich)
Outcome:Passed
Duration:0.015 sec
Test case:[Boost/Program Options] parsers_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.129 sec
Test case:[Boost/Program Options] positional_options_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.118 sec
Test case:[Boost/Program Options] required_test on master host(gnu15/mpich)
Outcome:Passed
Duration:0.014 sec
Test case:[Boost/Program Options] required_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.148 sec
Test case:[Boost/Program Options] unicode_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.126 sec
Test case:[Boost/Program Options] unrecognized_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.12 sec
Test case:[Boost/Program Options] variable_map_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.124 sec

Test Suite: rm_execution

Results

Duration1.184 sec
Tests11
Failures0

Tests

rm_execution

Test case:[Boost/Program Options] cmdline_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.14 sec
Test case:[Boost/Program Options] exception_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.123 sec
Test case:[Boost/Program Options] options_description_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.149 sec
Test case:[Boost/Program Options] parsers_test on master host(gnu15/mvapich2)
Outcome:Passed
Duration:0.015 sec
Test case:[Boost/Program Options] parsers_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.121 sec
Test case:[Boost/Program Options] positional_options_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.122 sec
Test case:[Boost/Program Options] required_test on master host(gnu15/mvapich2)
Outcome:Passed
Duration:0.014 sec
Test case:[Boost/Program Options] required_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.12 sec
Test case:[Boost/Program Options] unicode_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.121 sec
Test case:[Boost/Program Options] unrecognized_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.136 sec
Test case:[Boost/Program Options] variable_map_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.123 sec

Test Suite: rm_execution

Results

Duration1.797 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Random] test_piecewise_linear under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:1.14 sec
Test case:[Boost/Random] test_discrete under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.381 sec
Test case:[Boost/Random] test_random_device under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.128 sec
Test case:[Boost/Random] test_random_number_generator under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.148 sec

Test Suite: rm_execution

Results

Duration1.762 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Random] test_piecewise_linear under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.121 sec
Test case:[Boost/Random] test_discrete under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.384 sec
Test case:[Boost/Random] test_random_device under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.131 sec
Test case:[Boost/Random] test_random_number_generator under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.126 sec

Test Suite: rm_execution

Results

Duration1.754 sec
Tests4
Failures0

Tests

rm_execution

Test case:[Boost/Random] test_piecewise_linear under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:1.132 sec
Test case:[Boost/Random] test_discrete under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.378 sec
Test case:[Boost/Random] test_random_device under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.118 sec
Test case:[Boost/Random] test_random_number_generator under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.126 sec

Test Suite: rm_execution

Results

Duration0.171 sec
Tests1
Failures0

Tests

rm_execution

Test case:[Boost] regress under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.171 sec

Test Suite: rm_execution

Results

Duration0.791 sec
Tests3
Failures0

Tests

rm_execution

Test case:[Boost] bad_expression_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.471 sec
Test case:[Boost] named_subexpressions_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.145 sec
Test case:[Boost] recursion_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.175 sec

Test Suite: rm_execution

Results

Duration0.79 sec
Tests3
Failures0

Tests

rm_execution

Test case:[Boost] bad_expression_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.44 sec
Test case:[Boost] named_subexpressions_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.179 sec
Test case:[Boost] recursion_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.171 sec

Test Suite: rm_execution

Results

Duration0.779 sec
Tests3
Failures0

Tests

rm_execution

Test case:[Boost] bad_expression_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.463 sec
Test case:[Boost] named_subexpressions_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.144 sec
Test case:[Boost] recursion_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.172 sec

Test Suite: test_module

Results

Duration0.265 sec
Tests7
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.212 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.011 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.007 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec

Test Suite: test_module

Results

Duration0.27 sec
Tests7
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.214 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.012 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.007 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.009 sec

Test Suite: test_module

Results

Duration0.275 sec
Tests7
Failures0

Tests

test_module

Test case:[BOOST] Verify BOOST module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.218 sec
Test case:[BOOST] Verify module BOOST_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.012 sec
Test case:[BOOST] Verify module BOOST_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify dynamic library available in BOOST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[BOOST] Verify static library is not present in BOOST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[BOOST] Verify module BOOST_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[BOOST] Verify header file is present in BOOST_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec

Test Suite: rm_execution

Results

Duration61.839 sec
Tests7
Failures0

Tests

rm_execution

Test case:[Boost/MPI] all_gather_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.272 sec
Test case:[Boost/MPI] all_reduce_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:5.376 sec
Test case:[Boost/MPI] all_to_all_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.431 sec
Test case:[Boost/MPI] groups_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:8.523 sec
Test case:[Boost/MPI] broadcast_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:10.632 sec
Test case:[Boost/MPI] ring_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:12.766 sec
Test case:[Boost/MPI] pointer_test under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:14.839 sec

Test Suite: rm_execution

Results

Duration17.64 sec
Tests7
Failures0

Tests

rm_execution

Test case:[Boost/MPI] all_gather_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.169 sec
Test case:[Boost/MPI] all_reduce_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.165 sec
Test case:[Boost/MPI] all_to_all_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.219 sec
Test case:[Boost/MPI] groups_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.218 sec
Test case:[Boost/MPI] broadcast_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.268 sec
Test case:[Boost/MPI] ring_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.279 sec
Test case:[Boost/MPI] pointer_test under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.322 sec

Test Suite: rm_execution

Results

Duration52.59 sec
Tests7
Failures6

Tests

rm_execution

Test case:[Boost/MPI] all_gather_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.248 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 24)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.4443
Batch job 129 submitted

Job 129 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./all_gather_test atest (family=mvapich2)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_gather_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
[1776235839.442546] [c1:14989:0]        ib_iface.c:1011 UCX  ERROR iface=0x160727e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235839.442721] [c1:14983:0]        ib_iface.c:1011 UCX  ERROR iface=0x2463c7e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235839.445438] [c1:14984:0]        ib_iface.c:1011 UCX  ERROR iface=0x3e9c67e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235839.453255] [c2:8442 :0]        ib_iface.c:1011 UCX  ERROR iface=0xedd3480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235839.453260] [c2:8445 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3eee7480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235839.453259] [c2:8448 :0]        ib_iface.c:1011 UCX  ERROR iface=0x38d77480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235839.453433] [c2:8447 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3dbc3480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235839.453817] [c2:8443 :0]        ib_iface.c:1011 UCX  ERROR iface=0xe684480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235839.453872] [c2:8444 :0]        ib_iface.c:1011 UCX  ERROR iface=0x392af480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 8: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 11: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 14: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 13: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 10: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 14984 RUNNING AT c1
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:1@c2] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:1@c2] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:1@c2] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c2: task 1: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion
Test case:[Boost/MPI] all_reduce_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.343 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 33)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.25024
Batch job 130 submitted

Job 130 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./all_reduce_test atest (family=mvapich2)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_reduce_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
[1776235841.072675] [c2:8540 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3efa5480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235841.073004] [c2:8544 :0]        ib_iface.c:1011 UCX  ERROR iface=0xad1c480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235841.073319] [c2:8543 :0]        ib_iface.c:1011 UCX  ERROR iface=0x26f7480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 13: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 12: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235841.075133] [c2:8539 :0]        ib_iface.c:1011 UCX  ERROR iface=0xa067480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235841.077859] [c1:15175:0]        ib_iface.c:1011 UCX  ERROR iface=0x233de7e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235841.078056] [c1:15170:0]        ib_iface.c:1011 UCX  ERROR iface=0x2506d7e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 15171 RUNNING AT c1
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:1@c2] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:1@c2] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:1@c2] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c2: task 1: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion
Test case:[Boost/MPI] all_to_all_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:5.388 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 42)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.8919
Batch job 131 submitted

Job 131 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./all_to_all_test atest (family=mvapich2)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: all_to_all_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
[1776235842.647429] [c2:8634 :0]        ib_iface.c:1011 UCX  ERROR iface=0x2b4f1450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235842.647427] [c2:8641 :0]        ib_iface.c:1011 UCX  ERROR iface=0xa2ae450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235842.647571] [c2:8635 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3f2bd450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235842.647579] [c2:8636 :0]        ib_iface.c:1011 UCX  ERROR iface=0x3a4bd450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235842.647575] [c2:8639 :0]        ib_iface.c:1011 UCX  ERROR iface=0x1fb45450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235842.648609] [c2:8638 :0]        ib_iface.c:1011 UCX  ERROR iface=0x2852c450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 8: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 15: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 10: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 13: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 12: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235842.649145] [c1:15360:0]        ib_iface.c:1011 UCX  ERROR iface=0x2147d7b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235842.649155] [c1:15362:0]        ib_iface.c:1011 UCX  ERROR iface=0x218f47b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235842.649151] [c1:15363:0]        ib_iface.c:1011 UCX  ERROR iface=0x39d967b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 15357 RUNNING AT c1
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:1@c2] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:1@c2] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:1@c2] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c2: task 1: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion
Test case:[Boost/MPI] groups_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:6.443 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 51)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.21018
Batch job 132 submitted

Job 132 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./groups_test atest (family=mvapich2)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: groups_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
[1776235844.230823] [c2:8736 :0]        ib_iface.c:1011 UCX  ERROR iface=0x10519420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.230962] [c2:8735 :0]        ib_iface.c:1011 UCX  ERROR iface=0x21a4a420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.230978] [c2:8729 :0]        ib_iface.c:1011 UCX  ERROR iface=0x18b41420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 15: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 8: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 14: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235844.232299] [c2:8734 :0]        ib_iface.c:1011 UCX  ERROR iface=0x249b3420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.232584] [c2:8731 :0]        ib_iface.c:1011 UCX  ERROR iface=0x38d24420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 13: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235844.233195] [c1:15545:0]        ib_iface.c:1011 UCX  ERROR iface=0x36e28780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.233651] [c1:15547:0]        ib_iface.c:1011 UCX  ERROR iface=0x17d01780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235844.234802] [c1:15549:0]        ib_iface.c:1011 UCX  ERROR iface=0x21608780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.234939] [c1:15551:0]        ib_iface.c:1011 UCX  ERROR iface=0x2d17780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.235069] [c1:15550:0]        ib_iface.c:1011 UCX  ERROR iface=0x1a0ec780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235844.235523] [c1:15546:0]        ib_iface.c:1011 UCX  ERROR iface=0xfa59780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 8730 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c1: task 0: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion
Test case:[Boost/MPI] broadcast_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:7.49 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 60)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.1553
Batch job 133 submitted

Job 133 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./broadcast_test atest (family=mvapich2)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: broadcast_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
[1776235844.873741] [c1:15738:0]        ib_iface.c:1011 UCX  ERROR iface=0x3569d820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.873839] [c1:15732:0]        ib_iface.c:1011 UCX  ERROR iface=0x3110d820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.873946] [c1:15737:0]        ib_iface.c:1011 UCX  ERROR iface=0x5a95820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.876204] [c2:8825 :0]        ib_iface.c:1011 UCX  ERROR iface=0x267634c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.874085] [c1:15731:0]        ib_iface.c:1011 UCX  ERROR iface=0x22e2e820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.874104] [c1:15736:0]        ib_iface.c:1011 UCX  ERROR iface=0x35098820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.874532] [c1:15734:0]        ib_iface.c:1011 UCX  ERROR iface=0x2a0a5820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235844.877995] [c2:8828 :0]        ib_iface.c:1011 UCX  ERROR iface=0x22eed4c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235844.878058] [c2:8829 :0]        ib_iface.c:1011 UCX  ERROR iface=0x1c184c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235844.878900] [c2:8830 :0]        ib_iface.c:1011 UCX  ERROR iface=0x32bfa4c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235844.879239] [c2:8826 :0]        ib_iface.c:1011 UCX  ERROR iface=0x297314c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 8826 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c1: task 0: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion
Test case:[Boost/MPI] ring_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:18.136 sec
Test case:[Boost/MPI] pointer_test under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:8.542 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 79)
  `run_mpi_binary ./$test atest 2 "${TASKS}"' failed
job script = /tmp/job.ohpc-test.14550
Batch job 135 submitted

Job 135 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pointer_test atest (family=mvapich2)
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_testatest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
 --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument
                    and should be placed after all Boost.Test arguments and the -- separator.
                    For example: pointer_test --random -- atest
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
Running 1 test case...
[1776235847.188727] [c2:9016 :0]        ib_iface.c:1011 UCX  ERROR iface=0x64c14f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235847.188723] [c2:9023 :0]        ib_iface.c:1011 UCX  ERROR iface=0x13e124f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235847.188918] [c2:9017 :0]        ib_iface.c:1011 UCX  ERROR iface=0x2db024f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235847.189090] [c2:9020 :0]        ib_iface.c:1011 UCX  ERROR iface=0x2c4544f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235847.189787] [c2:9022 :0]        ib_iface.c:1011 UCX  ERROR iface=0x1d5a04f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 15: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 8: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235847.190396] [c2:9018 :0]        ib_iface.c:1011 UCX  ERROR iface=0xd5f94f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 12: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 14: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776235847.189946] [c1:16105:0]        ib_iface.c:1011 UCX  ERROR iface=0x1f074850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235847.189933] [c1:16106:0]        ib_iface.c:1011 UCX  ERROR iface=0x29cf4850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235847.189909] [c1:16111:0]        ib_iface.c:1011 UCX  ERROR iface=0x2cf18850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235847.189965] [c1:16112:0]        ib_iface.c:1011 UCX  ERROR iface=0x31ea4850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235847.190031] [c1:16109:0]        ib_iface.c:1011 UCX  ERROR iface=0x38d92850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776235847.190100] [c1:16110:0]        ib_iface.c:1011 UCX  ERROR iface=0x436d850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 9018 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

Test Suite: test_module

Results

Duration0.233 sec
Tests7
Failures0

Tests

test_module

Test case:[FFTW] Verify FFTW module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.172 sec
Test case:[FFTW] Verify module FFTW_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.012 sec
Test case:[FFTW] Verify module FFTW_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify dynamic library available in FFTW_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify static library is not present in FFTW_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify module FFTW_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[FFTW] Verify header file is present in FFTW_INC (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec

Test Suite: test_module

Results

Duration0.23 sec
Tests7
Failures0

Tests

test_module

Test case:[FFTW] Verify FFTW module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.172 sec
Test case:[FFTW] Verify module FFTW_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.012 sec
Test case:[FFTW] Verify module FFTW_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify dynamic library available in FFTW_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.009 sec
Test case:[FFTW] Verify static library is not present in FFTW_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.009 sec
Test case:[FFTW] Verify module FFTW_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.009 sec
Test case:[FFTW] Verify header file is present in FFTW_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.009 sec

Test Suite: test_module

Results

Duration0.243 sec
Tests7
Failures0

Tests

test_module

Test case:[FFTW] Verify FFTW module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.182 sec
Test case:[FFTW] Verify module FFTW_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.012 sec
Test case:[FFTW] Verify module FFTW_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[FFTW] Verify dynamic library available in FFTW_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify static library is not present in FFTW_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify module FFTW_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[FFTW] Verify header file is present in FFTW_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec

Test Suite: rm_execution

Results

Duration4.5 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/FFTW] Serial C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.165 sec
Test case:[libs/FFTW] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.22 sec
Test case:[libs/FFTW] Serial Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.115 sec

Test Suite: rm_execution

Results

Duration2.175 sec
Tests3
Failures1

Tests

rm_execution

Test case:[libs/FFTW] Serial C binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.136 sec
Test case:[libs/FFTW] MPI C binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.189 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 41)
  `run_mpi_binary -t "${TIMEOUT}" ./C_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.12079
Batch job 149 submitted

Job 149 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_mpi_test 8 (family=mvapich2)
[1776236096.028670] [c1:18489:0]        ib_iface.c:1011 UCX  ERROR iface=0x13520e00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236096.029030] [c1:18487:0]        ib_iface.c:1011 UCX  ERROR iface=0x2ef79e00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236096.031310] [c2:10161:0]        ib_iface.c:1011 UCX  ERROR iface=0xea69bf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236096.031865] [c2:10160:0]        ib_iface.c:1011 UCX  ERROR iface=0xec66bf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428367) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff903867dc, argv=0x7fff903867d0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(742517007) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd989e517c, argv=0x7ffd989e5170) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(1010952463) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdc369f1bc, argv=0x7ffdc369f1b0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339863823) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc3314843c, argv=0x7ffc33148430) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/FFTW] Serial Fortran binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.85 sec

Test Suite: rm_execution

Results

Duration8.332 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/FFTW] Serial C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.187 sec
Test case:[libs/FFTW] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.317 sec
Test case:[libs/FFTW] Serial Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.828 sec

Test Suite: rm_execution

Results

Duration34.268 sec
Tests50
Failures0

Tests

rm_execution

Test case:[libs/GSL] run test_gsl_histogram (gnu15)
Outcome:Passed
Duration:0.027 sec
Test case:[libs/GSL] run block under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.184 sec
Test case:[libs/GSL] run bspline under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.507 sec
Test case:[libs/GSL] run cblas under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.122 sec
Test case:[libs/GSL] run cdf under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.149 sec
Test case:[libs/GSL] run cheb under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.144 sec
Test case:[libs/GSL] run combination under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.114 sec
Test case:[libs/GSL] run complex under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.107 sec
Test case:[libs/GSL] run const under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.142 sec
Test case:[libs/GSL] run deriv under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.116 sec
Test case:[libs/GSL] run dht under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.213 sec
Test case:[libs/GSL] run diff under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.113 sec
Test case:[libs/GSL] run eigen under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.293 sec
Test case:[libs/GSL] run err under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.112 sec
Test case:[libs/GSL] run fft under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.427 sec
Test case:[libs/GSL] run fit under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.118 sec
Test case:[libs/GSL] run histogram under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.167 sec
Test case:[libs/GSL] run ieee-utils under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.12 sec
Test case:[libs/GSL] run integration under resource manager (slurm/gnu15)
Outcome:Passed
Duration:2.784 sec
Test case:[libs/GSL] run interpolation under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.139 sec
Test case:[libs/GSL] run linalg under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.252 sec
Test case:[libs/GSL] run matrix under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.623 sec
Test case:[libs/GSL] run min under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.143 sec
Test case:[libs/GSL] run monte under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.355 sec
Test case:[libs/GSL] run multifit under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.784 sec
Test case:[libs/GSL] run multilarge under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.159 sec
Test case:[libs/GSL] run multimin under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.122 sec
Test case:[libs/GSL] run multiroots under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.121 sec
Test case:[libs/GSL] run multiset under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.16 sec
Test case:[libs/GSL] run ntuple under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.122 sec
Test case:[libs/GSL] run ode-initval under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.392 sec
Test case:[libs/GSL] run ode-initval2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:5.651 sec
Test case:[libs/GSL] run permutation under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.137 sec
Test case:[libs/GSL] run poly under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.125 sec
Test case:[libs/GSL] run qrng under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.11 sec
Test case:[libs/GSL] run randist under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.967 sec
Test case:[libs/GSL] run rng under resource manager (slurm/gnu15)
Outcome:Passed
Duration:2.975 sec
Test case:[libs/GSL] run roots under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.114 sec
Test case:[libs/GSL] run rstat under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.428 sec
Test case:[libs/GSL] run siman under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.211 sec
Test case:[libs/GSL] run sort under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.194 sec
Test case:[libs/GSL] run spblas under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.323 sec
Test case:[libs/GSL] run specfunc under resource manager (slurm/gnu15)
Outcome:Passed
Duration:2.414 sec
Test case:[libs/GSL] run splinalg under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.315 sec
Test case:[libs/GSL] run spmatrix under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.137 sec
Test case:[libs/GSL] run statistics under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.133 sec
Test case:[libs/GSL] run sum under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.143 sec
Test case:[libs/GSL] run sys under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.122 sec
Test case:[libs/GSL] run vector under resource manager (slurm/gnu15)
Outcome:Passed
Duration:4.665 sec
Test case:[libs/GSL] run wavelet under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.773 sec

Test Suite: test_module

Results

Duration0.222 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/GSL] Verify GSL module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.164 sec
Test case:[libs/GSL] Verify module GSL_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.011 sec
Test case:[libs/GSL] Verify module GSL_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/GSL] Verify dynamic library available in GSL_LIB (gnu15)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/GSL] Verify static library is not present in GSL_LIB (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/GSL] Verify module GSL_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/GSL] Verify header file is present in GSL_INC (gnu15)
Outcome:Passed
Duration:0.01 sec

Test Suite: rm_execution

Results

Duration20.195 sec
Tests12
Failures9

Tests

rm_execution

Test case:[libs/HYPRE] 2 PE structured test binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.247 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 28)
  `run_mpi_binary -t "${TIMEOUT}" ./ex1 "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.19967
Batch job 214 submitted

Job 214 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex1 8 (family=mvapich2)
[1776236354.443845] [c2:11186:0]        ib_iface.c:1011 UCX  ERROR iface=0x31385cc0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(742517007) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffcb40103cc, argv=0x7ffcb40103c0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236354.445849] [c1:24560:0]        ib_iface.c:1011 UCX  ERROR iface=0x1f3bae20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(742517007) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd08fc42ec, argv=0x7ffd08fc42e0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/HYPRE] 2 PE PCG with SMG preconditioner binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.247 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 37)
  `run_mpi_binary -t "${TIMEOUT}" ./ex2 "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.9708
Batch job 215 submitted

Job 215 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex2 8 (family=mvapich2)
[1776236354.988885] [c1:24734:0]        ib_iface.c:1011 UCX  ERROR iface=0x14641e20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd78e72f0c, argv=0x7ffd78e72f00) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236354.994371] [c2:11269:0]        ib_iface.c:1011 UCX  ERROR iface=0x7782cc0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe99a27c2c, argv=0x7ffe99a27c20) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/HYPRE] 2 PE Semi-Structured PCG binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.294 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 46)
  `run_mpi_binary -t "${TIMEOUT}" ./ex6 "" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.29829
Batch job 216 submitted

Job 216 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex6  (family=mvapich2)
[1776236355.537650] [c1:24910:0]        ib_iface.c:1011 UCX  ERROR iface=0x18686f10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236355.545743] [c2:11352:0]        ib_iface.c:1011 UCX  ERROR iface=0x1d23ab40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339863823) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe74b1c34c, argv=0x7ffe74b1c340) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(71428367) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd59a5964c, argv=0x7ffd59a59640) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/HYPRE] 2 PE Three-part stencil binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.296 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 55)
  `run_mpi_binary -t "${TIMEOUT}" ./ex8 "" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.5668
Batch job 217 submitted

Job 217 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex8  (family=mvapich2)
[1776236356.110244] [c2:11434:0]        ib_iface.c:1011 UCX  ERROR iface=0x908ab40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236356.108517] [c1:25085:0]        ib_iface.c:1011 UCX  ERROR iface=0x2bc9cf10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(205646095) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffff2573a6c, argv=0x7ffff2573a60) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(474081551) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc952b630c, argv=0x7ffc952b6300) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/HYPRE] 2 PE FORTRAN PCG with PFMG preconditioner binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.246 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 64)
  `run_mpi_binary -t "${TIMEOUT}" ./ex12f "" 1 2' failed
job script = /tmp/job.ohpc-test.8746
Batch job 218 submitted

Job 218 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex12f  (family=mvapich2)
[1776236357.398145] [c1:25259:0]        ib_iface.c:1011 UCX  ERROR iface=0x162bfd70: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/HYPRE] 2 PE -Delta u = 1 binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.243 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 73)
  `run_mpi_binary -t "${TIMEOUT}" ./ex3 "-n 33 -solver 0 -v 1 1" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.25785
Batch job 219 submitted

Job 219 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex3 -n 33 -solver 0 -v 1 1 (family=mvapich2)
[1776236357.933046] [c2:11517:0]        ib_iface.c:1011 UCX  ERROR iface=0xa8decc0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428367) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffca3c6b54c, argv=0x7ffca3c6b540) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236357.933441] [c1:25433:0]        ib_iface.c:1011 UCX  ERROR iface=0x2d4ade20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffde195f72c, argv=0x7ffde195f720) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/HYPRE] 2 PE convection-reaction-diffusion binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.192 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 82)
  `run_mpi_binary -t "${TIMEOUT}" ./ex4 "-n 33 -solver 10 -K 3 -B 0 -C 1 -U0 2 -F 4" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.7510
Batch job 220 submitted

Job 220 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex4 -n 33 -solver 10 -K 3 -B 0 -C 1 -U0 2 -F 4 (family=mvapich2)
[1776236358.529260] [c1:25609:0]        ib_iface.c:1011 UCX  ERROR iface=0x2841ef10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdf589654c, argv=0x7ffdf5896540) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236358.543281] [c2:11601:0]        ib_iface.c:1011 UCX  ERROR iface=0x15c74a30: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdb8d1ac2c, argv=0x7ffdb8d1ac20) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/HYPRE] 2 PE FORTRAN 2-D Laplacian binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.239 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 91)
  `run_mpi_binary -t "${TIMEOUT}" ./ex5f "" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.23528
Batch job 221 submitted

Job 221 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex5f  (family=mvapich2)
[1776236359.106829] [c2:11683:0]        ib_iface.c:1011 UCX  ERROR iface=0x1cee3c10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236359.105490] [c1:25783:0]        ib_iface.c:1011 UCX  ERROR iface=0x17d73dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/HYPRE] 2 PE Semi-Structured convection binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE biharmonic binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE C++ Finite Element Interface binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Skipped
Duration:0.0 sec
Skipped
C++ example depends on non-installed header
Test case:[libs/HYPRE] 2 PE 2-D Laplacian eigenvalue binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.191 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 130)
  `run_mpi_binary -t "${TIMEOUT}" ./ex11 "" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.14385
Batch job 222 submitted

Job 222 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex11  (family=mvapich2)
[1776236359.639125] [c2:11767:0]        ib_iface.c:1011 UCX  ERROR iface=0xe0e8b40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236359.637674] [c1:25958:0]        ib_iface.c:1011 UCX  ERROR iface=0x275e2f10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe32e0d6fc, argv=0x7ffe32e0d6f0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(608299279) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffde567587c, argv=0x7ffde5675870) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_module

Results

Duration0.331 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/HYPRE] Verify HYPRE module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.204 sec
Test case:[libs/HYPRE] Verify HYPRE_DIR is defined and directory exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/HYPRE] Verify module HYPRE_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/HYPRE] Verify dynamic library available in HYPRE_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/HYPRE] Verify static library is not present in HYPRE_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/HYPRE] Verify module HYPRE_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/HYPRE] Verify header file is present in HYPRE_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec

Test Suite: rm_execution

Results

Duration56.886 sec
Tests12
Failures0

Tests

rm_execution

Test case:[libs/HYPRE] 2 PE structured test binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.223 sec
Test case:[libs/HYPRE] 2 PE PCG with SMG preconditioner binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.335 sec
Test case:[libs/HYPRE] 2 PE Semi-Structured PCG binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.435 sec
Test case:[libs/HYPRE] 2 PE Three-part stencil binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:8.548 sec
Test case:[libs/HYPRE] 2 PE FORTRAN PCG with PFMG preconditioner binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:7.494 sec
Test case:[libs/HYPRE] 2 PE -Delta u = 1 binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.436 sec
Test case:[libs/HYPRE] 2 PE convection-reaction-diffusion binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.432 sec
Test case:[libs/HYPRE] 2 PE FORTRAN 2-D Laplacian binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.432 sec
Test case:[libs/HYPRE] 2 PE Semi-Structured convection binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE biharmonic binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE C++ Finite Element Interface binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
C++ example depends on non-installed header
Test case:[libs/HYPRE] 2 PE 2-D Laplacian eigenvalue binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:8.551 sec

Test Suite: test_module

Results

Duration0.339 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/HYPRE] Verify HYPRE module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.211 sec
Test case:[libs/HYPRE] Verify HYPRE_DIR is defined and directory exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/HYPRE] Verify module HYPRE_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/HYPRE] Verify dynamic library available in HYPRE_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/HYPRE] Verify static library is not present in HYPRE_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/HYPRE] Verify module HYPRE_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/HYPRE] Verify header file is present in HYPRE_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration21.037 sec
Tests12
Failures0

Tests

rm_execution

Test case:[libs/HYPRE] 2 PE structured test binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.168 sec
Test case:[libs/HYPRE] 2 PE PCG with SMG preconditioner binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.222 sec
Test case:[libs/HYPRE] 2 PE Semi-Structured PCG binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.218 sec
Test case:[libs/HYPRE] 2 PE Three-part stencil binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.275 sec
Test case:[libs/HYPRE] 2 PE FORTRAN PCG with PFMG preconditioner binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.219 sec
Test case:[libs/HYPRE] 2 PE -Delta u = 1 binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.218 sec
Test case:[libs/HYPRE] 2 PE convection-reaction-diffusion binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.221 sec
Test case:[libs/HYPRE] 2 PE FORTRAN 2-D Laplacian binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec
Test case:[libs/HYPRE] 2 PE Semi-Structured convection binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE biharmonic binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[libs/HYPRE] 2 PE C++ Finite Element Interface binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
C++ example depends on non-installed header
Test case:[libs/HYPRE] 2 PE 2-D Laplacian eigenvalue binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.273 sec

Test Suite: test_module

Results

Duration0.326 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/HYPRE] Verify HYPRE module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.202 sec
Test case:[libs/HYPRE] Verify HYPRE_DIR is defined and directory exists (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/HYPRE] Verify module HYPRE_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/HYPRE] Verify dynamic library available in HYPRE_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/HYPRE] Verify static library is not present in HYPRE_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/HYPRE] Verify module HYPRE_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/HYPRE] Verify header file is present in HYPRE_INC (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_metis

Results

Duration3.56 sec
Tests4
Failures0

Tests

test_metis

Test case:[libs/Metis] Graph partition (gnu15)
Outcome:Passed
Duration:0.358 sec
Test case:[libs/Metis] Fill-reducing ordering (gnu15)
Outcome:Passed
Duration:2.752 sec
Test case:[libs/Metis] Mesh to graph conversion (gnu15)
Outcome:Passed
Duration:0.054 sec
Test case:[libs/Metis] C API mesh partitioning (slurm/gnu15)
Outcome:Passed
Duration:0.396 sec

Test Suite: test_module

Results

Duration0.267 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/Metis] Verify METIS module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.178 sec
Test case:[libs/Metis] Verify module METIS_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/Metis] Verify module METIS_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/Metis] Verify availability of m2gmetis binary (gnu15)
Outcome:Passed
Duration:0.028 sec
Test case:[libs/Metis] Verify module METIS_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/Metis] Verify dynamic library available in METIS_LIB (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/Metis] Verify static library is not present in METIS_LIB (gnu15)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/Metis] Verify module METIS_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/Metis] Verify header file is present in METIS_INC (gnu15)
Outcome:Passed
Duration:0.008 sec

Test Suite: rm_execution

Results

Duration10.072 sec
Tests5
Failures0

Tests

rm_execution

Test case:[libs/Mumps] C (double precision) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.224 sec
Test case:[libs/Mumps] Fortran (single precision) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.224 sec
Test case:[libs/Mumps] Fortran (double precision) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.172 sec
Test case:[libs/Mumps] Fortran (complex) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.226 sec
Test case:[libs/Mumps] Fortran (double complex) runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.226 sec

Test Suite: test_module

Results

Duration0.2 sec
Tests2
Failures0

Tests

test_module

Test case:[libs/MUMPS] Verify MUMPS module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.19 sec
Test case:[libs/MUMPS] Verify MUMPS_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.01 sec

Test Suite: rm_execution

Results

Duration15.33 sec
Tests5
Failures0

Tests

rm_execution

Test case:[libs/Mumps] C (double precision) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.274 sec
Test case:[libs/Mumps] Fortran (single precision) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.277 sec
Test case:[libs/Mumps] Fortran (double precision) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.225 sec
Test case:[libs/Mumps] Fortran (complex) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.276 sec
Test case:[libs/Mumps] Fortran (double complex) runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.278 sec

Test Suite: test_module

Results

Duration0.202 sec
Tests2
Failures0

Tests

test_module

Test case:[libs/MUMPS] Verify MUMPS module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.192 sec
Test case:[libs/MUMPS] Verify MUMPS_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.01 sec

Test Suite: rm_execution

Results

Duration13.358 sec
Tests5
Failures5

Tests

rm_execution

Test case:[libs/Mumps] C (double precision) runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.204 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 26)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.17037
Batch job 239 submitted

Job 239 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_double null (family=mvapich2)
[1776236458.418566] [c1:28719:0]        ib_iface.c:1011 UCX  ERROR iface=0x7b34e20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236458.421813] [c2:13066:0]        ib_iface.c:1011 UCX  ERROR iface=0x323cecc0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(205646095) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffecdd05c4c, argv=0x7ffecdd05c40) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339863823) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc30279a4c, argv=0x7ffc30279a40) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/Mumps] Fortran (single precision) runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.305 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 36)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.27980
Batch job 240 submitted

Job 240 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_single null (family=mvapich2)
[1776236461.388371] [c1:28894:0]        ib_iface.c:1011 UCX  ERROR iface=0x35283dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236461.403985] [c2:13148:0]        ib_iface.c:1011 UCX  ERROR iface=0x1119dc10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/Mumps] Fortran (double precision) runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.195 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 46)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.27291
Batch job 241 submitted

Job 241 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_double null (family=mvapich2)
[1776236463.399125] [c2:13247:0]        ib_iface.c:1011 UCX  ERROR iface=0x270cac10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236463.403860] [c1:29083:0]        ib_iface.c:1011 UCX  ERROR iface=0xf17ddd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/Mumps] Fortran (complex) runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.299 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 56)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.4362
Batch job 242 submitted

Job 242 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_complex null (family=mvapich2)
[1776236466.415930] [c1:29258:0]        ib_iface.c:1011 UCX  ERROR iface=0x33d12dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236466.426075] [c2:13330:0]        ib_iface.c:1011 UCX  ERROR iface=0x323a6c10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/Mumps] Fortran (double complex) runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.355 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 66)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.31553
Batch job 243 submitted

Job 243 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_doublecomplex null (family=mvapich2)
[1776236470.404394] [c2:13412:0]        ib_iface.c:1011 UCX  ERROR iface=0x282b9c10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236470.406633] [c1:29434:0]        ib_iface.c:1011 UCX  ERROR iface=0x38fe6dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_module

Results

Duration0.195 sec
Tests2
Failures0

Tests

test_module

Test case:[libs/MUMPS] Verify MUMPS module is loaded and matches rpm version (gnu15-mvapich2)
Outcome:Passed
Duration:0.186 sec
Test case:[libs/MUMPS] Verify MUMPS_DIR is defined and directory exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.009 sec

Test Suite: test_C_module

Results

Duration0.392 sec
Tests9
Failures0

Tests

test_C_module

Test case:[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.196 sec
Test case:[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify availability of nc-config binary (gnu15)
Outcome:Passed
Duration:0.049 sec
Test case:[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu15)
Outcome:Passed
Duration:0.019 sec

Test Suite: test_CXX_module

Results

Duration0.388 sec
Tests9
Failures0

Tests

test_CXX_module

Test case:[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.201 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu15)
Outcome:Passed
Duration:0.047 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu15)
Outcome:Passed
Duration:0.02 sec

Test Suite: test_Fortran_module

Results

Duration0.399 sec
Tests9
Failures0

Tests

test_Fortran_module

Test case:[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.203 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu15)
Outcome:Passed
Duration:0.049 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu15)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_netcdf

Results

Duration2.359 sec
Tests8
Failures0

Tests

test_netcdf

Test case:[libs/NetCDF] ncdump availability (gnu15)
Outcome:Passed
Duration:0.088 sec
Test case:[libs/NetCDF] nctest availability (gnu15)
Outcome:Passed
Duration:0.288 sec
Test case:[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu15)
Outcome:Passed
Duration:0.083 sec
Test case:[libs/NetCDF] verify nc4 available for Fortran interface (gnu15)
Outcome:Passed
Duration:0.06 sec
Test case:[libs/NetCDF] verify nc4 available for C++ interface (gnu15)
Outcome:Skipped
Duration:0.0 sec
Skipped
option no longer supported
Test case:[libs/NetCDF] C write/read (slurm/gnu15)
Outcome:Passed
Duration:0.554 sec
Test case:[libs/NetCDF] Fortran write/read (slurm/gnu15)
Outcome:Passed
Duration:0.609 sec
Test case:[libs/NetCDF] C++ write/read (slurm/gnu15)
Outcome:Passed
Duration:0.677 sec

Test Suite: test_C_module

Results

Duration0.395 sec
Tests9
Failures0

Tests

test_C_module

Test case:[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.202 sec
Test case:[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify availability of nc-config binary (gnu15/mpich)
Outcome:Passed
Duration:0.048 sec
Test case:[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_CXX_module

Results

Duration0.406 sec
Tests9
Failures0

Tests

test_CXX_module

Test case:[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.21 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu15/mpich)
Outcome:Passed
Duration:0.047 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec

Test Suite: test_Fortran_module

Results

Duration0.401 sec
Tests9
Failures0

Tests

test_Fortran_module

Test case:[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.209 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu15/mpich)
Outcome:Passed
Duration:0.047 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_netcdf

Results

Duration2.565 sec
Tests8
Failures0

Tests

test_netcdf

Test case:[libs/NetCDF] ncdump availability (gnu15/mpich)
Outcome:Passed
Duration:0.093 sec
Test case:[libs/NetCDF] nctest availability (gnu15/mpich)
Outcome:Passed
Duration:0.301 sec
Test case:[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu15/mpich)
Outcome:Passed
Duration:0.084 sec
Test case:[libs/NetCDF] verify nc4 available for Fortran interface (gnu15/mpich)
Outcome:Passed
Duration:0.063 sec
Test case:[libs/NetCDF] verify nc4 available for C++ interface (gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
option no longer supported
Test case:[libs/NetCDF] C write/read (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.604 sec
Test case:[libs/NetCDF] Fortran write/read (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.769 sec
Test case:[libs/NetCDF] C++ write/read (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.651 sec

Test Suite: test_C_module

Results

Duration0.388 sec
Tests9
Failures0

Tests

test_C_module

Test case:[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.193 sec
Test case:[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/NetCDF] Verify availability of nc-config binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.049 sec
Test case:[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.023 sec

Test Suite: test_CXX_module

Results

Duration0.393 sec
Tests9
Failures0

Tests

test_CXX_module

Test case:[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.201 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.049 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec

Test Suite: test_Fortran_module

Results

Duration0.395 sec
Tests9
Failures0

Tests

test_Fortran_module

Test case:[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.204 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.047 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_netcdf

Results

Duration2.511 sec
Tests8
Failures0

Tests

test_netcdf

Test case:[libs/NetCDF] ncdump availability (gnu15/mvapich2)
Outcome:Passed
Duration:0.094 sec
Test case:[libs/NetCDF] nctest availability (gnu15/mvapich2)
Outcome:Passed
Duration:0.321 sec
Test case:[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu15/mvapich2)
Outcome:Passed
Duration:0.083 sec
Test case:[libs/NetCDF] verify nc4 available for Fortran interface (gnu15/mvapich2)
Outcome:Passed
Duration:0.061 sec
Test case:[libs/NetCDF] verify nc4 available for C++ interface (gnu15/mvapich2)
Outcome:Skipped
Duration:0.0 sec
Skipped
option no longer supported
Test case:[libs/NetCDF] C write/read (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.589 sec
Test case:[libs/NetCDF] Fortran write/read (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.762 sec
Test case:[libs/NetCDF] C++ write/read (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.601 sec

Test Suite: test_C_module

Results

Duration0.405 sec
Tests9
Failures0

Tests

test_C_module

Test case:[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.207 sec
Test case:[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify availability of nc-config binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.049 sec
Test case:[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec

Test Suite: test_CXX_module

Results

Duration0.431 sec
Tests9
Failures0

Tests

test_CXX_module

Test case:[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.236 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.047 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_Fortran_module

Results

Duration0.404 sec
Tests9
Failures0

Tests

test_Fortran_module

Test case:[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.211 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.045 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: test_netcdf

Results

Duration3.274 sec
Tests8
Failures0

Tests

test_netcdf

Test case:[libs/NetCDF] ncdump availability (gnu15/openmpi5)
Outcome:Passed
Duration:0.139 sec
Test case:[libs/NetCDF] nctest availability (gnu15/openmpi5)
Outcome:Passed
Duration:0.345 sec
Test case:[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu15/openmpi5)
Outcome:Passed
Duration:0.082 sec
Test case:[libs/NetCDF] verify nc4 available for Fortran interface (gnu15/openmpi5)
Outcome:Passed
Duration:0.063 sec
Test case:[libs/NetCDF] verify nc4 available for C++ interface (gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
option no longer supported
Test case:[libs/NetCDF] C write/read (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.814 sec
Test case:[libs/NetCDF] Fortran write/read (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.983 sec
Test case:[libs/NetCDF] C++ write/read (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.848 sec

Test Suite: test_pnetcdf

Results

Duration1.325 sec
Tests2
Failures0

Tests

test_pnetcdf

Test case:[libs/NetCDF] C parallel I/O (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.325 sec
Test case:[libs/NetCDF] Fortran parallel I/O (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
Skipping fortran test bindings with mpich

Test Suite: test_pnetcdf

Results

Duration3.326 sec
Tests2
Failures1

Tests

test_pnetcdf

Test case:[libs/NetCDF] C parallel I/O (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.326 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_pnetcdf, line 47)
  `run_mpi_binary ./C_parallel "atest" 2 4' failed
job script = /tmp/job.ohpc-test.11518
Batch job 273 submitted

Job 273 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_parallel atest (family=mvapich2)
[1776236614.421847] [c1:32020:0]        ib_iface.c:1011 UCX  ERROR iface=0x306438d0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(1010952463) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff584d7e3c, argv=0x7fff584d7e30) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236614.430951] [c2:15084:0]        ib_iface.c:1011 UCX  ERROR iface=0x2c7937d0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffca40a871c, argv=0x7ffca40a8710) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/NetCDF] Fortran parallel I/O (slurm/gnu15/mvapich2)
Outcome:Skipped
Duration:0.0 sec
Skipped
Skipping fortran test bindings with mvapich2

Test Suite: test_pnetcdf

Results

Duration9.147 sec
Tests2
Failures0

Tests

test_pnetcdf

Test case:[libs/NetCDF] C parallel I/O (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.527 sec
Test case:[libs/NetCDF] Fortran parallel I/O (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:5.62 sec

Test Suite: rm_execution

Results

Duration4.017 sec
Tests11
Failures0

Tests

rm_execution

Test case:[libs/OpenBLAS/dblat1] dblat1 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.131 sec
Test case:[libs/OpenBLAS/xccblat1] xccblat1 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.166 sec
Test case:[libs/OpenBLAS/xzcblat1] xzcblat1 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.176 sec
Test case:[libs/OpenBLAS/xscblat2] xscblat2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.384 sec
Test case:[libs/OpenBLAS/xdcblat2] xdcblat2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.372 sec
Test case:[libs/OpenBLAS/xccblat2] xccblat2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.846 sec
Test case:[libs/OpenBLAS/xzcblat2] xzcblat2 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.661 sec
Test case:[libs/OpenBLAS/xscblat3] xscblat3 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.176 sec
Test case:[libs/OpenBLAS/xdcblat3] xdcblat3 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.197 sec
Test case:[libs/OpenBLAS/xccblat3] xccblat3 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.213 sec
Test case:[libs/OpenBLAS/xzcblat3] xzcblat3 under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.695 sec

Test Suite: test_lapack

Results

Duration0.286 sec
Tests1
Failures0

Tests

test_lapack

Test case:[libs/OpenBLAS/eigen] run lapack eigen-value solver (gnu15)
Outcome:Passed
Duration:0.286 sec

Test Suite: test_module

Results

Duration0.271 sec
Tests5
Failures0

Tests

test_module

Test case:[libs/OpenBLAS] Verify OPENBLAS module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.192 sec
Test case:[libs/OpenBLAS] Verify OPENBLAS_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/OpenBLAS] Verify module OPENBLAS_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.017 sec
Test case:[libs/OpenBLAS] Verify dynamic library available in OPENBLAS_LIB (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/OpenBLAS] Verify static library is not present in OPENBLAS_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration6.668 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/PETSc] C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.225 sec
Test case:[libs/PETSc] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.222 sec
Test case:[libs/PETSc] MPI F90 binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.221 sec

Test Suite: test_module

Results

Duration2.595 sec
Tests8
Failures0

Tests

test_module

Test case:[libs/PETSc] Verify PETSC module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.226 sec
Test case:[libs/PETSc] Verify module PETSC_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PETSc] Verify module PETSC_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Verify dynamic library available in PETSC_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PETSc] Verify static library is not present in PETSC_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Verify module PETSC_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PETSc] Verify header file is present in PETSC_INC (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Sample job (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.24 sec

Test Suite: rm_execution

Results

Duration5.684 sec
Tests3
Failures3

Tests

rm_execution

Test case:[libs/PETSc] C binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.192 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 27)
  `run_mpi_binary ./C_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.28853
Batch job 304 submitted

Job 304 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_test 8 (family=mvapich2)
[1776236737.393273] [c1:34406:0]        ib_iface.c:1011 UCX  ERROR iface=0x1885e400: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236737.393242] [c1:34407:0]        ib_iface.c:1011 UCX  ERROR iface=0x27906400: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(809626127) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffccd96980c, argv=0x7ffccd969800, required=1, provided=0x7ffccd9693bc) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(205646351) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff38c2e7dc, argv=0x7fff38c2e7d0, required=1, provided=0x7fff38c2e38c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236737.399092] [c2:16494:0]        ib_iface.c:1011 UCX  ERROR iface=0x1c2d2f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236737.399217] [c2:16495:0]        ib_iface.c:1011 UCX  ERROR iface=0x3eda82f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339864079) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff7db8963c, argv=0x7fff7db89630, required=1, provided=0x7fff7db891ec) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(272755215) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc89602a2c, argv=0x7ffc89602a20, required=1, provided=0x7ffc896025dc) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236737.409587] [c2:16493:0]        ib_iface.c:1011 UCX  ERROR iface=0x27b282f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(1010952719) on node 5: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff89689f9c, argv=0x7fff89689f90, required=1, provided=0x7fff89689b4c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/PETSc] MPI C binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.246 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 36)
  `run_mpi_binary ./C_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.23475
Batch job 305 submitted

Job 305 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_mpi_test 8 (family=mvapich2)
[1776236738.005393] [c2:16582:0]        ib_iface.c:1011 UCX  ERROR iface=0xc8762f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236738.005639] [c2:16581:0]        ib_iface.c:1011 UCX  ERROR iface=0x37c852f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(272755215) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff1dbb770c, argv=0x7fff1dbb7700, required=1, provided=0x7fff1dbb72ac) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(541190671) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc1acbf72c, argv=0x7ffc1acbf720, required=1, provided=0x7ffc1acbf2cc) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236738.010085] [c1:34586:0]        ib_iface.c:1011 UCX  ERROR iface=0x356b0400: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236738.010806] [c1:34584:0]        ib_iface.c:1011 UCX  ERROR iface=0x803f400: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(272755215) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd0aee81bc, argv=0x7ffd0aee81b0, required=1, provided=0x7ffd0aee7d5c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(138537487) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd37c8286c, argv=0x7ffd37c82860, required=1, provided=0x7ffd37c8240c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/PETSc] MPI F90 binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.246 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 45)
  `run_mpi_binary ./F90_test 2 "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.15329
Batch job 306 submitted

Job 306 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F90_test 2 (family=mvapich2)
[1776236738.569433] [c1:34765:0]        ib_iface.c:1011 UCX  ERROR iface=0x1907b3c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236738.569568] [c1:34762:0]        ib_iface.c:1011 UCX  ERROR iface=0x3df9d3c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236738.577082] [c2:16668:0]        ib_iface.c:1011 UCX  ERROR iface=0x355181b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236738.578485] [c2:16669:0]        ib_iface.c:1011 UCX  ERROR iface=0x3c1661b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 34765 RUNNING AT c1
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

Test Suite: test_module

Results

Duration2.608 sec
Tests8
Failures1

Tests

test_module

Test case:[libs/PETSc] Verify PETSC module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.224 sec
Test case:[libs/PETSc] Verify module PETSC_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/PETSc] Verify module PETSC_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PETSc] Verify dynamic library available in PETSC_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PETSc] Verify static library is not present in PETSC_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PETSc] Verify module PETSC_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Verify header file is present in PETSC_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PETSc] Sample job (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.261 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file test_module, line 132)
  `run_mpi_binary ./C_test "atest" 1 1' failed
job script = /tmp/job.ohpc-test.19216
Batch job 303 submitted

Job 303 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_test atest (family=mvapich2)
[1776236734.441049] [c1:34231:0]        ib_iface.c:1011 UCX  ERROR iface=0x2996c3f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(272755215) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd546ea22c, argv=0x7ffd546ea220, required=1, provided=0x7ffd546e9ddc) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: rm_execution

Results

Duration14.056 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/PETSc] C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.279 sec
Test case:[libs/PETSc] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.335 sec
Test case:[libs/PETSc] MPI F90 binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.442 sec

Test Suite: test_module

Results

Duration3.64 sec
Tests8
Failures0

Tests

test_module

Test case:[libs/PETSc] Verify PETSC module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.225 sec
Test case:[libs/PETSc] Verify module PETSC_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PETSc] Verify module PETSC_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PETSc] Verify dynamic library available in PETSC_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/PETSc] Verify static library is not present in PETSC_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PETSc] Verify module PETSC_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PETSc] Verify header file is present in PETSC_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PETSc] Sample job (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.295 sec

Test Suite: rm_execution

Results

Duration3.397 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/PHDF5] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.174 sec
Test case:[libs/PHDF5] Parallel Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec

Test Suite: test_module

Results

Duration0.336 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/PHDF5] Verify HDF5 module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.216 sec
Test case:[libs/PHDF5] Verify HDF5_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify module HDF5_LIB is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PHDF5] Verify dynamic library available in HDF5_LIB (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify static library is not present in HDF5_LIB (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify module HDF5_INC is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PHDF5] Verify header file is present in HDF5_INC (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec

Test Suite: rm_execution

Results

Duration3.438 sec
Tests2
Failures2

Tests

rm_execution

Test case:[libs/PHDF5] MPI C binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.196 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 43)
  `run_mpi_binary -t "${TIMEOUT}" ./C_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.12576
Batch job 313 submitted

Job 313 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_mpi_test /home/ohpc-test/phdf5_test.AMt5WF/phdf_test.out (family=mvapich2)
[1776236785.414032] [c2:17231:0]        ib_iface.c:1011 UCX  ERROR iface=0x32e8fc10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236785.414028] [c2:17232:0]        ib_iface.c:1011 UCX  ERROR iface=0xf3ecc10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236785.415183] [c1:36052:0]        ib_iface.c:1011 UCX  ERROR iface=0x245dfd20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236785.415734] [c1:36051:0]        ib_iface.c:1011 UCX  ERROR iface=0x7fe8d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428367) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff079091cc, argv=0x7fff079091c0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(71428367) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe9686e40c, argv=0x7ffe9686e400) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(1010952463) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff6a1fb6fc, argv=0x7fff6a1fb6f0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/PHDF5] Parallel Fortran binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.242 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 52)
  `run_mpi_binary -t "${TIMEOUT}" ./F_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.18114
Batch job 314 submitted

Job 314 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_mpi_test /home/ohpc-test/phdf5_test.AMt5WF/phdf_test.out (family=mvapich2)
[1776236786.022950] [c2:17320:0]        ib_iface.c:1011 UCX  ERROR iface=0x3f699120: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236786.021986] [c1:36229:0]        ib_iface.c:1011 UCX  ERROR iface=0x176d3330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776236786.022682] [c1:36228:0]        ib_iface.c:1011 UCX  ERROR iface=0x1dff1330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236786.023646] [c2:17317:0]        ib_iface.c:1011 UCX  ERROR iface=0x261f3120: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776236786.032264] [c1:36230:0]        ib_iface.c:1011 UCX  ERROR iface=0x18004330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument

Test Suite: test_module

Results

Duration0.324 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/PHDF5] Verify HDF5 module is loaded and matches rpm version (gnu15-mvapich2)
Outcome:Passed
Duration:0.199 sec
Test case:[libs/PHDF5] Verify HDF5_DIR is defined and directory exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify module HDF5_LIB is defined and exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/PHDF5] Verify dynamic library available in HDF5_LIB (gnu15-mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PHDF5] Verify static library is not present in HDF5_LIB (gnu15-mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PHDF5] Verify module HDF5_INC is defined and exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PHDF5] Verify header file is present in HDF5_INC (gnu15-mvapich2)
Outcome:Passed
Duration:0.02 sec

Test Suite: rm_execution

Results

Duration8.652 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/PHDF5] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.278 sec
Test case:[libs/PHDF5] Parallel Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:5.374 sec

Test Suite: test_module

Results

Duration0.322 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/PHDF5] Verify HDF5 module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.201 sec
Test case:[libs/PHDF5] Verify HDF5_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify module HDF5_LIB is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify dynamic library available in HDF5_LIB (gnu15-openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify static library is not present in HDF5_LIB (gnu15-openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PHDF5] Verify module HDF5_INC is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PHDF5] Verify header file is present in HDF5_INC (gnu15-openmpi5)
Outcome:Passed
Duration:0.019 sec

Test Suite: rm_execution

Results

Duration1.963 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/PLASMA/C_test] C_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.802 sec
Test case:[libs/PLASMA/F90_test] F90_test under resource manager (slurm/gnu15)
Outcome:Passed
Duration:1.161 sec

Test Suite: test_module

Results

Duration0.315 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/PLASMA] Verify PLASMA module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.192 sec
Test case:[libs/PLASMA] Verify PLASMA_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PLASMA] Verify module PLASMA_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PLASMA] Verify dynamic library available in PLASMA_LIB (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PLASMA] Verify static library is not present in PLASMA_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PLASMA] Verify module PLASMA_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PLASMA] Verify header file is present in PLASMA_INC (gnu15)
Outcome:Passed
Duration:0.02 sec

Test Suite: test_module

Results

Duration0.245 sec
Tests5
Failures0

Tests

test_module

Test case:[PNETCDF] Verify PNETCDF module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.206 sec
Test case:[PNETCDF] Verify module PNETCDF_DIR is defined and exists (gnu15)
Outcome:Passed
Duration:0.011 sec
Test case:[PNETCDF] Verify module PNETCDF_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.01 sec
Test case:[PNETCDF] Verify module PNETCDF_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.009 sec
Test case:[PNETCDF] Verify header file is present in PNETCDF_INC (gnu15)
Outcome:Passed
Duration:0.009 sec

Test Suite: rm_execution

Results

Duration8.889 sec
Tests4
Failures0

Tests

rm_execution

Test case:[libs/PNETCDF] Parallel Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.171 sec
Test case:[libs/PNETCDF] Parallel Fortran binary 2 runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.27 sec
Test case:[libs/PNETCDF] Parallel Fortran binary 3 runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.224 sec
Test case:[libs/PNETCDF] Parallel Fortran binary 4 runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.224 sec

Test Suite: rm_execution

Results

Duration7.921 sec
Tests4
Failures4

Tests

rm_execution

Test case:[libs/PNETCDF] Parallel Fortran binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.193 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 22)
  `run_mpi_binary ./f90tst_parallel f90tst_parallel.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.32628
Batch job 323 submitted

Job 323 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./f90tst_parallel f90tst_parallel.nc (family=mvapich2)
[1776236893.459527] [c1:37564:0]        ib_iface.c:1011 UCX  ERROR iface=0x1b47c50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/PNETCDF] Parallel Fortran binary 2 runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.244 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 31)
  `run_mpi_binary ./f90tst_parallel2 f90tst_parallel2.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.14654
Batch job 324 submitted

Job 324 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./f90tst_parallel2 f90tst_parallel2.nc (family=mvapich2)
[1776236895.364027] [c1:37739:0]        ib_iface.c:1011 UCX  ERROR iface=0x4f06c50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/PNETCDF] Parallel Fortran binary 3 runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.244 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 40)
  `run_mpi_binary ./f90tst_parallel3 f90tst_parallel3.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.29301
Batch job 325 submitted

Job 325 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./f90tst_parallel3 f90tst_parallel3.nc (family=mvapich2)
[1776236897.357586] [c1:37912:0]        ib_iface.c:1011 UCX  ERROR iface=0x14347c50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/PNETCDF] Parallel Fortran binary 4 runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.24 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 49)
  `run_mpi_binary ./f90tst_parallel4 f90tst_parallel4.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.20648
Batch job 326 submitted

Job 326 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./f90tst_parallel4 f90tst_parallel4.nc (family=mvapich2)
[1776236900.404354] [c1:38086:0]        ib_iface.c:1011 UCX  ERROR iface=0x2b46cc50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: rm_execution

Results

Duration10.989 sec
Tests4
Failures0

Tests

rm_execution

Test case:[libs/PNETCDF] Parallel Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.224 sec
Test case:[libs/PNETCDF] Parallel Fortran binary 2 runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.273 sec
Test case:[libs/PNETCDF] Parallel Fortran binary 3 runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.27 sec
Test case:[libs/PNETCDF] Parallel Fortran binary 4 runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.222 sec

Test Suite: rm_execution

Results

Duration5.614 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/PTScotch] dgraph_redist binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.169 sec
Test case:[libs/PTScotch] strat_par binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec
Test case:[libs/PTScotch] dgord binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.222 sec

Test Suite: test_module

Results

Duration0.372 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/PTScotch] Verify PTSCOTCH module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.197 sec
Test case:[libs/PTScotch] Verify PTSCOTCH_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_LIB is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify dynamic library available in PTSCOTCH_LIB (gnu15-mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify static library is not present in PTSCOTCH_LIB (gnu15-mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_INC is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify header file is present in PTSCOTCH_INC (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_BIN is defined and exists (gnu15-mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify availability of dgscat binary (gnu15-mpich)
Outcome:Passed
Duration:0.031 sec

Test Suite: rm_execution

Results

Duration4.612 sec
Tests3
Failures2

Tests

rm_execution

Test case:[libs/PTScotch] dgraph_redist binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.193 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 27)
  `run_mpi_binary -t "${TIMEOUT}" ./dgraph_redist "bump.grf" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.25806
Batch job 334 submitted

Job 334 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./dgraph_redist bump.grf (family=mvapich2)
[1776237005.362684] [c1:39565:0]        ib_iface.c:1011 UCX  ERROR iface=0x14258ff0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237005.364881] [c2:17990:0]        ib_iface.c:1011 UCX  ERROR iface=0x1f951e40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd7727e33c, argv=0x7ffd7727e330) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc7425d7ac, argv=0x7ffc7425d7a0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/PTScotch] strat_par binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:2.224 sec
Test case:[libs/PTScotch] dgord binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.195 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 45)
  `run_mpi_binary -t "${TIMEOUT}" dgord "bump.grf /dev/null -vt" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.26357
Batch job 336 submitted

Job 336 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm dgord bump.grf /dev/null -vt (family=mvapich2)
[1776237007.406847] [c2:18153:0]        ib_iface.c:1011 UCX  ERROR iface=0xfa23e40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(742517263) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc3cc71a2c, argv=0x7ffc3cc71a20, required=3, provided=0x7ffc3cc71a44) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237007.409895] [c1:39914:0]        ib_iface.c:1011 UCX  ERROR iface=0x1525cff0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428623) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc5066d0ac, argv=0x7ffc5066d0a0, required=3, provided=0x7ffc5066d0c4) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_module

Results

Duration0.372 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/PTScotch] Verify PTSCOTCH module is loaded and matches rpm version (gnu15-mvapich2)
Outcome:Passed
Duration:0.199 sec
Test case:[libs/PTScotch] Verify PTSCOTCH_DIR is defined and directory exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_LIB is defined and exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify dynamic library available in PTSCOTCH_LIB (gnu15-mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify static library is not present in PTSCOTCH_LIB (gnu15-mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_INC is defined and exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify header file is present in PTSCOTCH_INC (gnu15-mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_BIN is defined and exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/PTScotch] Verify availability of dgscat binary (gnu15-mvapich2)
Outcome:Passed
Duration:0.029 sec

Test Suite: rm_execution

Results

Duration9.831 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/PTScotch] dgraph_redist binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.225 sec
Test case:[libs/PTScotch] strat_par binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.276 sec
Test case:[libs/PTScotch] dgord binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.33 sec

Test Suite: test_module

Results

Duration0.374 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/PTScotch] Verify PTSCOTCH module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.2 sec
Test case:[libs/PTScotch] Verify PTSCOTCH_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_LIB is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/PTScotch] Verify dynamic library available in PTSCOTCH_LIB (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify static library is not present in PTSCOTCH_LIB (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_INC is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/PTScotch] Verify header file is present in PTSCOTCH_INC (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify module PTSCOTCH_BIN is defined and exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/PTScotch] Verify availability of dgscat binary (gnu15-openmpi5)
Outcome:Passed
Duration:0.031 sec

Test Suite: rm_execution

Results

Duration8.886 sec
Tests4
Failures0

Tests

rm_execution

Test case:[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.173 sec
Test case:[libs/ScaLAPACK/PDSCAEX] DPCGESV under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.217 sec
Test case:[libs/ScaLAPACK/PSSCAEX] SPCGESV under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.221 sec
Test case:[libs/ScaLAPACK/PZSCAEX] ZPCGESV under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.275 sec

Test Suite: test_module

Results

Duration0.27 sec
Tests5
Failures0

Tests

test_module

Test case:[libs/ScaLAPACK] Verify SCALAPACK module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.189 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/ScaLAPACK] Verify dynamic library available in SCALAPACK_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/ScaLAPACK] Verify static library is not present in SCALAPACK_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec

Test Suite: rm_execution

Results

Duration15.278 sec
Tests4
Failures4

Tests

rm_execution

Test case:[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.297 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 35)
  `run_mpi_binary ./pcscaex "${ARGS}" "${NODES}" "${TASKS}"' failed
run_mpi_binary ./pcscaex 0 2 8
SHELL=/bin/bash
BATS_RUN_LOGS_DIRECTORY=
BATS_TEST_TMPDIR=/tmp/bats-run-hbwgwc/test/1
SUDO_GID=0
HISTCONTROL=ignoredups
OMPI_MCA_rmaps_base_oversubscribe=1
PARALLEL_JOBSLOT0=0
PKG_CONFIG_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib/pkgconfig:/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/lib/pkgconfig
HOSTNAME=ohpc-lenovo-sms
HISTSIZE=1048576
FPATH=/opt/ohpc/admin/lmod/lmod/init/ksh_funcs
BATS_LIBDIR=lib
BATS_SUITE_TMPDIR=/tmp/bats-run-hbwgwc/suite
BATS_ROOT_PID=380896
BATS_WARNING_FILE=/tmp/bats-run-hbwgwc/warnings.log
BATS_FILE_TMPDIR=/tmp/bats-run-hbwgwc/file/0
_ModuleTable002_=dGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAiZ251MTUiLAp3ViA9ICIwMDAwMDAwMTUuMDAwMDAwMDAyLip6ZmluYWwiLAp9LAptdmFwaWNoMiA9IHsKYWN0aW9uQSA9IHsKW1twcmVwZW5kX3BhdGgoIk1PRFVMRVBBVEgiLCIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUtbXZhcGljaDIiKV1dLAp9LApmbiA9ICIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUvbXZhcGljaDIvNC4xIiwKZnVsbE5hbWUgPSAibXZhcGljaDIvNC4xIiwKbG9hZE9yZGVyID0gMywKcHJvcFQgPSB7fSwKc3RhY2tEZXB0aCA9IDAsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJtdmFwaWNoMiIsCndWID0gIjAwMDAwMDAwNC4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCm9w
BATS_SEMAPHORE_DIR=/tmp/bats-run-hbwgwc/semaphores
__LMOD_REF_COUNT_MODULEPATH=/opt/ohpc/pub/moduledeps/gnu15-mvapich2:1;/opt/ohpc/pub/moduledeps/gnu15:1;/opt/ohpc/pub/modulefiles:1
_ModuleTable005_=LAp9LAp9LAptcGF0aEEgPSB7CiIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUtbXZhcGljaDIiLCAiL29wdC9vaHBjL3B1Yi9tb2R1bGVkZXBzL2dudTE1IiwgIi9vcHQvb2hwYy9wdWIvbW9kdWxlZmlsZXMiLAp9LApzeXN0ZW1CYXNlTVBBVEggPSAiL29wdC9vaHBjL3B1Yi9tb2R1bGVmaWxlcyIsCn0K
TEST_LOGS=test_module.log rm_execution.log
SSHPASS=
BATS_END_CODE_QUOTE='
SUDO_COMMAND=/bin/bash -c /tmp/user_integration_tests
SUDO_USER=root
BATS_TEST_SOURCE=/home/ohpc-test/tests/libs/scalapack/tests/rm_execution
BATS_TEST_NAME=test_-5b-24TESTNAME-2fPCSCAEX-5d_CPCGESV_under_resource_manager_-28-24RESOURCE-5fMANAGER-2f-24LMOD-5fFAMILY-5fCOMPILER-2f-24LMOD-5fFAMILY-5fMPI-29
GPG_TTY=not a tty
LMOD_DIR=/opt/ohpc/admin/lmod/lmod/libexec
LMOD_FAMILY_OPENBLAS_VERSION=0.3.32
PWD=/home/ohpc-test/tests/libs/scalapack/tests
LOGNAME=ohpc-test
MODULESHOME=/opt/ohpc/admin/lmod/lmod
PARALLEL_TMP=/tmp/pary5Rcq
MANPATH=/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/share/man:/opt/ohpc/pub/compiler/gcc/15.2.0/share/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:/opt/ohpc/admin/lmod/lmod/share/man::
OPENBLAS_INC=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/include
BATS_JUNIT_FORMAT=1
CI_CLUSTER=lenovo
LMOD_PREPEND_BLOCK=normal
BATS_TMPDIR=/tmp
__LMOD_REF_COUNT_INCLUDE=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/include:1;/opt/ohpc/pub/compiler/gcc/15.2.0/include:1
BATS_USE_MKTEMP=1
BATS_TEST_DESCRIPTION=[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu15/mvapich2)
__LMOD_REF_COUNT_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/bin:1;/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/bin:1;/opt/ohpc/pub/compiler/gcc/15.2.0/bin:1;/opt/ohpc/pub/utils/prun/2.2:1;/home/ohpc-test/.local/bin:1;/home/ohpc-test/bin:1;/sbin:1;/bin:1;/usr/sbin:1;/usr/bin:1;/usr/local/sbin:1
HOME=/home/ohpc-test
FI_PROVIDER=sockets
ARGS=0
PARALLEL_RSYNC_OPTS=-rlDzR
_ModuleTable_Sz_=5
LANG=en_US.UTF-8
BATS_TEST_FILENAME=/home/ohpc-test/tests/libs/scalapack/tests/rm_execution
__LMOD_REF_COUNT_PKG_CONFIG_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib/pkgconfig:1;/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/lib/pkgconfig:1
SCALAPACK_LIB=/opt/ohpc/pub/libs/gnu15/mvapich2/scalapack/2.2.3/lib
srcdir=.
PARALLEL_PID=380957
BATS_RUN_TMPDIR=/tmp/bats-run-hbwgwc
LMOD_SETTARG_FULL_SUPPORT=no
BATS_JUNIT_CLASS=scalapack
BATS_VERSION=1.11.1
TMPDIR=/tmp
https_proxy=http://10.241.58.130:3128
OPENBLAS_DIR=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32
LMOD_VERSION=9.2
BATS_GUARANTEED_MINIMUM_VERSION=0.0.0
TESTNAME=libs/ScaLAPACK
SINGULARITY_TMPDIR=/var/tmp
MFLAGS=-k --no-print-directory
_ModuleTable003_=ZW5ibGFzID0gewpmbiA9ICIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUvb3BlbmJsYXMvMC4zLjMyIiwKZnVsbE5hbWUgPSAib3BlbmJsYXMvMC4zLjMyIiwKbG9hZE9yZGVyID0gNCwKcHJvcFQgPSB7fSwKcmVmX2NvdW50ID0gMSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJvcGVuYmxhcyIsCndWID0gIjAwMDAwMDAwMC4wMDAwMDAwMDMuMDAwMDAwMDMyLip6ZmluYWwiLAp9LApwcnVuID0gewpmbiA9ICIvb3B0L29ocGMvcHViL21vZHVsZWZpbGVzL3BydW4vMi4yIiwKZnVsbE5hbWUgPSAicHJ1bi8yLjIiLApsb2FkT3JkZXIgPSAxLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVz
PARALLEL_SEQ=1
__LMOD_REF_COUNT_MPI_DIR=/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1:1
AUTOMAKE_JUNIT_FILE=1
LMOD_FAMILY_OPENBLAS=openblas
BATS_CWD=/home/ohpc-test/tests/libs/scalapack/tests
MAKEFLAGS=k --no-print-directory -- TEST_LOGS=test_module.log\ rm_execution.log
LMOD_PKG=/opt/ohpc/admin/lmod/lmod
TERM=unknown
BATS_LINE_REFERENCE_FORMAT=comma_line
BATS_SEMAPHORE_NUMBER_OF_SLOTS=4
LESSOPEN=||/usr/bin/lesspipe.sh %s
PARALLEL_SSHLOGIN=:
USER=ohpc-test
BATS_BEGIN_CODE_QUOTE=`
NODES=2
LOADEDMODULES=prun/2.2:gnu15/15.2.0:mvapich2/4.1:openblas/0.3.32:scalapack/2.2.3
BATS_TEMPDIR_CLEANUP=1
INCLUDE=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/include:/opt/ohpc/pub/compiler/gcc/15.2.0/include
LMOD_ROOT=/opt/ohpc/admin/lmod
LMOD_SETTARG_CMD=:
TASKS=8
SHLVL=10
BASH_ENV=/opt/ohpc/admin/lmod/lmod/init/bash
BATS_REPORT_FILENAME=rm_execution.log.xml
MAKELEVEL=8
LMOD_FULL_SETTARG_SUPPORT=no
MPI_DIR=/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1
__LMOD_REF_COUNT_MANPATH=/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/share/man:1;/opt/ohpc/pub/compiler/gcc/15.2.0/share/man:1;/usr/local/share/man:1;/usr/share/man/overrides:1;/usr/share/man/en:1;/usr/share/man:1;/opt/ohpc/admin/lmod/lmod/share/man:1;:1
http_proxy=http://10.241.58.130:3128
PARALLEL_SSHHOST=:
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpNUEkgPSAibXZhcGljaDIiLApjb21waWxlciA9ICJnbnUxNSIsCm9wZW5ibGFzID0gIm9wZW5ibGFzIiwKfSwKbVQgPSB7CmdudTE1ID0gewphY3Rpb25BID0gewpbW3ByZXBlbmRfcGF0aCgiTU9EVUxFUEFUSCIsIi9vcHQvb2hwYy9wdWIvbW9kdWxlZGVwcy9nbnUxNSIpXV0sCn0sCmZuID0gIi9vcHQvb2hwYy9wdWIvbW9kdWxlZmlsZXMvZ251MTUvMTUuMi4wLmx1YSIsCmZ1bGxOYW1lID0gImdudTE1LzE1LjIuMCIsCmxvYWRPcmRlciA9IDIsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApz
LD_LIBRARY_PATH=/opt/ohpc/pub/libs/gnu15/mvapich2/scalapack/2.2.3/lib:/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib:/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/lib:/opt/ohpc/pub/compiler/gcc/15.2.0/lib64
LMOD_FAMILY_COMPILER=gnu15
LMOD_COLORIZE=no
_ModuleTable004_=ZXJOYW1lID0gInBydW4iLAp3ViA9ICIwMDAwMDAwMDIuMDAwMDAwMDAyLip6ZmluYWwiLAp9LApzY2FsYXBhY2sgPSB7CmRlcFQgPSB7CmRlcEEgPSB7CnsKc24gPSAib3BlbmJsYXMiLAp2ZXJzaW9uID0gewpraW5kID0gImJvb2wiLAp2YWx1ZSA9IHRydWUsCn0sCn0sCn0sCn0sCmZuID0gIi9vcHQvb2hwYy9wdWIvbW9kdWxlZGVwcy9nbnUxNS1tdmFwaWNoMi9zY2FsYXBhY2svMi4yLjMiLApmdWxsTmFtZSA9ICJzY2FsYXBhY2svMi4yLjMiLApsb2FkT3JkZXIgPSA1LApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gInNjYWxhcGFjayIsCndWID0gIjAwMDAwMDAwMi4wMDAwMDAwMDIuMDAwMDAwMDAzLip6ZmluYWwi
LMOD_FAMILY_MPI_VERSION=4.1
RESOURCE_MANAGER=slurm
BATS_TEST_RETRIES=0
OMP_NUM_THREADS=2
LMOD_FAMILY_MPI=mvapich2
DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima:
BATS_RUNLOG_FILE=/dev/null
LMOD_FAMILY_COMPILER_VERSION=15.2.0
PARALLEL_REMOTE_TMPDIR=/tmp
BATS_TEST_NUMBER=1
BATS_ENABLE_TIMING=-T
PATH=/usr/libexec/bats-core:/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/bin:/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/bin:/opt/ohpc/pub/compiler/gcc/15.2.0/bin:/opt/ohpc/pub/utils/prun/2.2:/home/ohpc-test/.local/bin:/home/ohpc-test/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin
MODULEPATH=/opt/ohpc/pub/moduledeps/gnu15-mvapich2:/opt/ohpc/pub/moduledeps/gnu15:/opt/ohpc/pub/modulefiles
SCALAPACK_DIR=/opt/ohpc/pub/libs/gnu15/mvapich2/scalapack/2.2.3
_LMFILES_=/opt/ohpc/pub/modulefiles/prun/2.2:/opt/ohpc/pub/modulefiles/gnu15/15.2.0.lua:/opt/ohpc/pub/moduledeps/gnu15/mvapich2/4.1:/opt/ohpc/pub/moduledeps/gnu15/openblas/0.3.32:/opt/ohpc/pub/moduledeps/gnu15-mvapich2/scalapack/2.2.3
SUDO_UID=0
LMOD_CMD=/opt/ohpc/admin/lmod/lmod/libexec/lmod
MAIL=/var/spool/mail/ohpc-test
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/opt/ohpc/pub/libs/gnu15/mvapich2/scalapack/2.2.3/lib:1;/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib:1;/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/lib:1;/opt/ohpc/pub/compiler/gcc/15.2.0/lib64:1
BATS_JUNIT_GROUP=UserLevelTests
BATS_LIB_PATH=/usr/lib/bats
OPENBLAS_LIB=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib
PARALLEL_JOBSLOT=1
BATS_LIBEXEC=/usr/libexec/bats-core
BATS_SUITE_TEST_NUMBER=1
BATS_ROOT=/usr
OLDPWD=/home/ohpc-test/tests/libs/scalapack/tests
VERBOSE=1
BATS_SAVED_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/bin:/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/bin:/opt/ohpc/pub/compiler/gcc/15.2.0/bin:/opt/ohpc/pub/utils/prun/2.2:/home/ohpc-test/.local/bin:/home/ohpc-test/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin
BASH_FUNC_ml%%=() {  eval "$($LMOD_DIR/ml_cmd "$@")"
}
BASH_FUNC_bats_semaphore_acquire_while_locked%%=() {  if [[ $(bats_semaphore_get_free_slot_count) -gt 0 ]]; then
 local slot=0;
 while [[ -e "$BATS_SEMAPHORE_DIR/slot-$slot" ]]; do
 ((++slot));
 done;
 if [[ $slot -lt $BATS_SEMAPHORE_NUMBER_OF_SLOTS ]]; then
 touch "$BATS_SEMAPHORE_DIR/slot-$slot" && printf "%d\n" "$slot" && return 0;
 fi;
 fi;
 return 1
}
BASH_FUNC_bats_readlinkf%%=() {  readlink -f "$1"
}
BASH_FUNC_bats_semaphore_get_free_slot_count%%=() {  until used_slots=$(find "$BATS_SEMAPHORE_DIR" -name 'slot-*' 2> /dev/null | wc -l); do
 :;
 done;
 echo $((BATS_SEMAPHORE_NUMBER_OF_SLOTS - used_slots))
}
BASH_FUNC_module%%=() {  if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
 case "$-" in
 *v*x*)
 __lmod_sh_dbg='vx'
 ;;
 *v*)
 __lmod_sh_dbg='v'
 ;;
 *x*)
 __lmod_sh_dbg='x'
 ;;
 esac;
 fi;
 if [ -n "${__lmod_sh_dbg:-}" ]; then
 set +$__lmod_sh_dbg;
 echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output" 1>&2;
 fi;
 eval "$($LMOD_CMD shell "$@")" && eval "$(${LMOD_SETTARG_CMD:-:} -s sh)";
 __lmod_my_status=$?;
 if [ -n "${__lmod_sh_dbg:-}" ]; then
 echo "Shell debugging restarted" 1>&2;
 set -$__lmod_sh_dbg;
 fi;
 unset __lmod_sh_dbg;
 return $__lmod_my_status
}
_=/bin/printenv
job script = /tmp/job.ohpc-test.23372
Batch job 345 submitted

Job 345 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pcscaex 0 (family=mvapich2)
[1776237084.455493] [c2:18932:0]        ib_iface.c:1011 UCX  ERROR iface=0x315301b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc16ac7aac, argv=0x7ffc16ac7ab0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237084.460879] [c1:41520:0]        ib_iface.c:1011 UCX  ERROR iface=0x2d7503c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(541190415) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffee542126c, argv=0x7ffee5421270) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237084.464020] [c1:41521:0]        ib_iface.c:1011 UCX  ERROR iface=0x87b53c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 18931 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c1: task 0: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion
Test case:[libs/ScaLAPACK/PDSCAEX] DPCGESV under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.293 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 44)
  `run_mpi_binary ./pdscaex "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.19768
Batch job 346 submitted

Job 346 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pdscaex 0 (family=mvapich2)
[1776237086.010790] [c2:19020:0]        ib_iface.c:1011 UCX  ERROR iface=0x21afa1b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237086.010929] [c2:19018:0]        ib_iface.c:1011 UCX  ERROR iface=0x180941b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(541190415) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd28577e8c, argv=0x7ffd28577e90) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(138537231) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fffb71ec03c, argv=0x7fffb71ec040) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237086.017710] [c1:41697:0]        ib_iface.c:1011 UCX  ERROR iface=0x3f733c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237086.020242] [c1:41699:0]        ib_iface.c:1011 UCX  ERROR iface=0x1f0923c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237086.028815] [c1:41698:0]        ib_iface.c:1011 UCX  ERROR iface=0xbc983c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(742517007) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffcb417b09c, argv=0x7ffcb417b0a0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 41697 RUNNING AT c1
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:1@c2] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:1@c2] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:1@c2] main (proxy/pmip.c:182): demux engine error waiting for event
srun: error: c2: task 1: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion
Test case:[libs/ScaLAPACK/PSSCAEX] SPCGESV under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.342 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 53)
  `run_mpi_binary ./psscaex "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.11113
Batch job 347 submitted

Job 347 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./psscaex 0 (family=mvapich2)
[1776237086.573790] [c1:41876:0]        ib_iface.c:1011 UCX  ERROR iface=0x332053c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237086.573872] [c1:41879:0]        ib_iface.c:1011 UCX  ERROR iface=0x8d063c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(541190415) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd85cde81c, argv=0x7ffd85cde820) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(4319503) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffeecc9d2fc, argv=0x7ffeecc9d300) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237086.585943] [c2:19104:0]        ib_iface.c:1011 UCX  ERROR iface=0x341b31b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237086.586786] [c2:19107:0]        ib_iface.c:1011 UCX  ERROR iface=0x775e1b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(406972687) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffddb9932dc, argv=0x7ffddb9932e0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(272754959) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fffdbe5662c, argv=0x7fffdbe56630) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/ScaLAPACK/PZSCAEX] ZPCGESV under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.346 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 62)
  `run_mpi_binary ./pzscaex "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.4524
Batch job 348 submitted

Job 348 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pzscaex 0 (family=mvapich2)
[1776237087.193830] [c1:42055:0]        ib_iface.c:1011 UCX  ERROR iface=0x3c3153c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237087.195093] [c1:42058:0]        ib_iface.c:1011 UCX  ERROR iface=0x12a923c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(4319503) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc10be078c, argv=0x7ffc10be0790) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(809625871) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe0da86fbc, argv=0x7ffe0da86fc0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237087.200229] [c2:19193:0]        ib_iface.c:1011 UCX  ERROR iface=0x6c1d1b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237087.200841] [c2:19191:0]        ib_iface.c:1011 UCX  ERROR iface=0x75541b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(272754959) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffda99738cc, argv=0x7ffda99738d0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(138537231) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc12bbc86c, argv=0x7ffc12bbc870) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_module

Results

Duration0.272 sec
Tests5
Failures0

Tests

test_module

Test case:[libs/ScaLAPACK] Verify SCALAPACK module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.192 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/ScaLAPACK] Verify dynamic library available in SCALAPACK_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/ScaLAPACK] Verify static library is not present in SCALAPACK_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec

Test Suite: rm_execution

Results

Duration22.578 sec
Tests4
Failures0

Tests

rm_execution

Test case:[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.278 sec
Test case:[libs/ScaLAPACK/PDSCAEX] DPCGESV under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.332 sec
Test case:[libs/ScaLAPACK/PSSCAEX] SPCGESV under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.435 sec
Test case:[libs/ScaLAPACK/PZSCAEX] ZPCGESV under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:8.533 sec

Test Suite: test_module

Results

Duration0.275 sec
Tests5
Failures0

Tests

test_module

Test case:[libs/ScaLAPACK] Verify SCALAPACK module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.196 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/ScaLAPACK] Verify module SCALAPACK_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/ScaLAPACK] Verify dynamic library available in SCALAPACK_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/ScaLAPACK] Verify static library is not present in SCALAPACK_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration0.574 sec
Tests3
Failures0

Tests

rm_execution

Test case:[libs/Scotch] graph_map binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.304 sec
Test case:[libs/Scotch] strat_seq binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.144 sec
Test case:[libs/Scotch] gout binary runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.126 sec

Test Suite: test_module

Results

Duration0.366 sec
Tests9
Failures0

Tests

test_module

Test case:[libs/Scotch] Verify SCOTCH module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.194 sec
Test case:[libs/Scotch] Verify SCOTCH_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/Scotch] Verify module SCOTCH_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/Scotch] Verify dynamic library available in SCOTCH_LIB (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/Scotch] Verify static library is not present in SCOTCH_LIB (gnu15)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/Scotch] Verify module SCOTCH_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/Scotch] Verify header file is present in SCOTCH_INC (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/Scotch] Verify module SCOTCH_BIN is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/Scotch] Verify availability of gout binary (gnu15)
Outcome:Passed
Duration:0.029 sec

Test Suite: rm_execution

Results

Duration7.847 sec
Tests4
Failures0

Tests

rm_execution

Test case:[libs/SLEPc] F90 SVD test binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.173 sec
Test case:[libs/SLEPc] C SVD of the Lauchli matrix binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec
Test case:[libs/SLEPc] F90 quadratic eigensystem with PEP object binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.226 sec
Test case:[libs/SLEPc] C nonsymmetric eignenproblem binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.225 sec

Test Suite: test_module

Results

Duration0.259 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/slepc] Verify slepc module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.205 sec
Test case:[libs/slepc] Verify module SLEPC_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/slepc] Verify module SLEPC_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/slepc] Verify dynamic library available in SLEPC_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/slepc] Verify static library is not present in SLEPC_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/slepc] Verify module SLEPC_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/slepc] Verify header file is present in SLEPC_INC (gnu15/mpich)
Outcome:Passed
Duration:0.008 sec

Test Suite: rm_execution

Results

Duration13.117 sec
Tests4
Failures0

Tests

rm_execution

Test case:[libs/SLEPc] F90 SVD test binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.276 sec
Test case:[libs/SLEPc] C SVD of the Lauchli matrix binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.279 sec
Test case:[libs/SLEPc] F90 quadratic eigensystem with PEP object binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.281 sec
Test case:[libs/SLEPc] C nonsymmetric eignenproblem binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.281 sec

Test Suite: test_module

Results

Duration0.259 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/slepc] Verify slepc module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.208 sec
Test case:[libs/slepc] Verify module SLEPC_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/slepc] Verify module SLEPC_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/slepc] Verify dynamic library available in SLEPC_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/slepc] Verify static library is not present in SLEPC_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/slepc] Verify module SLEPC_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/slepc] Verify header file is present in SLEPC_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.007 sec

Test Suite: rm_execution

Results

Duration8.987 sec
Tests4
Failures4

Tests

rm_execution

Test case:[libs/SLEPc] F90 SVD test binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.194 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 23)
  `run_mpi_binary ./test4f "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.32590
Batch job 360 submitted

Job 360 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./test4f  (family=mvapich2)
[1776237158.401284] [c1:45329:0]        ib_iface.c:1011 UCX  ERROR iface=0x1721fd80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237158.404509] [c2:19993:0]        ib_iface.c:1011 UCX  ERROR iface=0x7245c60: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SLEPc] C SVD of the Lauchli matrix binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.248 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 32)
  `run_mpi_binary ./ex15 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.16889
Batch job 361 submitted

Job 361 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex15  (family=mvapich2)
[1776237160.417548] [c2:20076:0]        ib_iface.c:1011 UCX  ERROR iface=0x17123b60: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237160.416455] [c1:45503:0]        ib_iface.c:1011 UCX  ERROR iface=0x278daf10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299535) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffce72c8bcc, argv=0x7ffce72c8bc0, required=1, provided=0x7ffce72c873c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339864079) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe10ac480c, argv=0x7ffe10ac4800, required=1, provided=0x7ffe10ac437c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SLEPc] F90 quadratic eigensystem with PEP object binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.298 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 41)
  `run_mpi_binary ./ex16f90 " -pep_nev 4 -terse " 1 1' failed
job script = /tmp/job.ohpc-test.15559
Batch job 362 submitted

Job 362 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex16f90 -pep_nev 4 -terse (family=mvapich2)
[1776237163.417388] [c1:45677:0]        ib_iface.c:1011 UCX  ERROR iface=0x283de90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SLEPc] C nonsymmetric eignenproblem binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.247 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 50)
  `run_mpi_binary ./ex29 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.23753
Batch job 363 submitted

Job 363 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex29  (family=mvapich2)
[1776237166.427269] [c1:45850:0]        ib_iface.c:1011 UCX  ERROR iface=0x2e22af10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(943843855) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffcfb05044c, argv=0x7ffcfb050440, required=1, provided=0x7ffcfb04ffbc) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237166.431138] [c2:20159:0]        ib_iface.c:1011 UCX  ERROR iface=0x1a8e9b60: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428623) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffff5a213bc, argv=0x7ffff5a213b0, required=1, provided=0x7ffff5a20f2c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_module

Results

Duration0.251 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/slepc] Verify slepc module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.202 sec
Test case:[libs/slepc] Verify module SLEPC_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.01 sec
Test case:[libs/slepc] Verify module SLEPC_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.009 sec
Test case:[libs/slepc] Verify dynamic library available in SLEPC_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.007 sec
Test case:[libs/slepc] Verify static library is not present in SLEPC_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/slepc] Verify module SLEPC_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.008 sec
Test case:[libs/slepc] Verify header file is present in SLEPC_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.007 sec

Test Suite: rm_execution

Results

Duration0.272 sec
Tests2
Failures0

Tests

rm_execution

Test case:[libs/SuperLU] C test runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.14 sec
Test case:[libs/SuperLU] F77 test runs under resource manager (slurm/gnu15)
Outcome:Passed
Duration:0.132 sec

Test Suite: test_module

Results

Duration0.324 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/SuperLU] Verify SUPERLU module is loaded and matches rpm version (gnu15)
Outcome:Passed
Duration:0.198 sec
Test case:[libs/SuperLU] Verify SUPERLU_DIR is defined and directory exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/SuperLU] Verify module SUPERLU_LIB is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/SuperLU] Verify dynamic library available in SUPERLU_LIB (gnu15)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/SuperLU] Verify static library is not present in SUPERLU_LIB (gnu15)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/SuperLU] Verify module SUPERLU_INC is defined and exists (gnu15)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/SuperLU] Verify header file is present in SUPERLU_INC (gnu15)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration100.106 sec
Tests9
Failures0

Tests

rm_execution

Test case:[libs/SuperLU_DIST] PDGSSVX with full (default) options (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.28 sec
Test case:[libs/SuperLU_DIST] pdgssvx_ABglobal with full (default) options (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:5.387 sec
Test case:[libs/SuperLU_DIST] vary RHS (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:7.497 sec
Test case:[libs/SuperLU_DIST] vary RHS ABglobal (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:9.604 sec
Test case:[libs/SuperLU_DIST] reuse permutation vector (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:10.659 sec
Test case:[libs/SuperLU_DIST] reuse permutation vector ABglobal (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:12.763 sec
Test case:[libs/SuperLU_DIST] reuse symbolic factorization (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:14.867 sec
Test case:[libs/SuperLU_DIST] reuse symbolic factorization ABglobal (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:16.97 sec
Test case:[libs/SuperLU_DIST] multi-grid (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:19.079 sec

Test Suite: test_module

Results

Duration0.34 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.215 sec
Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST_DIR is defined and directory exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_LIB is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[libs/SuperLU_DIST] Verify dynamic library available in SUPERLU_DIST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/SuperLU_DIST] Verify static library is not present in SUPERLU_DIST_LIB (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_INC is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.022 sec
Test case:[libs/SuperLU_DIST] Verify header file is present in SUPERLU_DIST_INC (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration34.962 sec
Tests9
Failures9

Tests

rm_execution

Test case:[libs/SuperLU_DIST] PDGSSVX with full (default) options (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.247 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 39)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.1004
Batch job 379 submitted

Job 379 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive -r 2 -c 2 g20.rua (family=mvapich2)
[1776237251.406530] [c2:21376:0]        ib_iface.c:1011 UCX  ERROR iface=0x1ff59d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734991) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffcdce6842c, argv=0x7ffcdce68420, required=3, provided=0x7ffcdce6843c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237251.410623] [c1:48442:0]        ib_iface.c:1011 UCX  ERROR iface=0x3cfccd80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(406972943) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe4da1a1ec, argv=0x7ffe4da1a1e0, required=3, provided=0x7ffe4da1a1fc) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SuperLU_DIST] pdgssvx_ABglobal with full (default) options (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.247 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 48)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.9720
Batch job 380 submitted

Job 380 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive_ABglobal -r 1 -c 1 g20.rua (family=mvapich2)
[1776237251.989096] [c2:21458:0]        ib_iface.c:1011 UCX  ERROR iface=0x3970cd20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc0748961c, argv=0x7ffc07489610) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237251.989621] [c1:48616:0]        ib_iface.c:1011 UCX  ERROR iface=0x1bc53d80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428367) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc0f653d0c, argv=0x7ffc0f653d00) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SuperLU_DIST] vary RHS (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.294 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 57)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive1 "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.10803
Batch job 381 submitted

Job 381 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive1 -r 2 -c 2 g20.rua (family=mvapich2)
[1776237252.562760] [c1:48792:0]        ib_iface.c:1011 UCX  ERROR iface=0x16abd80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339864079) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffdc21d7f0c, argv=0x7ffdc21d7f00, required=3, provided=0x7ffdc21d7f1c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237252.577533] [c2:21541:0]        ib_iface.c:1011 UCX  ERROR iface=0x2d12ad20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339864079) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe215bcedc, argv=0x7ffe215bced0, required=3, provided=0x7ffe215bceec) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SuperLU_DIST] vary RHS ABglobal (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.296 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 66)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive1_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.9084
Batch job 382 submitted

Job 382 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive1_ABglobal -r 1 -c 1 g20.rua (family=mvapich2)
[1776237253.152976] [c1:48967:0]        ib_iface.c:1011 UCX  ERROR iface=0x305a7d80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdd01568cc, argv=0x7ffdd01568c0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237253.161263] [c2:21625:0]        ib_iface.c:1011 UCX  ERROR iface=0x39b68d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc54277a4c, argv=0x7ffc54277a40) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SuperLU_DIST] reuse permutation vector (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.353 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 75)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive2 "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.29039
Batch job 383 submitted

Job 383 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive2 -r 2 -c 2 g20.rua (family=mvapich2)
[1776237253.735389] [c1:49141:0]        ib_iface.c:1011 UCX  ERROR iface=0x289cfd80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(205646351) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc44c9590c, argv=0x7ffc44c95900, required=3, provided=0x7ffc44c9591c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237253.739945] [c2:21707:0]        ib_iface.c:1011 UCX  ERROR iface=0x187d0d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(943843855) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe0900e47c, argv=0x7ffe0900e470, required=3, provided=0x7ffe0900e48c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SuperLU_DIST] reuse permutation vector ABglobal (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.352 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 84)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive2_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.24938
Batch job 384 submitted

Job 384 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive2_ABglobal -r 1 -c 1 g20.rua (family=mvapich2)
[1776237254.294116] [c1:49317:0]        ib_iface.c:1011 UCX  ERROR iface=0x38cc8d80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428367) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffff8c8066c, argv=0x7ffff8c80660) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237254.297433] [c2:21791:0]        ib_iface.c:1011 UCX  ERROR iface=0xab8d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe8f47649c, argv=0x7ffe8f476490) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SuperLU_DIST] reuse symbolic factorization (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.352 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 93)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive3 "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.32549
Batch job 385 submitted

Job 385 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive3 -r 2 -c 2 g20.rua (family=mvapich2)
[1776237254.842889] [c2:21873:0]        ib_iface.c:1011 UCX  ERROR iface=0x3aaed20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(742517263) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff7295f6dc, argv=0x7fff7295f6d0, required=3, provided=0x7fff7295f6ec) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237254.844015] [c1:49491:0]        ib_iface.c:1011 UCX  ERROR iface=0x233aad80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(272755215) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffde07c5ffc, argv=0x7ffde07c5ff0, required=3, provided=0x7ffde07c600c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SuperLU_DIST] reuse symbolic factorization ABglobal (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:5.409 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 102)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive3_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.6982
Batch job 386 submitted

Job 386 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive3_ABglobal -r 1 -c 1 g20.rua (family=mvapich2)
[1776237255.414046] [c2:21957:0]        ib_iface.c:1011 UCX  ERROR iface=0x15897d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(205646095) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc9614a7ec, argv=0x7ffc9614a7e0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237255.420912] [c1:49666:0]        ib_iface.c:1011 UCX  ERROR iface=0x6261d80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff671274fc, argv=0x7fff671274f0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/SuperLU_DIST] multi-grid (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:5.412 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 115)
  `run_mpi_binary -t "${TIMEOUT}" ./pddrive4 "g20.rua" "${NODES}" 10' failed
job script = /tmp/job.ohpc-test.18847
Batch job 387 submitted

Job 387 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive4 g20.rua (family=mvapich2)
[1776237255.965516] [c1:49841:0]        ib_iface.c:1011 UCX  ERROR iface=0x19410490: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237255.965525] [c1:49842:0]        ib_iface.c:1011 UCX  ERROR iface=0x366a5490: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237255.965758] [c1:49843:0]        ib_iface.c:1011 UCX  ERROR iface=0x1e1ce490: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(272755215) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd5223005c, argv=0x7ffd52230050, required=3, provided=0x7ffd5223007c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(608299535) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffce3520d6c, argv=0x7ffce3520d60, required=3, provided=0x7ffce3520d8c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(809626127) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff6c2c43bc, argv=0x7fff6c2c43b0, required=3, provided=0x7fff6c2c43dc) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237255.971368] [c2:22041:0]        ib_iface.c:1011 UCX  ERROR iface=0x2b616380: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237255.971375] [c2:22043:0]        ib_iface.c:1011 UCX  ERROR iface=0x2ed04380: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237255.971551] [c2:22040:0]        ib_iface.c:1011 UCX  ERROR iface=0x36c94380: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(272755215) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff92227edc, argv=0x7fff92227ed0, required=3, provided=0x7fff92227efc) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(876734991) on node 9: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc79eb47ec, argv=0x7ffc79eb47e0, required=3, provided=0x7ffc79eb480c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(1010952719) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fffc8f6881c, argv=0x7fffc8f68810, required=3, provided=0x7fffc8f6883c) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_module

Results

Duration0.34 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.215 sec
Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST_DIR is defined and directory exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_LIB is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/SuperLU_DIST] Verify dynamic library available in SUPERLU_DIST_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/SuperLU_DIST] Verify static library is not present in SUPERLU_DIST_LIB (gnu15/mvapich2)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_INC is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/SuperLU_DIST] Verify header file is present in SUPERLU_DIST_INC (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec

Test Suite: rm_execution

Results

Duration31.588 sec
Tests9
Failures0

Tests

rm_execution

Test case:[libs/SuperLU_DIST] PDGSSVX with full (default) options (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.174 sec
Test case:[libs/SuperLU_DIST] pdgssvx_ABglobal with full (default) options (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.227 sec
Test case:[libs/SuperLU_DIST] vary RHS (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.218 sec
Test case:[libs/SuperLU_DIST] vary RHS ABglobal (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.277 sec
Test case:[libs/SuperLU_DIST] reuse permutation vector (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.275 sec
Test case:[libs/SuperLU_DIST] reuse permutation vector ABglobal (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.33 sec
Test case:[libs/SuperLU_DIST] reuse symbolic factorization (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.332 sec
Test case:[libs/SuperLU_DIST] reuse symbolic factorization ABglobal (slurm/gnu15/mpich)
Outcome:Passed
Duration:5.375 sec
Test case:[libs/SuperLU_DIST] multi-grid (slurm/gnu15/mpich)
Outcome:Passed
Duration:5.38 sec

Test Suite: test_module

Results

Duration0.333 sec
Tests7
Failures0

Tests

test_module

Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.21 sec
Test case:[libs/SuperLU_DIST] Verify SUPERLU_DIST_DIR is defined and directory exists (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_LIB is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/SuperLU_DIST] Verify dynamic library available in SUPERLU_DIST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec
Test case:[libs/SuperLU_DIST] Verify static library is not present in SUPERLU_DIST_LIB (gnu15/mpich)
Outcome:Passed
Duration:0.023 sec
Test case:[libs/SuperLU_DIST] Verify module SUPERLU_DIST_INC is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[libs/SuperLU_DIST] Verify header file is present in SUPERLU_DIST_INC (gnu15/mpich)
Outcome:Passed
Duration:0.02 sec

Test Suite: rm_execution

Results

Duration18.58 sec
Tests6
Failures0

Tests

rm_execution

Test case:[libs/Trilinos] Kokkos-MemorySpace runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.219 sec
Test case:[libs/Trilinos] Tpetra-InitMPI runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.218 sec
Test case:[libs/Trilinos] Tpetra-DataRedistribution runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.268 sec
Test case:[libs/Trilinos] Teuchos-ParameterList runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.277 sec
Test case:[libs/Trilinos] Teuchos-LAPACK runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.272 sec
Test case:[libs/Trilinos] Teuchos-BLAS runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.326 sec

Test Suite: rm_execution

Results

Duration30.155 sec
Tests6
Failures0

Tests

rm_execution

Test case:[libs/Trilinos] Kokkos-MemorySpace runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:1.169 sec
Test case:[libs/Trilinos] Tpetra-InitMPI runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.266 sec
Test case:[libs/Trilinos] Tpetra-DataRedistribution runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:5.378 sec
Test case:[libs/Trilinos] Teuchos-ParameterList runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.433 sec
Test case:[libs/Trilinos] Teuchos-LAPACK runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.428 sec
Test case:[libs/Trilinos] Teuchos-BLAS runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:7.481 sec

Test Suite: rm_execution

Results

Duration18.597 sec
Tests6
Failures2

Tests

rm_execution

Test case:[libs/Trilinos] Kokkos-MemorySpace runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:2.218 sec
Test case:[libs/Trilinos] Tpetra-InitMPI runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.239 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 39)
  `run_mpi_binary -t "${TIMEOUT}" "${EXE}" "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.31889
Batch job 404 submitted

Job 404 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./lesson_tpetra_init null (family=mvapich2)
[1776237365.925273] [c1:52896:0]        ib_iface.c:1011 UCX  ERROR iface=0x224fda40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc97dfc4ac, argv=0x7ffc97dfc4a0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237365.933934] [c2:23532:0]        ib_iface.c:1011 UCX  ERROR iface=0x2b1c58a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe2cf9122c, argv=0x7ffe2cf91220) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237365.936419] [c1:52897:0]        ib_iface.c:1011 UCX  ERROR iface=0x3d788a40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Test case:[libs/Trilinos] Tpetra-DataRedistribution runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.293 sec
Failed
(from function `run_mpi_binary' in file ../../../common/functions, line 434,
 in test file rm_execution, line 49)
  `run_mpi_binary -t "${TIMEOUT}" "${EXE}" "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.12902
Batch job 405 submitted

Job 405 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./lesson_tpetra_dataredist null (family=mvapich2)
[1776237366.492182] [c2:23616:0]        ib_iface.c:1011 UCX  ERROR iface=0xd11a940: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237366.490366] [c1:53072:0]        ib_iface.c:1011 UCX  ERROR iface=0xcd88b50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734735) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc6deab66c, argv=0x7ffc6deab660) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(474081551) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd5ea0c48c, argv=0x7ffd5ea0c480) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[libs/Trilinos] Teuchos-ParameterList runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:3.267 sec
Test case:[libs/Trilinos] Teuchos-LAPACK runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:3.261 sec
Test case:[libs/Trilinos] Teuchos-BLAS runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:4.319 sec

Test Suite: lmod_installed

Results

Duration0.039 sec
Tests1
Failures0

Tests

lmod_installed

Test case:[modules] Check if lmod RPM installed
Outcome:Passed
Duration:0.039 sec

Test Suite: interactive_commands

Results

Duration4.494 sec
Tests8
Failures0

Tests

interactive_commands

Test case:[modules] module purge
Outcome:Passed
Duration:0.141 sec
Test case:[modules] module list
Outcome:Passed
Duration:0.238 sec
Test case:[modules] module help
Outcome:Passed
Duration:0.28 sec
Test case:[modules] module load/unload
Outcome:Passed
Duration:1.612 sec
Test case:[modules] module whatis
Outcome:Passed
Duration:0.265 sec
Test case:[modules] module swap
Outcome:Passed
Duration:0.571 sec
Test case:[modules] path updated
Outcome:Passed
Duration:0.811 sec
Test case:[modules] module depends-on
Outcome:Passed
Duration:0.576 sec

Test Suite: rm_execution

Results

Duration1.812 sec
Tests4
Failures0

Tests

rm_execution

Test case:[modules] env variable passes through ()
Outcome:Passed
Duration:0.419 sec
Test case:[modules] loaded module passes through ()
Outcome:Passed
Duration:0.482 sec
Test case:[modules] module commands available in RMS job ()
Outcome:Passed
Duration:0.508 sec
Test case:[modules] module load propagates thru RMS ()
Outcome:Passed
Duration:0.403 sec

Test Suite: man_page_check

Results

Duration0.101 sec
Tests1
Failures0

Tests

man_page_check

Test case:[MPI] mpicc man page available (gnu15/openmpi5)
Outcome:Passed
Duration:0.101 sec

Test Suite: rm_execution_multi_host

Results

Duration12.984 sec
Tests3
Failures0

Tests

rm_execution_multi_host

Test case:[MPI] C binary runs on two nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.226 sec
Test case:[MPI] C++ binary runs on two nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.328 sec
Test case:[MPI] F90 binary runs on two nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.43 sec

Test Suite: rm_execution_single_host

Results

Duration10.871 sec
Tests3
Failures0

Tests

rm_execution_single_host

Test case:[MPI] C binary runs on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.275 sec
Test case:[MPI] C++ binary runs on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.272 sec
Test case:[MPI] F90 binary runs on single node under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.324 sec

Test Suite: version_match

Results

Duration0.515 sec
Tests3
Failures0

Tests

version_match

Test case:[MPI] MPI module loaded (gnu15/openmpi5)
Outcome:Passed
Duration:0.142 sec
Test case:[MPI] MPI module version available (gnu15/openmpi5)
Outcome:Passed
Duration:0.143 sec
Test case:[MPI] mpicc, mpicxx, and mpif90 versions match module (gnu15/openmpi5)
Outcome:Passed
Duration:0.23 sec

Test Suite: man_page_check

Results

Duration0.102 sec
Tests1
Failures0

Tests

man_page_check

Test case:[MPI] mpicc man page available (gnu15/mpich)
Outcome:Passed
Duration:0.102 sec

Test Suite: rm_execution_multi_host

Results

Duration5.613 sec
Tests3
Failures0

Tests

rm_execution_multi_host

Test case:[MPI] C binary runs on two nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.171 sec
Test case:[MPI] C++ binary runs on two nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec
Test case:[MPI] F90 binary runs on two nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.219 sec

Test Suite: rm_execution_single_host

Results

Duration6.668 sec
Tests3
Failures0

Tests

rm_execution_single_host

Test case:[MPI] C binary runs on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.223 sec
Test case:[MPI] C++ binary runs on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.224 sec
Test case:[MPI] F90 binary runs on single node under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.221 sec

Test Suite: version_match

Results

Duration0.441 sec
Tests3
Failures0

Tests

version_match

Test case:[MPI] MPI module loaded (gnu15/mpich)
Outcome:Passed
Duration:0.136 sec
Test case:[MPI] MPI module version available (gnu15/mpich)
Outcome:Passed
Duration:0.131 sec
Test case:[MPI] mpicc, mpicxx, and mpif90 versions match module (gnu15/mpich)
Outcome:Passed
Duration:0.174 sec

Test Suite: man_page_check

Results

Duration0.091 sec
Tests1
Failures0

Tests

man_page_check

Test case:[MPI] mpicc man page available (gnu15/mvapich2)
Outcome:Passed
Duration:0.091 sec

Test Suite: rm_execution_multi_host

Results

Duration11.983 sec
Tests3
Failures3

Tests

rm_execution_multi_host

Test case:[MPI] C binary runs on two nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.297 sec
Failed
(from function `run_mpi_binary' in file ../../common/functions, line 434,
 in test file rm_execution_multi_host, line 27)
  `run_mpi_binary -t "${TIMEOUT}" ./C_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.6765
Batch job 424 submitted

Job 424 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_test 8 (family=mvapich2)
[1776237440.400151] [c2:25268:0]        ib_iface.c:1011 UCX  ERROR iface=0x2e8f7bf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237440.400442] [c2:25266:0]        ib_iface.c:1011 UCX  ERROR iface=0x33b6abf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237440.404310] [c1:55915:0]        ib_iface.c:1011 UCX  ERROR iface=0x203ce00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237440.404311] [c1:55916:0]        ib_iface.c:1011 UCX  ERROR iface=0x34807e00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(1010952463) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff29db487c, argv=0x7fff29db4870) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(205646095) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffef2938abc, argv=0x7ffef2938ab0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(205646095) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fffeb25a1fc, argv=0x7fffeb25a1f0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(876734735) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc97a099cc, argv=0x7ffc97a099c0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[MPI] C++ binary runs on two nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.342 sec
Failed
(from function `run_mpi_binary' in file ../../common/functions, line 434,
 in test file rm_execution_multi_host, line 36)
  `run_mpi_binary -t "${TIMEOUT}" ./CXX_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.24094
Batch job 425 submitted

Job 425 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./CXX_test 8 (family=mvapich2)
[1776237440.950821] [c1:56094:0]        ib_iface.c:1011 UCX  ERROR iface=0x17f94e00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd8803e12c, argv=0x7ffd8803e120) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237440.955039] [c2:25353:0]        ib_iface.c:1011 UCX  ERROR iface=0x3387fbf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237440.953032] [c1:56092:0]        ib_iface.c:1011 UCX  ERROR iface=0xc56ae00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237440.955664] [c2:25352:0]        ib_iface.c:1011 UCX  ERROR iface=0x34e78bf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339863823) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdd814d7ec, argv=0x7ffdd814d7e0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(205646095) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffce34258fc, argv=0x7ffce34258f0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 56092 RUNNING AT c1
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
Test case:[MPI] F90 binary runs on two nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.344 sec
Failed
(from function `run_mpi_binary' in file ../../common/functions, line 434,
 in test file rm_execution_multi_host, line 45)
  `run_mpi_binary -t "${TIMEOUT}" ./F90_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.11802
Batch job 426 submitted

Job 426 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F90_test 8 (family=mvapich2)
[1776237441.568556] [c2:25441:0]        ib_iface.c:1011 UCX  ERROR iface=0x13919120: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237441.568726] [c1:56272:0]        ib_iface.c:1011 UCX  ERROR iface=0xc74b330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237441.568714] [c1:56274:0]        ib_iface.c:1011 UCX  ERROR iface=0x3a063330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237441.571317] [c2:25442:0]        ib_iface.c:1011 UCX  ERROR iface=0x10a4d120: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 25442 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c1: task 0: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion

Test Suite: rm_execution_single_host

Results

Duration6.727 sec
Tests3
Failures3

Tests

rm_execution_single_host

Test case:[MPI] C binary runs on single node under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.241 sec
Failed
(from function `run_mpi_binary' in file ../../common/functions, line 434,
 in test file rm_execution_single_host, line 27)
  `run_mpi_binary -t "${TIMEOUT}" ./C_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.13388
Batch job 421 submitted

Job 421 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_test 2 (family=mvapich2)
[1776237435.396832] [c1:55739:0]        ib_iface.c:1011 UCX  ERROR iface=0x1dd86190: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe4a01f3bc, argv=0x7ffe4a01f3b0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[MPI] C++ binary runs on single node under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.242 sec
Failed
(from function `run_mpi_binary' in file ../../common/functions, line 434,
 in test file rm_execution_single_host, line 36)
  `run_mpi_binary -t "${TIMEOUT}" ./CXX_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.10256
Batch job 422 submitted

Job 422 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c2
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./CXX_test 2 (family=mvapich2)
[1776237435.373714] [c2:25007:0]        ib_iface.c:1011 UCX  ERROR iface=0x5c74cb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffca39109dc, argv=0x7ffca39109d0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[MPI] F90 binary runs on single node under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.244 sec
Failed
(from function `run_mpi_binary' in file ../../common/functions, line 434,
 in test file rm_execution_single_host, line 45)
  `run_mpi_binary -t "${TIMEOUT}" ./F90_test "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.19852
Batch job 423 submitted

Job 423 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c2
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F90_test 2 (family=mvapich2)
[1776237435.920107] [c2:25181:0]        ib_iface.c:1011 UCX  ERROR iface=0x14818af0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: version_match

Results

Duration0.431 sec
Tests3
Failures0

Tests

version_match

Test case:[MPI] MPI module loaded (gnu15/mvapich2)
Outcome:Passed
Duration:0.135 sec
Test case:[MPI] MPI module version available (gnu15/mvapich2)
Outcome:Passed
Duration:0.134 sec
Test case:[MPI] mpicc, mpicxx, and mpif90 versions match module (gnu15/mvapich2)
Outcome:Passed
Duration:0.162 sec

Test Suite: conman

Results

Duration0.132 sec
Tests3
Failures0

Tests

conman

Test case:[ConMan] Verify conman binary available
Outcome:Passed
Duration:0.025 sec
Test case:[ConMan] Verify rpm version matches binary
Outcome:Passed
Duration:0.06 sec
Test case:[ConMan] Verify man page availability
Outcome:Passed
Duration:0.047 sec

Test Suite: warewulf-ipmi

Results

Duration0.0 sec
Tests1
Failures0

Tests

warewulf-ipmi

Test case:[warewulf-ipmi] ipmitool lanplus protocol
Outcome:Skipped
Duration:0.0 sec
Skipped
Warewulf not installed, skipping warewulf-ipmitool check

Test Suite: test_module

Results

Duration1.885 sec
Tests6
Failures0

Tests

test_module

Test case:[perf-tools/Dimemas] Verify dimemas module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.199 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.023 sec
Test case:[perf-tools/Dimemas] Verify availability of prv2dim binary (gnu15/mpich)
Outcome:Passed
Duration:0.027 sec
Test case:[perf-tools/Dimemas] Verify availability of Dimemas binary (gnu15/mpich)
Outcome:Passed
Duration:0.027 sec
Test case:[perf-tools/Dimemas] Run Dimemas simulation (gnu15/mpich)
Outcome:Passed
Duration:1.592 sec

Test Suite: test_module

Results

Duration1.88 sec
Tests6
Failures0

Tests

test_module

Test case:[perf-tools/Dimemas] Verify dimemas module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.2 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.02 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_BIN is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.019 sec
Test case:[perf-tools/Dimemas] Verify availability of prv2dim binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.027 sec
Test case:[perf-tools/Dimemas] Verify availability of Dimemas binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.027 sec
Test case:[perf-tools/Dimemas] Run Dimemas simulation (gnu15/mvapich2)
Outcome:Passed
Duration:1.587 sec

Test Suite: test_module

Results

Duration1.908 sec
Tests6
Failures0

Tests

test_module

Test case:[perf-tools/Dimemas] Verify dimemas module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.207 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.023 sec
Test case:[perf-tools/Dimemas] Verify DIMEMAS_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.019 sec
Test case:[perf-tools/Dimemas] Verify availability of prv2dim binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.026 sec
Test case:[perf-tools/Dimemas] Verify availability of Dimemas binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.028 sec
Test case:[perf-tools/Dimemas] Run Dimemas simulation (gnu15/openmpi5)
Outcome:Passed
Duration:1.605 sec

Test Suite: test_imb_mpi1

Results

Duration3.275 sec
Tests1
Failures0

Tests

test_imb_mpi1

Test case:[Libs/IMB] run IMB-MPI1 on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.275 sec

Test Suite: test_imb_mpi2

Results

Duration5.492 sec
Tests2
Failures0

Tests

test_imb_mpi2

Test case:[Libs/IMB] run IMB-EXT on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.219 sec
Test case:[Libs/IMB] run IMB-IO on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.273 sec

Test Suite: test_imb_mpi3

Results

Duration1.171 sec
Tests2
Failures0

Tests

test_imb_mpi3

Test case:[Libs/IMB] run IMB-NBC on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[Libs/IMB] run IMB-RMA on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.171 sec

Test Suite: test_module

Results

Duration0.201 sec
Tests3
Failures0

Tests

test_module

Test case:[IMB] Verify IMB module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.178 sec
Test case:[IMB] Verify IMB_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.01 sec
Test case:[IMB] Verify executables are present in IMB_DIR/bin (gnu15-mpich)
Outcome:Passed
Duration:0.013 sec

Test Suite: test_imb_mpi1

Results

Duration2.244 sec
Tests1
Failures1

Tests

test_imb_mpi1

Test case:[Libs/IMB] run IMB-MPI1 on 2 nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.244 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_imb_mpi1, line 38)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.2289
Batch job 437 submitted

Job 437 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm /opt/ohpc/pub/libs/gnu15/mvapich2/imb/2021.11/bin/IMB-MPI1 -off_cache -1 -time 1.5 -npmin 100 -msglog 1:4 PingPong Sendrecv Bcast Allreduce (family=mvapich2)
[1776237495.578867] [c1:58114:0]        ib_iface.c:1011 UCX  ERROR iface=0x29d3700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237495.578869] [c1:58116:0]        ib_iface.c:1011 UCX  ERROR iface=0x286c1700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237495.578863] [c1:58120:0]        ib_iface.c:1011 UCX  ERROR iface=0x3f43d700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237495.578864] [c1:58121:0]        ib_iface.c:1011 UCX  ERROR iface=0x2dab7700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237495.579663] [c1:58119:0]        ib_iface.c:1011 UCX  ERROR iface=0x1c6ae700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734991) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffde7a1277c, argv=0x7ffde7a12770, required=0, provided=0x7ffde7a12798) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(138537487) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc12ccfebc, argv=0x7ffc12ccfeb0, required=0, provided=0x7ffc12ccfed8) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(474081807) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fffdf364bac, argv=0x7fffdf364ba0, required=0, provided=0x7fffdf364bc8) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(138537487) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe5dcb1afc, argv=0x7ffe5dcb1af0, required=0, provided=0x7ffe5dcb1b18) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(339864079) on node 5: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fffcb3b316c, argv=0x7fffcb3b3160, required=0, provided=0x7fffcb3b3188) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237495.583691] [c2:26357:0]        ib_iface.c:1011 UCX  ERROR iface=0x81c34f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237495.583764] [c2:26359:0]        ib_iface.c:1011 UCX  ERROR iface=0x331694f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237495.583773] [c2:26361:0]        ib_iface.c:1011 UCX  ERROR iface=0x340864f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237495.583799] [c2:26356:0]        ib_iface.c:1011 UCX  ERROR iface=0x3d2d54f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237495.583820] [c2:26355:0]        ib_iface.c:1011 UCX  ERROR iface=0x1b1e04f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237495.583922] [c2:26360:0]        ib_iface.c:1011 UCX  ERROR iface=0x2acd64f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408399) on node 10: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc870732cc, argv=0x7ffc870732c0, required=0, provided=0x7ffc870732e8) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(809626127) on node 12: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffebaab069c, argv=0x7ffebaab0690, required=0, provided=0x7ffebaab06b8) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(876734991) on node 14: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff701dd75c, argv=0x7fff701dd750, required=0, provided=0x7fff701dd778) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(541190671) on node 8: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd7d181a1c, argv=0x7ffd7d181a10, required=0, provided=0x7ffd7d181a38) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(4319759) on node 9: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffee219fdcc, argv=0x7ffee219fdc0, required=0, provided=0x7ffee219fde8) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(406972943) on node 13: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc80ff770c, argv=0x7ffc80ff7700, required=0, provided=0x7ffc80ff7728) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_imb_mpi2

Results

Duration6.581 sec
Tests2
Failures2

Tests

test_imb_mpi2

Test case:[Libs/IMB] run IMB-EXT on 2 nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.34 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_imb_mpi2, line 41)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.30089
Batch job 438 submitted

Job 438 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm /opt/ohpc/pub/libs/gnu15/mvapich2/imb/2021.11/bin/IMB-EXT -npmin 100 -msglog 1:4 Window (family=mvapich2)
[1776237500.356734] [c1:58304:0]        ib_iface.c:1011 UCX  ERROR iface=0x98a5cd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(4319759) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffdc2c8960c, argv=0x7ffdc2c89600, required=0, provided=0x7ffdc2c89628) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237500.365558] [c2:26451:0]        ib_iface.c:1011 UCX  ERROR iface=0x299565c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(406972943) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd9771da4c, argv=0x7ffd9771da40, required=0, provided=0x7ffd9771da68) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[Libs/IMB] run IMB-IO on 2 nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.241 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_imb_mpi2, line 53)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.19936
Batch job 439 submitted

Job 439 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm /opt/ohpc/pub/libs/gnu15/mvapich2/imb/2021.11/bin/IMB-IO -npmin 100 Open_Close (family=mvapich2)
[1776237503.361229] [c2:26534:0]        ib_iface.c:1011 UCX  ERROR iface=0x171a9bd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(272755215) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc88042a8c, argv=0x7ffc88042a80, required=0, provided=0x7ffc88042aa8) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237503.361861] [c1:58479:0]        ib_iface.c:1011 UCX  ERROR iface=0x2c4cfdf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237503.372929] [c1:58480:0]        ib_iface.c:1011 UCX  ERROR iface=0xe33adf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428623) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff2dd911ec, argv=0x7fff2dd911e0, required=0, provided=0x7fff2dd91208) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_imb_mpi3

Results

Duration3.294 sec
Tests2
Failures1

Tests

test_imb_mpi3

Test case:[Libs/IMB] run IMB-NBC on 2 nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[Libs/IMB] run IMB-RMA on 2 nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.294 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file test_imb_mpi3, line 56)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.2871
Batch job 440 submitted

Job 440 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm /opt/ohpc/pub/libs/gnu15/mvapich2/imb/2021.11/bin/IMB-RMA -npmin 100 -msglog 1:4 Unidir_put Unidir_get (family=mvapich2)
[1776237507.402562] [c2:26618:0]        ib_iface.c:1011 UCX  ERROR iface=0xd4ce170: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(138537487) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe52ca2f1c, argv=0x7ffe52ca2f10, required=0, provided=0x7ffe52ca2f38) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237507.403436] [c1:58657:0]        ib_iface.c:1011 UCX  ERROR iface=0x3c736280: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299535) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff4083944c, argv=0x7fff40839440, required=0, provided=0x7fff40839468) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 26617 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c1: task 0: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion

Test Suite: test_module

Results

Duration0.206 sec
Tests3
Failures0

Tests

test_module

Test case:[IMB] Verify IMB module is loaded and matches rpm version (gnu15-mvapich2)
Outcome:Passed
Duration:0.182 sec
Test case:[IMB] Verify IMB_DIR is defined and directory exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.009 sec
Test case:[IMB] Verify executables are present in IMB_DIR/bin (gnu15-mvapich2)
Outcome:Passed
Duration:0.015 sec

Test Suite: test_imb_mpi1

Results

Duration4.326 sec
Tests1
Failures0

Tests

test_imb_mpi1

Test case:[Libs/IMB] run IMB-MPI1 on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.326 sec

Test Suite: test_imb_mpi2

Results

Duration9.696 sec
Tests2
Failures0

Tests

test_imb_mpi2

Test case:[Libs/IMB] run IMB-EXT on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:5.369 sec
Test case:[Libs/IMB] run IMB-IO on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.327 sec

Test Suite: test_imb_mpi3

Results

Duration3.275 sec
Tests2
Failures0

Tests

test_imb_mpi3

Test case:[Libs/IMB] run IMB-NBC on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
skipped
Test case:[Libs/IMB] run IMB-RMA on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.275 sec

Test Suite: test_module

Results

Duration0.209 sec
Tests3
Failures0

Tests

test_module

Test case:[IMB] Verify IMB module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.186 sec
Test case:[IMB] Verify IMB_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[IMB] Verify executables are present in IMB_DIR/bin (gnu15-openmpi5)
Outcome:Passed
Duration:0.014 sec

Test Suite: rm_execution

Results

Duration14.956 sec
Tests2
Failures0

Tests

rm_execution

Test case:[OMB] run osu_bw on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.273 sec
Test case:[OMB] run osu_latency on 2 nodes under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:11.683 sec

Test Suite: test_module

Results

Duration0.364 sec
Tests7
Failures0

Tests

test_module

Test case:[OMB] Verify OMB module is loaded and matches rpm version (gnu15-mpich)
Outcome:Passed
Duration:0.191 sec
Test case:[OMB] Verify OMB_DIR is defined and directory exists (gnu15-mpich)
Outcome:Passed
Duration:0.019 sec
Test case:[OMB] Verify osu_bw binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.028 sec
Test case:[OMB] Verify osu_latency binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.033 sec
Test case:[OMB] Verify osu_allgather binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.029 sec
Test case:[OMB] Verify osu_get_bw binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[OMB] Verify osu_put_latency binary is available (gnu15-mpich)
Outcome:Passed
Duration:0.033 sec

Test Suite: rm_execution

Results

Duration3.44 sec
Tests2
Failures2

Tests

rm_execution

Test case:[OMB] run osu_bw on 2 nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.194 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 23)
  `run_mpi_binary -t "$TIMEOUT" "$EXE" "-m 512" 2 2' failed
job script = /tmp/job.ohpc-test.16648
Batch job 447 submitted

Job 447 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm osu_bw -m 512 (family=mvapich2)
[1776237562.344082] [c1:59922:0]        ib_iface.c:1011 UCX  ERROR iface=0x31e1f1d0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe57acd58c, argv=0x7ffe57acd580) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237562.356814] [c2:27252:0]        ib_iface.c:1011 UCX  ERROR iface=0x2eaf5cf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(1010952463) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff5ddffb9c, argv=0x7fff5ddffb90) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[OMB] run osu_latency on 2 nodes under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.246 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 36)
  `run_mpi_binary -t "$TIMEOUT" "$EXE" "-m $MESSAGE_SIZE" 2 2' failed
job script = /tmp/job.ohpc-test.5809
Batch job 448 submitted

Job 448 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm osu_latency -m 512 (family=mvapich2)
[1776237562.875963] [c1:60095:0]        ib_iface.c:1011 UCX  ERROR iface=0x4299190: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe6d3a019c, argv=0x7ffe6d3a0190) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237562.900732] [c2:27333:0]        ib_iface.c:1011 UCX  ERROR iface=0x2e30bcb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc480a39cc, argv=0x7ffc480a39c0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

Test Suite: test_module

Results

Duration0.37 sec
Tests7
Failures0

Tests

test_module

Test case:[OMB] Verify OMB module is loaded and matches rpm version (gnu15-mvapich2)
Outcome:Passed
Duration:0.192 sec
Test case:[OMB] Verify OMB_DIR is defined and directory exists (gnu15-mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[OMB] Verify osu_bw binary is available (gnu15-mvapich2)
Outcome:Passed
Duration:0.03 sec
Test case:[OMB] Verify osu_latency binary is available (gnu15-mvapich2)
Outcome:Passed
Duration:0.033 sec
Test case:[OMB] Verify osu_allgather binary is available (gnu15-mvapich2)
Outcome:Passed
Duration:0.03 sec
Test case:[OMB] Verify osu_get_bw binary is available (gnu15-mvapich2)
Outcome:Passed
Duration:0.033 sec
Test case:[OMB] Verify osu_put_latency binary is available (gnu15-mvapich2)
Outcome:Passed
Duration:0.031 sec

Test Suite: rm_execution

Results

Duration7.605 sec
Tests2
Failures0

Tests

rm_execution

Test case:[OMB] run osu_bw on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:3.277 sec
Test case:[OMB] run osu_latency on 2 nodes under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.328 sec

Test Suite: test_module

Results

Duration0.381 sec
Tests7
Failures0

Tests

test_module

Test case:[OMB] Verify OMB module is loaded and matches rpm version (gnu15-openmpi5)
Outcome:Passed
Duration:0.205 sec
Test case:[OMB] Verify OMB_DIR is defined and directory exists (gnu15-openmpi5)
Outcome:Passed
Duration:0.019 sec
Test case:[OMB] Verify osu_bw binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.031 sec
Test case:[OMB] Verify osu_latency binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[OMB] Verify osu_allgather binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.031 sec
Test case:[OMB] Verify osu_get_bw binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.03 sec
Test case:[OMB] Verify osu_put_latency binary is available (gnu15-openmpi5)
Outcome:Passed
Duration:0.033 sec

Test Suite: rm_execution

Results

Duration1.55 sec
Tests3
Failures0

Tests

rm_execution

Test case:[perf-tools/Paraver] Trace MPI application with Extrae (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.216 sec
Test case:[perf-tools/Paraver] Merge trace files with mpi2prv (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.052 sec
Test case:[perf-tools/Paraver] Run paramedir analysis on trace (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.282 sec

Test Suite: test_module

Results

Duration0.377 sec
Tests9
Failures0

Tests

test_module

Test case:[perf-tools/Paraver] Verify paraver module is loaded and matches rpm version
Outcome:Passed
Duration:0.188 sec
Test case:[perf-tools/Paraver] Verify PARAVER_DIR is defined and exists
Outcome:Passed
Duration:0.021 sec
Test case:[perf-tools/Paraver] Verify PARAVER_BIN is defined and exists
Outcome:Passed
Duration:0.021 sec
Test case:[perf-tools/Paraver] Verify availability of paramedir binary
Outcome:Passed
Duration:0.026 sec
Test case:[perf-tools/Paraver] Verify availability of wxparaver binary
Outcome:Passed
Duration:0.028 sec
Test case:[perf-tools/Paraver] Verify availability of prvstats binary
Outcome:Passed
Duration:0.026 sec
Test case:[perf-tools/Paraver] Verify availability of prvmerger binary
Outcome:Passed
Duration:0.027 sec
Test case:[perf-tools/Paraver] Verify PARAVER_LIB is defined and exists
Outcome:Passed
Duration:0.02 sec
Test case:[perf-tools/Paraver] Verify PARAVER_INC is defined and exists
Outcome:Passed
Duration:0.02 sec

Test Suite: rm_execution

Results

Duration2.632 sec
Tests3
Failures0

Tests

rm_execution

Test case:[perf-tools/Paraver] Trace MPI application with Extrae (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:2.296 sec
Test case:[perf-tools/Paraver] Merge trace files with mpi2prv (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.051 sec
Test case:[perf-tools/Paraver] Run paramedir analysis on trace (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.285 sec

Test Suite: test_module

Results

Duration0.377 sec
Tests9
Failures0

Tests

test_module

Test case:[perf-tools/Paraver] Verify paraver module is loaded and matches rpm version
Outcome:Passed
Duration:0.194 sec
Test case:[perf-tools/Paraver] Verify PARAVER_DIR is defined and exists
Outcome:Passed
Duration:0.02 sec
Test case:[perf-tools/Paraver] Verify PARAVER_BIN is defined and exists
Outcome:Passed
Duration:0.02 sec
Test case:[perf-tools/Paraver] Verify availability of paramedir binary
Outcome:Passed
Duration:0.026 sec
Test case:[perf-tools/Paraver] Verify availability of wxparaver binary
Outcome:Passed
Duration:0.027 sec
Test case:[perf-tools/Paraver] Verify availability of prvstats binary
Outcome:Passed
Duration:0.026 sec
Test case:[perf-tools/Paraver] Verify availability of prvmerger binary
Outcome:Passed
Duration:0.025 sec
Test case:[perf-tools/Paraver] Verify PARAVER_LIB is defined and exists
Outcome:Passed
Duration:0.019 sec
Test case:[perf-tools/Paraver] Verify PARAVER_INC is defined and exists
Outcome:Passed
Duration:0.02 sec

Test Suite: rm_execution

Results

Duration2.606 sec
Tests3
Failures0

Tests

rm_execution

Test case:[perf-tools/Paraver] Trace MPI application with Extrae (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.27 sec
Test case:[perf-tools/Paraver] Merge trace files with mpi2prv (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.055 sec
Test case:[perf-tools/Paraver] Run paramedir analysis on trace (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.281 sec

Test Suite: test_module

Results

Duration0.39 sec
Tests9
Failures0

Tests

test_module

Test case:[perf-tools/Paraver] Verify paraver module is loaded and matches rpm version
Outcome:Passed
Duration:0.2 sec
Test case:[perf-tools/Paraver] Verify PARAVER_DIR is defined and exists
Outcome:Passed
Duration:0.021 sec
Test case:[perf-tools/Paraver] Verify PARAVER_BIN is defined and exists
Outcome:Passed
Duration:0.021 sec
Test case:[perf-tools/Paraver] Verify availability of paramedir binary
Outcome:Passed
Duration:0.027 sec
Test case:[perf-tools/Paraver] Verify availability of wxparaver binary
Outcome:Passed
Duration:0.028 sec
Test case:[perf-tools/Paraver] Verify availability of prvstats binary
Outcome:Passed
Duration:0.028 sec
Test case:[perf-tools/Paraver] Verify availability of prvmerger binary
Outcome:Passed
Duration:0.026 sec
Test case:[perf-tools/Paraver] Verify PARAVER_LIB is defined and exists
Outcome:Passed
Duration:0.02 sec
Test case:[perf-tools/Paraver] Verify PARAVER_INC is defined and exists
Outcome:Passed
Duration:0.019 sec

Test Suite: rm_execution

Results

Duration14.178 sec
Tests3
Failures0

Tests

rm_execution

Test case:[perf-tools/Scalasca] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:8.063 sec
Test case:[perf-tools/Scalasca] MPI C++ binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.842 sec
Test case:[perf-tools/Scalasca] Serial C OpenMP binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.273 sec

Test Suite: test_module

Results

Duration0.361 sec
Tests11
Failures0

Tests

test_module

Test case:[perf-tools/Scalasca] Verify scalasca module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.204 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.011 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.009 sec
Test case:[perf-tools/Scalasca] Verify availability of scalasca binary (gnu15/mpich)
Outcome:Passed
Duration:0.015 sec
Test case:[perf-tools/Scalasca] Verify availability of scan binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.mpi binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.omp binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.hyb binary (gnu15/mpich)
Outcome:Passed
Duration:0.018 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.ser binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of square binary (gnu15/mpich)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of user guide for scalasca (gnu15/mpich)
Outcome:Passed
Duration:0.019 sec

Test Suite: rm_execution

Results

Duration9.987 sec
Tests3
Failures2

Tests

rm_execution

Test case:[perf-tools/Scalasca] MPI C binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:5.413 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 24)
  `run_mpi_binary -s 1 ./mpi/C_mpi_test $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.31023
Batch job 485 submitted

Job 485 failed...
Reason=NonZeroExitCode

S=C=A=N: Scalasca 2.6.2 trace collection and analysis
S=C=A=N: ./scorep_C_mpi_test_8x2_trace experiment archive
S=C=A=N: Wed Apr 15 07:23:39 2026: Collect start
/opt/ohpc/pub/utils/prun/2.2/prun ./mpi/C_mpi_test 8
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/C_mpi_test 8 (family=mvapich2)
[1776237822.144656] [c2:30378:0]        ib_iface.c:1011 UCX  ERROR iface=0x244b6cb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(742517007) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff9437425c, argv=0x7fff94374250) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237822.149715] [c2:30380:0]        ib_iface.c:1011 UCX  ERROR iface=0x3c82ccb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237822.148585] [c1:65857:0]        ib_iface.c:1011 UCX  ERROR iface=0x1a743e30: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237822.148849] [c1:65856:0]        ib_iface.c:1011 UCX  ERROR iface=0x3a8cee30: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd0d5c48cc, argv=0x7ffd0d5c48c0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 30380 RUNNING AT c2
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event
srun: error: c1: task 0: Exited with exit code 7
[mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion
S=C=A=N: Wed Apr 15 07:23:43 2026: Collect done (status=0) 4s
Abort: missing experiment archive ./scorep_C_mpi_test_8x2_trace
Test case:[perf-tools/Scalasca] MPI C++ binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.249 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 38)
  `run_mpi_binary -s 1 ./mpi/CXX_mpi_test $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.22464
Batch job 486 submitted

Job 486 failed...
Reason=NonZeroExitCode

S=C=A=N: Scalasca 2.6.2 trace collection and analysis
S=C=A=N: ./scorep_CXX_mpi_test_8x2_trace experiment archive
S=C=A=N: Wed Apr 15 07:23:45 2026: Collect start
/opt/ohpc/pub/utils/prun/2.2/prun ./mpi/CXX_mpi_test 8
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/CXX_mpi_test 8 (family=mvapich2)
[1776237826.100954] [c2:30465:0]        ib_iface.c:1011 UCX  ERROR iface=0x1d548eb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237826.100955] [c2:30468:0]        ib_iface.c:1011 UCX  ERROR iface=0x1a66aeb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(71428367) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffca54210cc, argv=0x7ffca54210c0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(876734735) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd93c7f54c, argv=0x7ffd93c7f540) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237826.107632] [c1:66155:0]        ib_iface.c:1011 UCX  ERROR iface=0x324f1030: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237826.108330] [c1:66157:0]        ib_iface.c:1011 UCX  ERROR iface=0x3065b030: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd69093b4c, argv=0x7ffd69093b40) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(205646095) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc9653afcc, argv=0x7ffc9653afc0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
S=C=A=N: Wed Apr 15 07:23:46 2026: Collect done (status=0) 1s
Abort: missing experiment archive ./scorep_CXX_mpi_test_8x2_trace
Test case:[perf-tools/Scalasca] Serial C OpenMP binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:2.325 sec

Test Suite: test_module

Results

Duration0.356 sec
Tests11
Failures0

Tests

test_module

Test case:[perf-tools/Scalasca] Verify scalasca module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.204 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.008 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_BIN is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.007 sec
Test case:[perf-tools/Scalasca] Verify availability of scalasca binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scan binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.mpi binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.015 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.omp binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.hyb binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.018 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.ser binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.018 sec
Test case:[perf-tools/Scalasca] Verify availability of square binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of user guide for scalasca (gnu15/mvapich2)
Outcome:Passed
Duration:0.018 sec

Test Suite: rm_execution

Results

Duration20.557 sec
Tests3
Failures0

Tests

rm_execution

Test case:[perf-tools/Scalasca] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:11.218 sec
Test case:[perf-tools/Scalasca] MPI C++ binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:7.012 sec
Test case:[perf-tools/Scalasca] Serial C OpenMP binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.327 sec

Test Suite: test_module

Results

Duration0.36 sec
Tests11
Failures0

Tests

test_module

Test case:[perf-tools/Scalasca] Verify scalasca module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.204 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.009 sec
Test case:[perf-tools/Scalasca] Verify module SCALASCA_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.008 sec
Test case:[perf-tools/Scalasca] Verify availability of scalasca binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scan binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.mpi binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.omp binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.hyb binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of scout.ser binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of square binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.017 sec
Test case:[perf-tools/Scalasca] Verify availability of user guide for scalasca (gnu15/openmpi5)
Outcome:Passed
Duration:0.018 sec

Test Suite: instrumenter_test

Results

Duration2.295 sec
Tests9
Failures0

Tests

instrumenter_test

Test case:[perf-tools/Score-P] MPI C binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.181 sec
Test case:[perf-tools/Score-P] MPI C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.183 sec
Test case:[perf-tools/Score-P] MPI Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.183 sec
Test case:[perf-tools/Score-P] Serial C OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.204 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.206 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.202 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.371 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.386 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/mpich)
Outcome:Passed
Duration:0.379 sec

Test Suite: rm_execution

Results

Duration31.48 sec
Tests9
Failures0

Tests

rm_execution

Test case:[perf-tools/Score-P] MPI C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.412 sec
Test case:[perf-tools/Score-P] MPI C++ binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.406 sec
Test case:[perf-tools/Score-P] MPI Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.869 sec
Test case:[perf-tools/Score-P] Serial C OpenMP binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.542 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.432 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.619 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:3.486 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:4.546 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary runs under resource manager (slurm/gnu15/mpich)
Outcome:Passed
Duration:7.168 sec

Test Suite: test_module

Results

Duration0.709 sec
Tests14
Failures0

Tests

test_module

Test case:[perf-tools/Score-P] Verify scorep module is loaded and matches rpm version (gnu15/mpich)
Outcome:Passed
Duration:0.21 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_DIR is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.021 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_BIN is defined and exists (gnu15/mpich)
Outcome:Passed
Duration:0.018 sec
Test case:[perf-tools/Score-P] Verify availability of scorep binary (gnu15/mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-backend-info binary (gnu15/mpich)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-config binary (gnu15/mpich)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-info binary (gnu15/mpich)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-preload-init binary (gnu15/mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-score binary (gnu15/mpich)
Outcome:Passed
Duration:0.033 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-wrapper binary (gnu15/mpich)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep wrapper binaries (gnu15/mpich)
Outcome:Passed
Duration:0.135 sec
Test case:[perf-tools/Score-P] Verify availability of user guide for scorep (gnu15/mpich)
Outcome:Passed
Duration:0.034 sec
Test case:[perf-tools/Score-P] Verify availability of OPEN_ISSUES for scorep (gnu15/mpich)
Outcome:Passed
Duration:0.035 sec
Test case:[perf-tools/Score-P] Verify availability of COPYING for scorep (gnu15/mpich)
Outcome:Passed
Duration:0.034 sec

Test Suite: instrumenter_test

Results

Duration2.298 sec
Tests9
Failures0

Tests

instrumenter_test

Test case:[perf-tools/Score-P] MPI C binary includes expected Score-P instrumenter symbols (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.187 sec
Test case:[perf-tools/Score-P] MPI C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.171 sec
Test case:[perf-tools/Score-P] MPI Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.182 sec
Test case:[perf-tools/Score-P] Serial C OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.193 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.206 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.206 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary includes expected Score-P instrumenter symbols (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.377 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.39 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:0.386 sec

Test Suite: rm_execution

Results

Duration30.274 sec
Tests9
Failures6

Tests

rm_execution

Test case:[perf-tools/Score-P] MPI C binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.252 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 59)
  `run_mpi_binary ./mpi/main_mpi_c "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.30102
Batch job 463 submitted

Job 463 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_mpi_c 8 (family=mvapich2)
[1776237708.499348] [c2:28607:0]        ib_iface.c:1011 UCX  ERROR iface=0x368bff90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe8927276c, argv=0x7ffe89272760) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237708.500373] [c1:62384:0]        ib_iface.c:1011 UCX  ERROR iface=0x11834110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237708.501568] [c1:62383:0]        ib_iface.c:1011 UCX  ERROR iface=0x378eb110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(1010952463) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc27fae64c, argv=0x7ffc27fae640) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237708.504968] [c2:28606:0]        ib_iface.c:1011 UCX  ERROR iface=0x11190f90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffced7df64c, argv=0x7ffced7df640) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(71428367) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd226fc0dc, argv=0x7ffd226fc0d0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237708.507182] [c1:62382:0]        ib_iface.c:1011 UCX  ERROR iface=0x1cf71110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237708.509531] [c2:28604:0]        ib_iface.c:1011 UCX  ERROR iface=0x27205f90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(1010952463) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd1e15e05c, argv=0x7ffd1e15e050) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[perf-tools/Score-P] MPI C++ binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:2.24 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 73)
  `run_mpi_binary ./mpi/main_mpi_cxx "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.26937
Batch job 464 submitted

Job 464 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_mpi_cxx 8 (family=mvapich2)
[1776237709.131013] [c1:62562:0]        ib_iface.c:1011 UCX  ERROR iface=0x1fdbd110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237709.131540] [c1:62561:0]        ib_iface.c:1011 UCX  ERROR iface=0x1740d110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd1e80534c, argv=0x7ffd1e805340) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(71428367) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffdc14d4d3c, argv=0x7ffdc14d4d30) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237709.137700] [c2:28693:0]        ib_iface.c:1011 UCX  ERROR iface=0x80e1f90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237709.137876] [c2:28694:0]        ib_iface.c:1011 UCX  ERROR iface=0x16a92f90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339863823) on node 6: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7fff02ab037c, argv=0x7fff02ab0370) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(876734735) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe70f280ec, argv=0x7ffe70f280e0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[perf-tools/Score-P] MPI Fortran binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:3.304 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 87)
  `run_mpi_binary ./mpi/main_mpi_fort "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.24721
Batch job 465 submitted

Job 465 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_mpi_fort 8 (family=mvapich2)
[1776237709.763435] [c2:28778:0]        ib_iface.c:1011 UCX  ERROR iface=0x24032dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237709.763340] [c1:62742:0]        ib_iface.c:1011 UCX  ERROR iface=0x123ae050: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237709.765313] [c2:28781:0]        ib_iface.c:1011 UCX  ERROR iface=0x327b1dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237709.763986] [c1:62741:0]        ib_iface.c:1011 UCX  ERROR iface=0xf97c050: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[perf-tools/Score-P] Serial C OpenMP binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:2.793 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:2.692 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:2.865 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.356 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 143)
  `run_mpi_binary ./mpi/main_hyb_c "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.23342
Batch job 469 submitted

Job 469 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_hyb_c 8 (family=mvapich2)
[1776237710.936366] [c1:62999:0]        ib_iface.c:1011 UCX  ERROR iface=0x20fb4800: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(742517007) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe7baec4ec, argv=0x7ffe7baec4e0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237710.941828] [c2:29025:0]        ib_iface.c:1011 UCX  ERROR iface=0x3733680: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237710.941994] [c2:29026:0]        ib_iface.c:1011 UCX  ERROR iface=0x2e26a680: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(474081551) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffca8d0b40c, argv=0x7ffca8d0b400) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(205646095) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd018ab8dc, argv=0x7ffd018ab8d0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237710.941825] [c1:63002:0]        ib_iface.c:1011 UCX  ERROR iface=0x2247a800: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 63002 RUNNING AT c1
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:4.358 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 157)
  `run_mpi_binary ./mpi/main_hyb_cxx "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.25218
Batch job 470 submitted

Job 470 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_hyb_cxx 8 (family=mvapich2)
[1776237711.542356] [c1:63181:0]        ib_iface.c:1011 UCX  ERROR iface=0x1172f470: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(339863823) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd5b76ecec, argv=0x7ffd5b76ece0) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237711.544710] [c1:63180:0]        ib_iface.c:1011 UCX  ERROR iface=0x215c8470: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734735) on node 2: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffc2981990c, argv=0x7ffc29819900) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237711.549256] [c2:29112:0]        ib_iface.c:1011 UCX  ERROR iface=0x336112f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237711.549494] [c2:29115:0]        ib_iface.c:1011 UCX  ERROR iface=0x3d53d2f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(876734735) on node 4: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffd159fd13c, argv=0x7ffd159fd130) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(876734735) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=0x7ffe9a976c1c, argv=0x7ffe9a976c10) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary runs under resource manager (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:5.414 sec
Failed
(from function `run_mpi_binary' in file ./common/functions, line 434,
 in test file rm_execution, line 171)
  `run_mpi_binary ./mpi/main_hyb_fort "${ARGS}" "${NODES}" "${TASKS}"' failed
job script = /tmp/job.ohpc-test.30182
Batch job 471 submitted

Job 471 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_hyb_fort 8 (family=mvapich2)
[1776237712.119652] [c2:29200:0]        ib_iface.c:1011 UCX  ERROR iface=0x17192530: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237712.119794] [c2:29202:0]        ib_iface.c:1011 UCX  ERROR iface=0x23d3a530: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
[1776237712.123148] [c1:63358:0]        ib_iface.c:1011 UCX  ERROR iface=0x1049a7b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
[1776237712.123516] [c1:63360:0]        ib_iface.c:1011 UCX  ERROR iface=0x2c6457b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(288).........:
MPIR_init_comm_world(34)......:
MPIR_Comm_commit(794).........:
MPIR_Comm_commit_internal(579):
MPID_Comm_commit_pre_hook(151):
MPIDI_world_pre_init(668).....:
MPIDI_UCX_init_world(277).....:
init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 63358 RUNNING AT c1
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

Test Suite: test_module

Results

Duration0.716 sec
Tests14
Failures0

Tests

test_module

Test case:[perf-tools/Score-P] Verify scorep module is loaded and matches rpm version (gnu15/mvapich2)
Outcome:Passed
Duration:0.216 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_DIR is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.022 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_BIN is defined and exists (gnu15/mvapich2)
Outcome:Passed
Duration:0.021 sec
Test case:[perf-tools/Score-P] Verify availability of scorep binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-backend-info binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-config binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-info binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-preload-init binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.031 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-score binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.033 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-wrapper binary (gnu15/mvapich2)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep wrapper binaries (gnu15/mvapich2)
Outcome:Passed
Duration:0.134 sec
Test case:[perf-tools/Score-P] Verify availability of user guide for scorep (gnu15/mvapich2)
Outcome:Passed
Duration:0.033 sec
Test case:[perf-tools/Score-P] Verify availability of OPEN_ISSUES for scorep (gnu15/mvapich2)
Outcome:Passed
Duration:0.033 sec
Test case:[perf-tools/Score-P] Verify availability of COPYING for scorep (gnu15/mvapich2)
Outcome:Passed
Duration:0.034 sec

Test Suite: instrumenter_test

Results

Duration2.385 sec
Tests9
Failures0

Tests

instrumenter_test

Test case:[perf-tools/Score-P] MPI C binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.194 sec
Test case:[perf-tools/Score-P] MPI C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.193 sec
Test case:[perf-tools/Score-P] MPI Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.194 sec
Test case:[perf-tools/Score-P] Serial C OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.194 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.202 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.21 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.378 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.412 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary includes expected Score-P instrumenter symbols (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:0.408 sec

Test Suite: rm_execution

Results

Duration68.022 sec
Tests9
Failures0

Tests

rm_execution

Test case:[perf-tools/Score-P] MPI C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.41 sec
Test case:[perf-tools/Score-P] MPI C++ binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.52 sec
Test case:[perf-tools/Score-P] MPI Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:7.046 sec
Test case:[perf-tools/Score-P] Serial C OpenMP binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.252 sec
Test case:[perf-tools/Score-P] Serial C++ OpenMP binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.203 sec
Test case:[perf-tools/Score-P] Serial Fortran OpenMP binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:6.364 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:8.759 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:10.876 sec
Test case:[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary runs under resource manager (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:15.592 sec

Test Suite: test_module

Results

Duration0.761 sec
Tests14
Failures0

Tests

test_module

Test case:[perf-tools/Score-P] Verify scorep module is loaded and matches rpm version (gnu15/openmpi5)
Outcome:Passed
Duration:0.214 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_DIR is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[perf-tools/Score-P] Verify module SCOREP_BIN is defined and exists (gnu15/openmpi5)
Outcome:Passed
Duration:0.021 sec
Test case:[perf-tools/Score-P] Verify availability of scorep binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-backend-info binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.033 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-config binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.038 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-info binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-preload-init binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-score binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep-wrapper binary (gnu15/openmpi5)
Outcome:Passed
Duration:0.032 sec
Test case:[perf-tools/Score-P] Verify availability of scorep wrapper binaries (gnu15/openmpi5)
Outcome:Passed
Duration:0.171 sec
Test case:[perf-tools/Score-P] Verify availability of user guide for scorep (gnu15/openmpi5)
Outcome:Passed
Duration:0.035 sec
Test case:[perf-tools/Score-P] Verify availability of OPEN_ISSUES for scorep (gnu15/openmpi5)
Outcome:Passed
Duration:0.034 sec
Test case:[perf-tools/Score-P] Verify availability of COPYING for scorep (gnu15/openmpi5)
Outcome:Passed
Duration:0.034 sec

Test Suite: test_harness

Results

Duration3.311 sec
Tests3
Failures0

Tests

test_harness

Test case:[rms-harness] Verify zero exit code from MPI job runs OK (slurm/gnu15/mpich)
Outcome:Passed
Duration:1.125 sec
Test case:[rms-harness] Verify non-zero exit code from MPI job detected as failure (slurm/gnu15/mpich)
Outcome:Passed
Duration:2.186 sec
Test case:[rms-harness] Verify long-running MPI job terminates with timeout parameter (slurm/gnu15/mpich)
Outcome:Skipped
Duration:0.0 sec
Skipped
disabled timeout test

Test Suite: test_harness

Results

Duration3.482 sec
Tests3
Failures1

Tests

test_harness

Test case:[rms-harness] Verify zero exit code from MPI job runs OK (slurm/gnu15/mvapich2)
Outcome:Failed
Duration:1.299 sec
Failed
(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file test_harness, line 29)
  `assert_success' failed

-- command failed --
status : 1
output (31 lines):
  job script = /tmp/job.ohpc-test.16460
  Batch job 495 submitted

  Job 495 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi_exit 0 (family=mvapich2)
  [1776237888.382062] [c1:67689:0]        ib_iface.c:1011 UCX  ERROR iface=0x1d2af190: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
  Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack:
  internal_Init(70).............: MPI_Init(argc=0x7ffd2acfab2c, argv=0x7ffd2acfab20) failed
  MPII_Init_thread(288).........:
  MPIR_init_comm_world(34)......:
  MPIR_Comm_commit(794).........:
  MPIR_Comm_commit_internal(579):
  MPID_Comm_commit_pre_hook(151):
  MPIDI_world_pre_init(668).....:
  MPIDI_UCX_init_world(277).....:
  init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
  [1776237888.393156] [c2:31245:0]        ib_iface.c:1011 UCX  ERROR iface=0x270aacb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument
  Abort(339863823) on node 1: Fatal error in internal_Init: Other MPI error, error stack:
  internal_Init(70).............: MPI_Init(argc=0x7ffc3bc7e51c, argv=0x7ffc3bc7e510) failed
  MPII_Init_thread(288).........:
  MPIR_init_comm_world(34)......:
  MPIR_Comm_commit(794).........:
  MPIR_Comm_commit_internal(579):
  MPID_Comm_commit_pre_hook(151):
  MPIDI_world_pre_init(668).....:
  MPIDI_UCX_init_world(277).....:
  init_worker(48)...............:  ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)
--
Test case:[rms-harness] Verify non-zero exit code from MPI job detected as failure (slurm/gnu15/mvapich2)
Outcome:Passed
Duration:2.183 sec
Test case:[rms-harness] Verify long-running MPI job terminates with timeout parameter (slurm/gnu15/mvapich2)
Outcome:Skipped
Duration:0.0 sec
Skipped
disabled timeout test

Test Suite: test_harness

Results

Duration6.423 sec
Tests3
Failures0

Tests

test_harness

Test case:[rms-harness] Verify zero exit code from MPI job runs OK (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:2.163 sec
Test case:[rms-harness] Verify non-zero exit code from MPI job detected as failure (slurm/gnu15/openmpi5)
Outcome:Passed
Duration:4.26 sec
Test case:[rms-harness] Verify long-running MPI job terminates with timeout parameter (slurm/gnu15/openmpi5)
Outcome:Skipped
Duration:0.0 sec
Skipped
disabled timeout test

Test Suite: run

Results

Duration3.812 sec
Tests5
Failures0

Tests

run

Test case:[charliecloud] check for RPM
Outcome:Passed
Duration:0.282 sec
Test case:[charliecloud] build image
Outcome:Skipped
Duration:0.0 sec
Skipped
skip experimental charliecloud image builder
Test case:[charliecloud] build alpine image from Docker (using singularity)
Outcome:Passed
Duration:2.76 sec
Test case:[charliecloud] exec image locally
Outcome:Passed
Duration:0.305 sec
Test case:[charliecloud] exec image via slurm
Outcome:Passed
Duration:0.465 sec

Test Suite: run

Results

Duration8.337 sec
Tests4
Failures0

Tests

run

Test case:[singularity] check for RPM
Outcome:Passed
Duration:0.018 sec
Test case:[singularity] pull down ubuntu docker image
Outcome:Passed
Duration:7.092 sec
Test case:[singularity] exec image
Outcome:Passed
Duration:0.277 sec
Test case:[singularity] exec image via slurm
Outcome:Passed
Duration:0.95 sec

Test Suite: completion

Results

Duration0.077 sec
Tests1
Failures0

Tests

completion

Test case:[singularity] check for bash completion
Outcome:Passed
Duration:0.077 sec

Test Suite: ntp

Results

Duration0.384 sec
Tests3
Failures0

Tests

ntp

Test case:[ntp] check for chronyc binary
Outcome:Passed
Duration:0.018 sec
Test case:[ntp] verify local time in sync on SMS
Outcome:Passed
Duration:0.152 sec
Test case:[ntp] verify local time in sync on compute
Outcome:Passed
Duration:0.214 sec

Test Suite: mem_limits

Results

Duration0.302 sec
Tests2
Failures0

Tests

mem_limits

Test case:[memlock] check increased soft limit
Outcome:Passed
Duration:0.142 sec
Test case:[memlock] check increased hard limit
Outcome:Passed
Duration:0.16 sec

Test Suite: pdsh

Results

Duration2.741 sec
Tests4
Failures0

Tests

pdsh

Test case:[pdsh] check for RPM (slurm)
Outcome:Passed
Duration:0.045 sec
Test case:[pdsh] run a shell command on c[1-2] (slurm)
Outcome:Passed
Duration:1.656 sec
Test case:[pdsh] check for pdsh-mod-slurm RPM (slurm)
Outcome:Passed
Duration:0.044 sec
Test case:[pdsh] run a shell command on -P normal (slurm)
Outcome:Passed
Duration:0.996 sec

Test Suite: ompi_info

Results

Duration0.502 sec
Tests1
Failures0

Tests

ompi_info

Test case:[openmpi] check for no output to stderr with ompi_info
Outcome:Passed
Duration:0.502 sec

Test Suite: magpie

Results

Duration1.002 sec
Tests3
Failures0

Tests

magpie

Test case:[magpie] check for RPM
Outcome:Passed
Duration:0.058 sec
Test case:[magpie] Verify MAGPIE module is loaded and matches rpm version
Outcome:Passed
Duration:0.555 sec
Test case:[magpie] Verify module MAGPIE_DIR is defined and exists
Outcome:Passed
Duration:0.389 sec

Test Suite: munge

Results

Duration1.099 sec
Tests4
Failures0

Tests

munge

Test case:[munge] check for OS provdied RPM
Outcome:Passed
Duration:0.044 sec
Test case:[munge] Generate a credential
Outcome:Passed
Duration:0.016 sec
Test case:[munge] Decode credential locally
Outcome:Passed
Duration:0.017 sec
Test case:[munge] Run benchmark
Outcome:Passed
Duration:1.022 sec

Test Suite: sacct

Results

Duration0.0 sec
Tests1
Failures0

Tests

sacct

Test case:[slurm] check for working sacct
Outcome:Skipped
Duration:0.0 sec
Skipped
Temporarily skip sacct check

Test Suite: sinfo

Results

Duration0.079 sec
Tests1
Failures0

Tests

sinfo

Test case:[slurm] Verify SLURM RPM version matches sinfo binary
Outcome:Passed
Duration:0.079 sec

Test Suite: oom

Results

Duration0.0 sec
Tests1
Failures0

Tests

oom

Test case:[oom] Test job OOM condition (gnu15/slurm)
Outcome:Skipped
Duration:0.0 sec
Skipped
disable to invistigate disk fillup

Generated by junit2html