(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution_multi_host, line 46) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.14887 Batch job 14 submitted Job 14 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./bin/xhpcg.gnu15.mvapich2 32 32 32 10 (family=mvapich2) [1776234727.429983] [c1:5191 :0] ib_iface.c:1011 UCX ERROR iface=0x192345b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234727.430056] [c1:5194 :0] ib_iface.c:1011 UCX ERROR iface=0x2d7555b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234727.430108] [c1:5196 :0] ib_iface.c:1011 UCX ERROR iface=0xf6ef5b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234727.430315] [c1:5190 :0] ib_iface.c:1011 UCX ERROR iface=0x12f6f5b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234727.430321] [c1:5197 :0] ib_iface.c:1011 UCX ERROR iface=0x340835b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234727.430433] [c1:5193 :0] ib_iface.c:1011 UCX ERROR iface=0x1e7835b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(1010952463) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffcd92535cc, argv=0x7ffcd92535c0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(876734735) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fffe0c0236c, argv=0x7fffe0c02360) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(205646095) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc140a5d2c, argv=0x7ffc140a5d20) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(742517007) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe4cecd88c, argv=0x7ffe4cecd880) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc9d7c24ec, argv=0x7ffc9d7c24e0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(71428367) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd1c0bc42c, argv=0x7ffd1c0bc420) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776234727.435793] [c2:3593 :0] ib_iface.c:1011 UCX ERROR iface=0x3eef63a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234727.436186] [c2:3594 :0] ib_iface.c:1011 UCX ERROR iface=0x2d3803a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234727.436202] [c2:3591 :0] ib_iface.c:1011 UCX ERROR iface=0x194413a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234727.436310] [c2:3588 :0] ib_iface.c:1011 UCX ERROR iface=0x3e2033a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234727.436388] [c2:3590 :0] ib_iface.c:1011 UCX ERROR iface=0x3667d3a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(742517007) on node 13: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fffaf77876c, argv=0x7fffaf778760) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(608299279) on node 8: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd042ed3ac, argv=0x7ffd042ed3a0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339863823) on node 14: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffcb0af494c, argv=0x7ffcb0af4940) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(1010952463) on node 11: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdef86d36c, argv=0x7ffdef86d360) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(71428367) on node 10: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffeeb9649cc, argv=0x7ffeeb9649c0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776234727.438496] [c2:3592 :0] ib_iface.c:1011 UCX ERROR iface=0x329933a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 3592 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions(in test file rm_execution_multi_host, line 55) `mv $run_yaml $wrk_yaml' failed Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory Moving to HPCG.32x32x32.P16.gnu15.mvapich2.yaml mv: missing destination file operand after 'HPCG.32x32x32.P16.gnu15.mvapich2.yaml' Try 'mv --help' for more information.(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution_single_host, line 46) `run_mpi_binary -t $CMD_TIMEOUT ${EXE} "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.31957 Batch job 13 submitted Job 13 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./bin/xhpcg.gnu15.mvapich2 32 32 32 10 (family=mvapich2) [1776234724.416413] [c1:5005 :0] ib_iface.c:1011 UCX ERROR iface=0x1380deb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234724.416411] [c1:5008 :0] ib_iface.c:1011 UCX ERROR iface=0x28d60eb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234724.416415] [c1:5009 :0] ib_iface.c:1011 UCX ERROR iface=0x3f7cdeb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234724.416579] [c1:5003 :0] ib_iface.c:1011 UCX ERROR iface=0x3117aeb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234724.417005] [c1:5007 :0] ib_iface.c:1011 UCX ERROR iface=0x26e99eb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234724.417211] [c1:5006 :0] ib_iface.c:1011 UCX ERROR iface=0x22201eb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734735) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff4594966c, argv=0x7fff45949660) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(608299279) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff4f4e9dac, argv=0x7fff4f4e9da0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339863823) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc91cd17ac, argv=0x7ffc91cd17a0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(742517007) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fffcddce70c, argv=0x7fffcddce700) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(474081551) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fffbf03a7ec, argv=0x7fffbf03a7e0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339863823) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdeabd454c, argv=0x7ffdeabd4540) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(in test file rm_execution_single_host, line 55) `mv $run_yaml $wrk_yaml' failed Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory Moving to HPCG.32x32x32.P8.gnu15.mvapich2.yaml mv: missing destination file operand after 'HPCG.32x32x32.P8.gnu15.mvapich2.yaml' Try 'mv --help' for more information.(from function `run_mpi_binary' in file ./common-test/functions, line 434, in test file rm_execution_multi_host, line 46) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.25487 Batch job 20 submitted Job 20 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./src/miniFE.x.gnu15.mvapich2 nx=256 ny=256 nz=256 verify_solution=0 (family=mvapich2) [1776234825.432335] [c2:3948 :0] ib_iface.c:1011 UCX ERROR iface=0x1cea1ad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234825.432340] [c2:3950 :0] ib_iface.c:1011 UCX ERROR iface=0xb664ad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234825.433393] [c2:3946 :0] ib_iface.c:1011 UCX ERROR iface=0x3a09ead0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234825.430679] [c1:6447 :0] ib_iface.c:1011 UCX ERROR iface=0x375ee600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234825.430718] [c1:6448 :0] ib_iface.c:1011 UCX ERROR iface=0x3d0e5600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(1010952463) on node 12: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc27de094c, argv=0x7ffc27de0940) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776234825.430752] [c1:6451 :0] ib_iface.c:1011 UCX ERROR iface=0x3aaf2600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234825.430689] [c1:6453 :0] ib_iface.c:1011 UCX ERROR iface=0x3c7b6600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428367) on node 14: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd670976dc, argv=0x7ffd670976d0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776234825.430815] [c1:6454 :0] ib_iface.c:1011 UCX ERROR iface=0x236fa600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234825.434039] [c2:3949 :0] ib_iface.c:1011 UCX ERROR iface=0x754ad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234825.434062] [c2:3945 :0] ib_iface.c:1011 UCX ERROR iface=0x34b24ad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234825.431232] [c1:6452 :0] ib_iface.c:1011 UCX ERROR iface=0x33f5e600: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234825.434095] [c2:3944 :0] ib_iface.c:1011 UCX ERROR iface=0xbfacad0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(1010952463) on node 10: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffddda5a78c, argv=0x7ffddda5a780) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdb35aa14c, argv=0x7ffdb35aa140) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe73d7230c, argv=0x7ffe73d72300) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(1010952463) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd3c33a3fc, argv=0x7ffd3c33a3f0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(205646095) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdab91dafc, argv=0x7ffdab91daf0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(608299279) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffda642bd0c, argv=0x7ffda642bd00) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339863823) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff9be163ec, argv=0x7fff9be163e0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(71428367) on node 9: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc78a834ac, argv=0x7ffc78a834a0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 3944 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions(from function `run_mpi_binary' in file ./common-test/functions, line 434, in test file rm_execution_single_host, line 46) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.17307 Batch job 19 submitted Job 19 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./src/miniFE.x.gnu15.mvapich2 nx=100 ny=100 nz=100 verify_solution=1 (family=mvapich2) [1776234823.386964] [c1:6260 :0] ib_iface.c:1011 UCX ERROR iface=0x34026ed0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234823.398042] [c1:6264 :0] ib_iface.c:1011 UCX ERROR iface=0x658aed0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428367) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffcf4971b3c, argv=0x7ffcf4971b30) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(742517007) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc63d4ee2c, argv=0x7ffc63d4ee20) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 6261 RUNNING AT c1 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestionsdisable BOS_RELEASE check(from function `run_mpi_binary' in file ./common/functions, line 434, in test file hipy, line 35) `run_mpi_binary "${_python} helloworld.py" $ARGS $NODES $TASKS' failed job script = /tmp/job.ohpc-test.18082 Batch job 32 submitted Job 32 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm python3 helloworld.py 8 (family=mvapich2) [1776234890.398008] [c2:4512 :0] ib_iface.c:1011 UCX ERROR iface=0x55b4740613e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408399) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffee7df2a58) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776234890.401553] [c1:7671 :0] ib_iface.c:1011 UCX ERROR iface=0x55ba9a2e85e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234890.401798] [c1:7670 :0] ib_iface.c:1011 UCX ERROR iface=0x557067e6f5e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299535) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7fffab7671d8) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408399) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffef166c2f8) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 4511 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c1: task 0: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 28) `run_mpi_binary -t "${TIMEOUT}" ./arrays_write "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.20850 Batch job 40 submitted Job 40 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./arrays_write (family=mvapich2) [1776234947.509003] [c1:8647 :0] ib_iface.c:1011 UCX ERROR iface=0x384c44c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff0314791c, argv=0x7fff03147910) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776234947.512290] [c1:8645 :0] ib_iface.c:1011 UCX ERROR iface=0x1bc4d4c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234947.519486] [c2:4957 :0] ib_iface.c:1011 UCX ERROR iface=0x243142d0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(205646095) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffee710cebc, argv=0x7ffee710ceb0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 4956 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c1: task 0: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 37) `run_mpi_binary -t "${TIMEOUT}" ./scalars_write "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.11040 Batch job 41 submitted Job 41 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./scalars_write (family=mvapich2) [1776234948.129391] [c1:8827 :0] ib_iface.c:1011 UCX ERROR iface=0x3fcd9a50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776234948.130233] [c1:8825 :0] ib_iface.c:1011 UCX ERROR iface=0x3b626a50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776234948.143896] [c2:5041 :0] ib_iface.c:1011 UCX ERROR iface=0x1dc7840: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 24) `run_mpi_binary ./$test atest 2 "${TASKS}"' failed job script = /tmp/job.ohpc-test.4443 Batch job 129 submitted Job 129 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./all_gather_test atest (family=mvapich2) Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_gather_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... [1776235839.442546] [c1:14989:0] ib_iface.c:1011 UCX ERROR iface=0x160727e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235839.442721] [c1:14983:0] ib_iface.c:1011 UCX ERROR iface=0x2463c7e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235839.445438] [c1:14984:0] ib_iface.c:1011 UCX ERROR iface=0x3e9c67e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235839.453255] [c2:8442 :0] ib_iface.c:1011 UCX ERROR iface=0xedd3480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235839.453260] [c2:8445 :0] ib_iface.c:1011 UCX ERROR iface=0x3eee7480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235839.453259] [c2:8448 :0] ib_iface.c:1011 UCX ERROR iface=0x38d77480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235839.453433] [c2:8447 :0] ib_iface.c:1011 UCX ERROR iface=0x3dbc3480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235839.453817] [c2:8443 :0] ib_iface.c:1011 UCX ERROR iface=0xe684480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235839.453872] [c2:8444 :0] ib_iface.c:1011 UCX ERROR iface=0x392af480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 8: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 11: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 14: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 13: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 10: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14984 RUNNING AT c1 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:1@c2] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:1@c2] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:1@c2] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c2: task 1: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 33) `run_mpi_binary ./$test atest 2 "${TASKS}"' failed job script = /tmp/job.ohpc-test.25024 Batch job 130 submitted Job 130 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./all_reduce_test atest (family=mvapich2) Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_reduce_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... [1776235841.072675] [c2:8540 :0] ib_iface.c:1011 UCX ERROR iface=0x3efa5480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235841.073004] [c2:8544 :0] ib_iface.c:1011 UCX ERROR iface=0xad1c480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235841.073319] [c2:8543 :0] ib_iface.c:1011 UCX ERROR iface=0x26f7480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 13: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 12: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235841.075133] [c2:8539 :0] ib_iface.c:1011 UCX ERROR iface=0xa067480: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235841.077859] [c1:15175:0] ib_iface.c:1011 UCX ERROR iface=0x233de7e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235841.078056] [c1:15170:0] ib_iface.c:1011 UCX ERROR iface=0x2506d7e0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 15171 RUNNING AT c1 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:1@c2] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:1@c2] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:1@c2] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c2: task 1: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 42) `run_mpi_binary ./$test atest 2 "${TASKS}"' failed job script = /tmp/job.ohpc-test.8919 Batch job 131 submitted Job 131 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./all_to_all_test atest (family=mvapich2) Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: all_to_all_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... [1776235842.647429] [c2:8634 :0] ib_iface.c:1011 UCX ERROR iface=0x2b4f1450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235842.647427] [c2:8641 :0] ib_iface.c:1011 UCX ERROR iface=0xa2ae450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235842.647571] [c2:8635 :0] ib_iface.c:1011 UCX ERROR iface=0x3f2bd450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235842.647579] [c2:8636 :0] ib_iface.c:1011 UCX ERROR iface=0x3a4bd450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235842.647575] [c2:8639 :0] ib_iface.c:1011 UCX ERROR iface=0x1fb45450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235842.648609] [c2:8638 :0] ib_iface.c:1011 UCX ERROR iface=0x2852c450: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 8: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 15: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 10: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 13: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 12: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235842.649145] [c1:15360:0] ib_iface.c:1011 UCX ERROR iface=0x2147d7b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235842.649155] [c1:15362:0] ib_iface.c:1011 UCX ERROR iface=0x218f47b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235842.649151] [c1:15363:0] ib_iface.c:1011 UCX ERROR iface=0x39d967b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 15357 RUNNING AT c1 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:1@c2] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:1@c2] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:1@c2] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c2: task 1: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 51) `run_mpi_binary ./$test atest 2 "${TASKS}"' failed job script = /tmp/job.ohpc-test.21018 Batch job 132 submitted Job 132 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./groups_test atest (family=mvapich2) Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: groups_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... [1776235844.230823] [c2:8736 :0] ib_iface.c:1011 UCX ERROR iface=0x10519420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.230962] [c2:8735 :0] ib_iface.c:1011 UCX ERROR iface=0x21a4a420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.230978] [c2:8729 :0] ib_iface.c:1011 UCX ERROR iface=0x18b41420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 15: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 8: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 14: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235844.232299] [c2:8734 :0] ib_iface.c:1011 UCX ERROR iface=0x249b3420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.232584] [c2:8731 :0] ib_iface.c:1011 UCX ERROR iface=0x38d24420: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 13: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235844.233195] [c1:15545:0] ib_iface.c:1011 UCX ERROR iface=0x36e28780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.233651] [c1:15547:0] ib_iface.c:1011 UCX ERROR iface=0x17d01780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235844.234802] [c1:15549:0] ib_iface.c:1011 UCX ERROR iface=0x21608780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.234939] [c1:15551:0] ib_iface.c:1011 UCX ERROR iface=0x2d17780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.235069] [c1:15550:0] ib_iface.c:1011 UCX ERROR iface=0x1a0ec780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235844.235523] [c1:15546:0] ib_iface.c:1011 UCX ERROR iface=0xfa59780: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 8730 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c1: task 0: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 60) `run_mpi_binary ./$test atest 2 "${TASKS}"' failed job script = /tmp/job.ohpc-test.1553 Batch job 133 submitted Job 133 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./broadcast_test atest (family=mvapich2) Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: broadcast_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... [1776235844.873741] [c1:15738:0] ib_iface.c:1011 UCX ERROR iface=0x3569d820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.873839] [c1:15732:0] ib_iface.c:1011 UCX ERROR iface=0x3110d820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.873946] [c1:15737:0] ib_iface.c:1011 UCX ERROR iface=0x5a95820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.876204] [c2:8825 :0] ib_iface.c:1011 UCX ERROR iface=0x267634c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.874085] [c1:15731:0] ib_iface.c:1011 UCX ERROR iface=0x22e2e820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.874104] [c1:15736:0] ib_iface.c:1011 UCX ERROR iface=0x35098820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.874532] [c1:15734:0] ib_iface.c:1011 UCX ERROR iface=0x2a0a5820: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235844.877995] [c2:8828 :0] ib_iface.c:1011 UCX ERROR iface=0x22eed4c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235844.878058] [c2:8829 :0] ib_iface.c:1011 UCX ERROR iface=0x1c184c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235844.878900] [c2:8830 :0] ib_iface.c:1011 UCX ERROR iface=0x32bfa4c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235844.879239] [c2:8826 :0] ib_iface.c:1011 UCX ERROR iface=0x297314c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 8826 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c1: task 0: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 79) `run_mpi_binary ./$test atest 2 "${TASKS}"' failed job script = /tmp/job.ohpc-test.14550 Batch job 135 submitted Job 135 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pointer_test atest (family=mvapich2) Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_testatest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Boost.Test WARNING: token "atest" does not correspond to the Boost.Test argument and should be placed after all Boost.Test arguments and the -- separator. For example: pointer_test --random -- atest Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... Running 1 test case... [1776235847.188727] [c2:9016 :0] ib_iface.c:1011 UCX ERROR iface=0x64c14f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235847.188723] [c2:9023 :0] ib_iface.c:1011 UCX ERROR iface=0x13e124f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235847.188918] [c2:9017 :0] ib_iface.c:1011 UCX ERROR iface=0x2db024f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235847.189090] [c2:9020 :0] ib_iface.c:1011 UCX ERROR iface=0x2c4544f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235847.189787] [c2:9022 :0] ib_iface.c:1011 UCX ERROR iface=0x1d5a04f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 15: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 8: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 9: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235847.190396] [c2:9018 :0] ib_iface.c:1011 UCX ERROR iface=0xd5f94f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 12: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 14: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776235847.189946] [c1:16105:0] ib_iface.c:1011 UCX ERROR iface=0x1f074850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235847.189933] [c1:16106:0] ib_iface.c:1011 UCX ERROR iface=0x29cf4850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235847.189909] [c1:16111:0] ib_iface.c:1011 UCX ERROR iface=0x2cf18850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235847.189965] [c1:16112:0] ib_iface.c:1011 UCX ERROR iface=0x31ea4850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235847.190031] [c1:16109:0] ib_iface.c:1011 UCX ERROR iface=0x38d92850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776235847.190100] [c1:16110:0] ib_iface.c:1011 UCX ERROR iface=0x436d850: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 9018 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 41) `run_mpi_binary -t "${TIMEOUT}" ./C_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.12079 Batch job 149 submitted Job 149 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_mpi_test 8 (family=mvapich2) [1776236096.028670] [c1:18489:0] ib_iface.c:1011 UCX ERROR iface=0x13520e00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236096.029030] [c1:18487:0] ib_iface.c:1011 UCX ERROR iface=0x2ef79e00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236096.031310] [c2:10161:0] ib_iface.c:1011 UCX ERROR iface=0xea69bf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236096.031865] [c2:10160:0] ib_iface.c:1011 UCX ERROR iface=0xec66bf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428367) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff903867dc, argv=0x7fff903867d0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(742517007) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd989e517c, argv=0x7ffd989e5170) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(1010952463) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdc369f1bc, argv=0x7ffdc369f1b0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339863823) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc3314843c, argv=0x7ffc33148430) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 28) `run_mpi_binary -t "${TIMEOUT}" ./ex1 "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.19967 Batch job 214 submitted Job 214 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex1 8 (family=mvapich2) [1776236354.443845] [c2:11186:0] ib_iface.c:1011 UCX ERROR iface=0x31385cc0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(742517007) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffcb40103cc, argv=0x7ffcb40103c0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236354.445849] [c1:24560:0] ib_iface.c:1011 UCX ERROR iface=0x1f3bae20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(742517007) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd08fc42ec, argv=0x7ffd08fc42e0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 37) `run_mpi_binary -t "${TIMEOUT}" ./ex2 "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.9708 Batch job 215 submitted Job 215 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex2 8 (family=mvapich2) [1776236354.988885] [c1:24734:0] ib_iface.c:1011 UCX ERROR iface=0x14641e20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd78e72f0c, argv=0x7ffd78e72f00) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236354.994371] [c2:11269:0] ib_iface.c:1011 UCX ERROR iface=0x7782cc0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe99a27c2c, argv=0x7ffe99a27c20) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 46) `run_mpi_binary -t "${TIMEOUT}" ./ex6 "" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.29829 Batch job 216 submitted Job 216 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex6 (family=mvapich2) [1776236355.537650] [c1:24910:0] ib_iface.c:1011 UCX ERROR iface=0x18686f10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236355.545743] [c2:11352:0] ib_iface.c:1011 UCX ERROR iface=0x1d23ab40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339863823) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe74b1c34c, argv=0x7ffe74b1c340) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(71428367) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd59a5964c, argv=0x7ffd59a59640) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 55) `run_mpi_binary -t "${TIMEOUT}" ./ex8 "" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.5668 Batch job 217 submitted Job 217 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex8 (family=mvapich2) [1776236356.110244] [c2:11434:0] ib_iface.c:1011 UCX ERROR iface=0x908ab40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236356.108517] [c1:25085:0] ib_iface.c:1011 UCX ERROR iface=0x2bc9cf10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(205646095) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffff2573a6c, argv=0x7ffff2573a60) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(474081551) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc952b630c, argv=0x7ffc952b6300) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 64) `run_mpi_binary -t "${TIMEOUT}" ./ex12f "" 1 2' failed job script = /tmp/job.ohpc-test.8746 Batch job 218 submitted Job 218 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex12f (family=mvapich2) [1776236357.398145] [c1:25259:0] ib_iface.c:1011 UCX ERROR iface=0x162bfd70: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 73) `run_mpi_binary -t "${TIMEOUT}" ./ex3 "-n 33 -solver 0 -v 1 1" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.25785 Batch job 219 submitted Job 219 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex3 -n 33 -solver 0 -v 1 1 (family=mvapich2) [1776236357.933046] [c2:11517:0] ib_iface.c:1011 UCX ERROR iface=0xa8decc0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428367) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffca3c6b54c, argv=0x7ffca3c6b540) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236357.933441] [c1:25433:0] ib_iface.c:1011 UCX ERROR iface=0x2d4ade20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffde195f72c, argv=0x7ffde195f720) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 82) `run_mpi_binary -t "${TIMEOUT}" ./ex4 "-n 33 -solver 10 -K 3 -B 0 -C 1 -U0 2 -F 4" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.7510 Batch job 220 submitted Job 220 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex4 -n 33 -solver 10 -K 3 -B 0 -C 1 -U0 2 -F 4 (family=mvapich2) [1776236358.529260] [c1:25609:0] ib_iface.c:1011 UCX ERROR iface=0x2841ef10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdf589654c, argv=0x7ffdf5896540) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236358.543281] [c2:11601:0] ib_iface.c:1011 UCX ERROR iface=0x15c74a30: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdb8d1ac2c, argv=0x7ffdb8d1ac20) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 91) `run_mpi_binary -t "${TIMEOUT}" ./ex5f "" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.23528 Batch job 221 submitted Job 221 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex5f (family=mvapich2) [1776236359.106829] [c2:11683:0] ib_iface.c:1011 UCX ERROR iface=0x1cee3c10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236359.105490] [c1:25783:0] ib_iface.c:1011 UCX ERROR iface=0x17d73dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)skippedskippedC++ example depends on non-installed header(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 130) `run_mpi_binary -t "${TIMEOUT}" ./ex11 "" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.14385 Batch job 222 submitted Job 222 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex11 (family=mvapich2) [1776236359.639125] [c2:11767:0] ib_iface.c:1011 UCX ERROR iface=0xe0e8b40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236359.637674] [c1:25958:0] ib_iface.c:1011 UCX ERROR iface=0x275e2f10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe32e0d6fc, argv=0x7ffe32e0d6f0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(608299279) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffde567587c, argv=0x7ffde5675870) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)skippedskippedC++ example depends on non-installed headerskippedskippedC++ example depends on non-installed header(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 26) `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed job script = /tmp/job.ohpc-test.17037 Batch job 239 submitted Job 239 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_double null (family=mvapich2) [1776236458.418566] [c1:28719:0] ib_iface.c:1011 UCX ERROR iface=0x7b34e20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236458.421813] [c2:13066:0] ib_iface.c:1011 UCX ERROR iface=0x323cecc0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(205646095) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffecdd05c4c, argv=0x7ffecdd05c40) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339863823) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc30279a4c, argv=0x7ffc30279a40) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 36) `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed job script = /tmp/job.ohpc-test.27980 Batch job 240 submitted Job 240 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_single null (family=mvapich2) [1776236461.388371] [c1:28894:0] ib_iface.c:1011 UCX ERROR iface=0x35283dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236461.403985] [c2:13148:0] ib_iface.c:1011 UCX ERROR iface=0x1119dc10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 46) `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed job script = /tmp/job.ohpc-test.27291 Batch job 241 submitted Job 241 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_double null (family=mvapich2) [1776236463.399125] [c2:13247:0] ib_iface.c:1011 UCX ERROR iface=0x270cac10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236463.403860] [c1:29083:0] ib_iface.c:1011 UCX ERROR iface=0xf17ddd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 56) `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed job script = /tmp/job.ohpc-test.4362 Batch job 242 submitted Job 242 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_complex null (family=mvapich2) [1776236466.415930] [c1:29258:0] ib_iface.c:1011 UCX ERROR iface=0x33d12dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236466.426075] [c2:13330:0] ib_iface.c:1011 UCX ERROR iface=0x323a6c10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 66) `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed job script = /tmp/job.ohpc-test.31553 Batch job 243 submitted Job 243 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_doublecomplex null (family=mvapich2) [1776236470.404394] [c2:13412:0] ib_iface.c:1011 UCX ERROR iface=0x282b9c10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236470.406633] [c1:29434:0] ib_iface.c:1011 UCX ERROR iface=0x38fe6dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)option no longer supportedoption no longer supportedoption no longer supportedoption no longer supportedSkipping fortran test bindings with mpich(from function `run_mpi_binary' in file ./common/functions, line 434, in test file test_pnetcdf, line 47) `run_mpi_binary ./C_parallel "atest" 2 4' failed job script = /tmp/job.ohpc-test.11518 Batch job 273 submitted Job 273 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_parallel atest (family=mvapich2) [1776236614.421847] [c1:32020:0] ib_iface.c:1011 UCX ERROR iface=0x306438d0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(1010952463) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff584d7e3c, argv=0x7fff584d7e30) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236614.430951] [c2:15084:0] ib_iface.c:1011 UCX ERROR iface=0x2c7937d0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffca40a871c, argv=0x7ffca40a8710) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)Skipping fortran test bindings with mvapich2(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 27) `run_mpi_binary ./C_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.28853 Batch job 304 submitted Job 304 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_test 8 (family=mvapich2) [1776236737.393273] [c1:34406:0] ib_iface.c:1011 UCX ERROR iface=0x1885e400: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236737.393242] [c1:34407:0] ib_iface.c:1011 UCX ERROR iface=0x27906400: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(809626127) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffccd96980c, argv=0x7ffccd969800, required=1, provided=0x7ffccd9693bc) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(205646351) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff38c2e7dc, argv=0x7fff38c2e7d0, required=1, provided=0x7fff38c2e38c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236737.399092] [c2:16494:0] ib_iface.c:1011 UCX ERROR iface=0x1c2d2f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236737.399217] [c2:16495:0] ib_iface.c:1011 UCX ERROR iface=0x3eda82f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339864079) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff7db8963c, argv=0x7fff7db89630, required=1, provided=0x7fff7db891ec) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(272755215) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc89602a2c, argv=0x7ffc89602a20, required=1, provided=0x7ffc896025dc) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236737.409587] [c2:16493:0] ib_iface.c:1011 UCX ERROR iface=0x27b282f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(1010952719) on node 5: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff89689f9c, argv=0x7fff89689f90, required=1, provided=0x7fff89689b4c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 36) `run_mpi_binary ./C_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.23475 Batch job 305 submitted Job 305 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_mpi_test 8 (family=mvapich2) [1776236738.005393] [c2:16582:0] ib_iface.c:1011 UCX ERROR iface=0xc8762f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236738.005639] [c2:16581:0] ib_iface.c:1011 UCX ERROR iface=0x37c852f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(272755215) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff1dbb770c, argv=0x7fff1dbb7700, required=1, provided=0x7fff1dbb72ac) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(541190671) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc1acbf72c, argv=0x7ffc1acbf720, required=1, provided=0x7ffc1acbf2cc) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236738.010085] [c1:34586:0] ib_iface.c:1011 UCX ERROR iface=0x356b0400: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236738.010806] [c1:34584:0] ib_iface.c:1011 UCX ERROR iface=0x803f400: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(272755215) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd0aee81bc, argv=0x7ffd0aee81b0, required=1, provided=0x7ffd0aee7d5c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(138537487) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd37c8286c, argv=0x7ffd37c82860, required=1, provided=0x7ffd37c8240c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 45) `run_mpi_binary ./F90_test 2 "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.15329 Batch job 306 submitted Job 306 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F90_test 2 (family=mvapich2) [1776236738.569433] [c1:34765:0] ib_iface.c:1011 UCX ERROR iface=0x1907b3c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236738.569568] [c1:34762:0] ib_iface.c:1011 UCX ERROR iface=0x3df9d3c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236738.577082] [c2:16668:0] ib_iface.c:1011 UCX ERROR iface=0x355181b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236738.578485] [c2:16669:0] ib_iface.c:1011 UCX ERROR iface=0x3c1661b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 34765 RUNNING AT c1 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file test_module, line 132) `run_mpi_binary ./C_test "atest" 1 1' failed job script = /tmp/job.ohpc-test.19216 Batch job 303 submitted Job 303 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_test atest (family=mvapich2) [1776236734.441049] [c1:34231:0] ib_iface.c:1011 UCX ERROR iface=0x2996c3f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(272755215) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd546ea22c, argv=0x7ffd546ea220, required=1, provided=0x7ffd546e9ddc) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 43) `run_mpi_binary -t "${TIMEOUT}" ./C_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.12576 Batch job 313 submitted Job 313 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_mpi_test /home/ohpc-test/phdf5_test.AMt5WF/phdf_test.out (family=mvapich2) [1776236785.414032] [c2:17231:0] ib_iface.c:1011 UCX ERROR iface=0x32e8fc10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236785.414028] [c2:17232:0] ib_iface.c:1011 UCX ERROR iface=0xf3ecc10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236785.415183] [c1:36052:0] ib_iface.c:1011 UCX ERROR iface=0x245dfd20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236785.415734] [c1:36051:0] ib_iface.c:1011 UCX ERROR iface=0x7fe8d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428367) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff079091cc, argv=0x7fff079091c0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(71428367) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe9686e40c, argv=0x7ffe9686e400) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(1010952463) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff6a1fb6fc, argv=0x7fff6a1fb6f0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 52) `run_mpi_binary -t "${TIMEOUT}" ./F_mpi_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.18114 Batch job 314 submitted Job 314 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F_mpi_test /home/ohpc-test/phdf5_test.AMt5WF/phdf_test.out (family=mvapich2) [1776236786.022950] [c2:17320:0] ib_iface.c:1011 UCX ERROR iface=0x3f699120: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236786.021986] [c1:36229:0] ib_iface.c:1011 UCX ERROR iface=0x176d3330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776236786.022682] [c1:36228:0] ib_iface.c:1011 UCX ERROR iface=0x1dff1330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236786.023646] [c2:17317:0] ib_iface.c:1011 UCX ERROR iface=0x261f3120: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776236786.032264] [c1:36230:0] ib_iface.c:1011 UCX ERROR iface=0x18004330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 22) `run_mpi_binary ./f90tst_parallel f90tst_parallel.nc $NODES $TASKS' failed job script = /tmp/job.ohpc-test.32628 Batch job 323 submitted Job 323 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./f90tst_parallel f90tst_parallel.nc (family=mvapich2) [1776236893.459527] [c1:37564:0] ib_iface.c:1011 UCX ERROR iface=0x1b47c50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 31) `run_mpi_binary ./f90tst_parallel2 f90tst_parallel2.nc $NODES $TASKS' failed job script = /tmp/job.ohpc-test.14654 Batch job 324 submitted Job 324 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./f90tst_parallel2 f90tst_parallel2.nc (family=mvapich2) [1776236895.364027] [c1:37739:0] ib_iface.c:1011 UCX ERROR iface=0x4f06c50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 40) `run_mpi_binary ./f90tst_parallel3 f90tst_parallel3.nc $NODES $TASKS' failed job script = /tmp/job.ohpc-test.29301 Batch job 325 submitted Job 325 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./f90tst_parallel3 f90tst_parallel3.nc (family=mvapich2) [1776236897.357586] [c1:37912:0] ib_iface.c:1011 UCX ERROR iface=0x14347c50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 49) `run_mpi_binary ./f90tst_parallel4 f90tst_parallel4.nc $NODES $TASKS' failed job script = /tmp/job.ohpc-test.20648 Batch job 326 submitted Job 326 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./f90tst_parallel4 f90tst_parallel4.nc (family=mvapich2) [1776236900.404354] [c1:38086:0] ib_iface.c:1011 UCX ERROR iface=0x2b46cc50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 27) `run_mpi_binary -t "${TIMEOUT}" ./dgraph_redist "bump.grf" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.25806 Batch job 334 submitted Job 334 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./dgraph_redist bump.grf (family=mvapich2) [1776237005.362684] [c1:39565:0] ib_iface.c:1011 UCX ERROR iface=0x14258ff0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237005.364881] [c2:17990:0] ib_iface.c:1011 UCX ERROR iface=0x1f951e40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd7727e33c, argv=0x7ffd7727e330) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc7425d7ac, argv=0x7ffc7425d7a0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 45) `run_mpi_binary -t "${TIMEOUT}" dgord "bump.grf /dev/null -vt" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.26357 Batch job 336 submitted Job 336 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm dgord bump.grf /dev/null -vt (family=mvapich2) [1776237007.406847] [c2:18153:0] ib_iface.c:1011 UCX ERROR iface=0xfa23e40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(742517263) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc3cc71a2c, argv=0x7ffc3cc71a20, required=3, provided=0x7ffc3cc71a44) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237007.409895] [c1:39914:0] ib_iface.c:1011 UCX ERROR iface=0x1525cff0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428623) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc5066d0ac, argv=0x7ffc5066d0a0, required=3, provided=0x7ffc5066d0c4) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 35) `run_mpi_binary ./pcscaex "${ARGS}" "${NODES}" "${TASKS}"' failed run_mpi_binary ./pcscaex 0 2 8 SHELL=/bin/bash BATS_RUN_LOGS_DIRECTORY= BATS_TEST_TMPDIR=/tmp/bats-run-hbwgwc/test/1 SUDO_GID=0 HISTCONTROL=ignoredups OMPI_MCA_rmaps_base_oversubscribe=1 PARALLEL_JOBSLOT0=0 PKG_CONFIG_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib/pkgconfig:/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/lib/pkgconfig HOSTNAME=ohpc-lenovo-sms HISTSIZE=1048576 FPATH=/opt/ohpc/admin/lmod/lmod/init/ksh_funcs BATS_LIBDIR=lib BATS_SUITE_TMPDIR=/tmp/bats-run-hbwgwc/suite BATS_ROOT_PID=380896 BATS_WARNING_FILE=/tmp/bats-run-hbwgwc/warnings.log BATS_FILE_TMPDIR=/tmp/bats-run-hbwgwc/file/0 _ModuleTable002_=dGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAiZ251MTUiLAp3ViA9ICIwMDAwMDAwMTUuMDAwMDAwMDAyLip6ZmluYWwiLAp9LAptdmFwaWNoMiA9IHsKYWN0aW9uQSA9IHsKW1twcmVwZW5kX3BhdGgoIk1PRFVMRVBBVEgiLCIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUtbXZhcGljaDIiKV1dLAp9LApmbiA9ICIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUvbXZhcGljaDIvNC4xIiwKZnVsbE5hbWUgPSAibXZhcGljaDIvNC4xIiwKbG9hZE9yZGVyID0gMywKcHJvcFQgPSB7fSwKc3RhY2tEZXB0aCA9IDAsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJtdmFwaWNoMiIsCndWID0gIjAwMDAwMDAwNC4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCm9w BATS_SEMAPHORE_DIR=/tmp/bats-run-hbwgwc/semaphores __LMOD_REF_COUNT_MODULEPATH=/opt/ohpc/pub/moduledeps/gnu15-mvapich2:1;/opt/ohpc/pub/moduledeps/gnu15:1;/opt/ohpc/pub/modulefiles:1 _ModuleTable005_=LAp9LAp9LAptcGF0aEEgPSB7CiIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUtbXZhcGljaDIiLCAiL29wdC9vaHBjL3B1Yi9tb2R1bGVkZXBzL2dudTE1IiwgIi9vcHQvb2hwYy9wdWIvbW9kdWxlZmlsZXMiLAp9LApzeXN0ZW1CYXNlTVBBVEggPSAiL29wdC9vaHBjL3B1Yi9tb2R1bGVmaWxlcyIsCn0K TEST_LOGS=test_module.log rm_execution.log SSHPASS= BATS_END_CODE_QUOTE=' SUDO_COMMAND=/bin/bash -c /tmp/user_integration_tests SUDO_USER=root BATS_TEST_SOURCE=/home/ohpc-test/tests/libs/scalapack/tests/rm_execution BATS_TEST_NAME=test_-5b-24TESTNAME-2fPCSCAEX-5d_CPCGESV_under_resource_manager_-28-24RESOURCE-5fMANAGER-2f-24LMOD-5fFAMILY-5fCOMPILER-2f-24LMOD-5fFAMILY-5fMPI-29 GPG_TTY=not a tty LMOD_DIR=/opt/ohpc/admin/lmod/lmod/libexec LMOD_FAMILY_OPENBLAS_VERSION=0.3.32 PWD=/home/ohpc-test/tests/libs/scalapack/tests LOGNAME=ohpc-test MODULESHOME=/opt/ohpc/admin/lmod/lmod PARALLEL_TMP=/tmp/pary5Rcq MANPATH=/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/share/man:/opt/ohpc/pub/compiler/gcc/15.2.0/share/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:/opt/ohpc/admin/lmod/lmod/share/man:: OPENBLAS_INC=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/include BATS_JUNIT_FORMAT=1 CI_CLUSTER=lenovo LMOD_PREPEND_BLOCK=normal BATS_TMPDIR=/tmp __LMOD_REF_COUNT_INCLUDE=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/include:1;/opt/ohpc/pub/compiler/gcc/15.2.0/include:1 BATS_USE_MKTEMP=1 BATS_TEST_DESCRIPTION=[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu15/mvapich2) __LMOD_REF_COUNT_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/bin:1;/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/bin:1;/opt/ohpc/pub/compiler/gcc/15.2.0/bin:1;/opt/ohpc/pub/utils/prun/2.2:1;/home/ohpc-test/.local/bin:1;/home/ohpc-test/bin:1;/sbin:1;/bin:1;/usr/sbin:1;/usr/bin:1;/usr/local/sbin:1 HOME=/home/ohpc-test FI_PROVIDER=sockets ARGS=0 PARALLEL_RSYNC_OPTS=-rlDzR _ModuleTable_Sz_=5 LANG=en_US.UTF-8 BATS_TEST_FILENAME=/home/ohpc-test/tests/libs/scalapack/tests/rm_execution __LMOD_REF_COUNT_PKG_CONFIG_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib/pkgconfig:1;/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/lib/pkgconfig:1 SCALAPACK_LIB=/opt/ohpc/pub/libs/gnu15/mvapich2/scalapack/2.2.3/lib srcdir=. PARALLEL_PID=380957 BATS_RUN_TMPDIR=/tmp/bats-run-hbwgwc LMOD_SETTARG_FULL_SUPPORT=no BATS_JUNIT_CLASS=scalapack BATS_VERSION=1.11.1 TMPDIR=/tmp https_proxy=http://10.241.58.130:3128 OPENBLAS_DIR=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32 LMOD_VERSION=9.2 BATS_GUARANTEED_MINIMUM_VERSION=0.0.0 TESTNAME=libs/ScaLAPACK SINGULARITY_TMPDIR=/var/tmp MFLAGS=-k --no-print-directory _ModuleTable003_=ZW5ibGFzID0gewpmbiA9ICIvb3B0L29ocGMvcHViL21vZHVsZWRlcHMvZ251MTUvb3BlbmJsYXMvMC4zLjMyIiwKZnVsbE5hbWUgPSAib3BlbmJsYXMvMC4zLjMyIiwKbG9hZE9yZGVyID0gNCwKcHJvcFQgPSB7fSwKcmVmX2NvdW50ID0gMSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJvcGVuYmxhcyIsCndWID0gIjAwMDAwMDAwMC4wMDAwMDAwMDMuMDAwMDAwMDMyLip6ZmluYWwiLAp9LApwcnVuID0gewpmbiA9ICIvb3B0L29ocGMvcHViL21vZHVsZWZpbGVzL3BydW4vMi4yIiwKZnVsbE5hbWUgPSAicHJ1bi8yLjIiLApsb2FkT3JkZXIgPSAxLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVz PARALLEL_SEQ=1 __LMOD_REF_COUNT_MPI_DIR=/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1:1 AUTOMAKE_JUNIT_FILE=1 LMOD_FAMILY_OPENBLAS=openblas BATS_CWD=/home/ohpc-test/tests/libs/scalapack/tests MAKEFLAGS=k --no-print-directory -- TEST_LOGS=test_module.log\ rm_execution.log LMOD_PKG=/opt/ohpc/admin/lmod/lmod TERM=unknown BATS_LINE_REFERENCE_FORMAT=comma_line BATS_SEMAPHORE_NUMBER_OF_SLOTS=4 LESSOPEN=||/usr/bin/lesspipe.sh %s PARALLEL_SSHLOGIN=: USER=ohpc-test BATS_BEGIN_CODE_QUOTE=` NODES=2 LOADEDMODULES=prun/2.2:gnu15/15.2.0:mvapich2/4.1:openblas/0.3.32:scalapack/2.2.3 BATS_TEMPDIR_CLEANUP=1 INCLUDE=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/include:/opt/ohpc/pub/compiler/gcc/15.2.0/include LMOD_ROOT=/opt/ohpc/admin/lmod LMOD_SETTARG_CMD=: TASKS=8 SHLVL=10 BASH_ENV=/opt/ohpc/admin/lmod/lmod/init/bash BATS_REPORT_FILENAME=rm_execution.log.xml MAKELEVEL=8 LMOD_FULL_SETTARG_SUPPORT=no MPI_DIR=/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1 __LMOD_REF_COUNT_MANPATH=/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/share/man:1;/opt/ohpc/pub/compiler/gcc/15.2.0/share/man:1;/usr/local/share/man:1;/usr/share/man/overrides:1;/usr/share/man/en:1;/usr/share/man:1;/opt/ohpc/admin/lmod/lmod/share/man:1;:1 http_proxy=http://10.241.58.130:3128 PARALLEL_SSHHOST=: _ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpNUEkgPSAibXZhcGljaDIiLApjb21waWxlciA9ICJnbnUxNSIsCm9wZW5ibGFzID0gIm9wZW5ibGFzIiwKfSwKbVQgPSB7CmdudTE1ID0gewphY3Rpb25BID0gewpbW3ByZXBlbmRfcGF0aCgiTU9EVUxFUEFUSCIsIi9vcHQvb2hwYy9wdWIvbW9kdWxlZGVwcy9nbnUxNSIpXV0sCn0sCmZuID0gIi9vcHQvb2hwYy9wdWIvbW9kdWxlZmlsZXMvZ251MTUvMTUuMi4wLmx1YSIsCmZ1bGxOYW1lID0gImdudTE1LzE1LjIuMCIsCmxvYWRPcmRlciA9IDIsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApz LD_LIBRARY_PATH=/opt/ohpc/pub/libs/gnu15/mvapich2/scalapack/2.2.3/lib:/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib:/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/lib:/opt/ohpc/pub/compiler/gcc/15.2.0/lib64 LMOD_FAMILY_COMPILER=gnu15 LMOD_COLORIZE=no _ModuleTable004_=ZXJOYW1lID0gInBydW4iLAp3ViA9ICIwMDAwMDAwMDIuMDAwMDAwMDAyLip6ZmluYWwiLAp9LApzY2FsYXBhY2sgPSB7CmRlcFQgPSB7CmRlcEEgPSB7CnsKc24gPSAib3BlbmJsYXMiLAp2ZXJzaW9uID0gewpraW5kID0gImJvb2wiLAp2YWx1ZSA9IHRydWUsCn0sCn0sCn0sCn0sCmZuID0gIi9vcHQvb2hwYy9wdWIvbW9kdWxlZGVwcy9nbnUxNS1tdmFwaWNoMi9zY2FsYXBhY2svMi4yLjMiLApmdWxsTmFtZSA9ICJzY2FsYXBhY2svMi4yLjMiLApsb2FkT3JkZXIgPSA1LApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gInNjYWxhcGFjayIsCndWID0gIjAwMDAwMDAwMi4wMDAwMDAwMDIuMDAwMDAwMDAzLip6ZmluYWwi LMOD_FAMILY_MPI_VERSION=4.1 RESOURCE_MANAGER=slurm BATS_TEST_RETRIES=0 OMP_NUM_THREADS=2 LMOD_FAMILY_MPI=mvapich2 DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima: BATS_RUNLOG_FILE=/dev/null LMOD_FAMILY_COMPILER_VERSION=15.2.0 PARALLEL_REMOTE_TMPDIR=/tmp BATS_TEST_NUMBER=1 BATS_ENABLE_TIMING=-T PATH=/usr/libexec/bats-core:/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/bin:/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/bin:/opt/ohpc/pub/compiler/gcc/15.2.0/bin:/opt/ohpc/pub/utils/prun/2.2:/home/ohpc-test/.local/bin:/home/ohpc-test/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin MODULEPATH=/opt/ohpc/pub/moduledeps/gnu15-mvapich2:/opt/ohpc/pub/moduledeps/gnu15:/opt/ohpc/pub/modulefiles SCALAPACK_DIR=/opt/ohpc/pub/libs/gnu15/mvapich2/scalapack/2.2.3 _LMFILES_=/opt/ohpc/pub/modulefiles/prun/2.2:/opt/ohpc/pub/modulefiles/gnu15/15.2.0.lua:/opt/ohpc/pub/moduledeps/gnu15/mvapich2/4.1:/opt/ohpc/pub/moduledeps/gnu15/openblas/0.3.32:/opt/ohpc/pub/moduledeps/gnu15-mvapich2/scalapack/2.2.3 SUDO_UID=0 LMOD_CMD=/opt/ohpc/admin/lmod/lmod/libexec/lmod MAIL=/var/spool/mail/ohpc-test __LMOD_REF_COUNT_LD_LIBRARY_PATH=/opt/ohpc/pub/libs/gnu15/mvapich2/scalapack/2.2.3/lib:1;/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib:1;/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/lib:1;/opt/ohpc/pub/compiler/gcc/15.2.0/lib64:1 BATS_JUNIT_GROUP=UserLevelTests BATS_LIB_PATH=/usr/lib/bats OPENBLAS_LIB=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/lib PARALLEL_JOBSLOT=1 BATS_LIBEXEC=/usr/libexec/bats-core BATS_SUITE_TEST_NUMBER=1 BATS_ROOT=/usr OLDPWD=/home/ohpc-test/tests/libs/scalapack/tests VERBOSE=1 BATS_SAVED_PATH=/opt/ohpc/pub/libs/gnu15/openblas/0.3.32/bin:/opt/ohpc/pub/mpi/mvapich2-gnu15/4.1/bin:/opt/ohpc/pub/compiler/gcc/15.2.0/bin:/opt/ohpc/pub/utils/prun/2.2:/home/ohpc-test/.local/bin:/home/ohpc-test/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin BASH_FUNC_ml%%=() { eval "$($LMOD_DIR/ml_cmd "$@")" } BASH_FUNC_bats_semaphore_acquire_while_locked%%=() { if [[ $(bats_semaphore_get_free_slot_count) -gt 0 ]]; then local slot=0; while [[ -e "$BATS_SEMAPHORE_DIR/slot-$slot" ]]; do ((++slot)); done; if [[ $slot -lt $BATS_SEMAPHORE_NUMBER_OF_SLOTS ]]; then touch "$BATS_SEMAPHORE_DIR/slot-$slot" && printf "%d\n" "$slot" && return 0; fi; fi; return 1 } BASH_FUNC_bats_readlinkf%%=() { readlink -f "$1" } BASH_FUNC_bats_semaphore_get_free_slot_count%%=() { until used_slots=$(find "$BATS_SEMAPHORE_DIR" -name 'slot-*' 2> /dev/null | wc -l); do :; done; echo $((BATS_SEMAPHORE_NUMBER_OF_SLOTS - used_slots)) } BASH_FUNC_module%%=() { if [ -z "${LMOD_SH_DBG_ON+x}" ]; then case "$-" in *v*x*) __lmod_sh_dbg='vx' ;; *v*) __lmod_sh_dbg='v' ;; *x*) __lmod_sh_dbg='x' ;; esac; fi; if [ -n "${__lmod_sh_dbg:-}" ]; then set +$__lmod_sh_dbg; echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output" 1>&2; fi; eval "$($LMOD_CMD shell "$@")" && eval "$(${LMOD_SETTARG_CMD:-:} -s sh)"; __lmod_my_status=$?; if [ -n "${__lmod_sh_dbg:-}" ]; then echo "Shell debugging restarted" 1>&2; set -$__lmod_sh_dbg; fi; unset __lmod_sh_dbg; return $__lmod_my_status } _=/bin/printenv job script = /tmp/job.ohpc-test.23372 Batch job 345 submitted Job 345 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pcscaex 0 (family=mvapich2) [1776237084.455493] [c2:18932:0] ib_iface.c:1011 UCX ERROR iface=0x315301b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc16ac7aac, argv=0x7ffc16ac7ab0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237084.460879] [c1:41520:0] ib_iface.c:1011 UCX ERROR iface=0x2d7503c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(541190415) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffee542126c, argv=0x7ffee5421270) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237084.464020] [c1:41521:0] ib_iface.c:1011 UCX ERROR iface=0x87b53c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 18931 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c1: task 0: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 44) `run_mpi_binary ./pdscaex "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.19768 Batch job 346 submitted Job 346 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pdscaex 0 (family=mvapich2) [1776237086.010790] [c2:19020:0] ib_iface.c:1011 UCX ERROR iface=0x21afa1b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237086.010929] [c2:19018:0] ib_iface.c:1011 UCX ERROR iface=0x180941b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(541190415) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd28577e8c, argv=0x7ffd28577e90) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(138537231) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fffb71ec03c, argv=0x7fffb71ec040) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237086.017710] [c1:41697:0] ib_iface.c:1011 UCX ERROR iface=0x3f733c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237086.020242] [c1:41699:0] ib_iface.c:1011 UCX ERROR iface=0x1f0923c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237086.028815] [c1:41698:0] ib_iface.c:1011 UCX ERROR iface=0xbc983c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(742517007) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffcb417b09c, argv=0x7ffcb417b0a0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 41697 RUNNING AT c1 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:1@c2] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:1@c2] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:1@c2] main (proxy/pmip.c:182): demux engine error waiting for event srun: error: c2: task 1: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 53) `run_mpi_binary ./psscaex "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.11113 Batch job 347 submitted Job 347 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./psscaex 0 (family=mvapich2) [1776237086.573790] [c1:41876:0] ib_iface.c:1011 UCX ERROR iface=0x332053c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237086.573872] [c1:41879:0] ib_iface.c:1011 UCX ERROR iface=0x8d063c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(541190415) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd85cde81c, argv=0x7ffd85cde820) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(4319503) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffeecc9d2fc, argv=0x7ffeecc9d300) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237086.585943] [c2:19104:0] ib_iface.c:1011 UCX ERROR iface=0x341b31b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237086.586786] [c2:19107:0] ib_iface.c:1011 UCX ERROR iface=0x775e1b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(406972687) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffddb9932dc, argv=0x7ffddb9932e0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(272754959) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fffdbe5662c, argv=0x7fffdbe56630) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 62) `run_mpi_binary ./pzscaex "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.4524 Batch job 348 submitted Job 348 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pzscaex 0 (family=mvapich2) [1776237087.193830] [c1:42055:0] ib_iface.c:1011 UCX ERROR iface=0x3c3153c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237087.195093] [c1:42058:0] ib_iface.c:1011 UCX ERROR iface=0x12a923c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(4319503) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc10be078c, argv=0x7ffc10be0790) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(809625871) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe0da86fbc, argv=0x7ffe0da86fc0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237087.200229] [c2:19193:0] ib_iface.c:1011 UCX ERROR iface=0x6c1d1b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237087.200841] [c2:19191:0] ib_iface.c:1011 UCX ERROR iface=0x75541b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(272754959) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffda99738cc, argv=0x7ffda99738d0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(138537231) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc12bbc86c, argv=0x7ffc12bbc870) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 23) `run_mpi_binary ./test4f "" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.32590 Batch job 360 submitted Job 360 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./test4f (family=mvapich2) [1776237158.401284] [c1:45329:0] ib_iface.c:1011 UCX ERROR iface=0x1721fd80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237158.404509] [c2:19993:0] ib_iface.c:1011 UCX ERROR iface=0x7245c60: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 32) `run_mpi_binary ./ex15 "" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.16889 Batch job 361 submitted Job 361 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex15 (family=mvapich2) [1776237160.417548] [c2:20076:0] ib_iface.c:1011 UCX ERROR iface=0x17123b60: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237160.416455] [c1:45503:0] ib_iface.c:1011 UCX ERROR iface=0x278daf10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299535) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffce72c8bcc, argv=0x7ffce72c8bc0, required=1, provided=0x7ffce72c873c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339864079) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe10ac480c, argv=0x7ffe10ac4800, required=1, provided=0x7ffe10ac437c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 41) `run_mpi_binary ./ex16f90 " -pep_nev 4 -terse " 1 1' failed job script = /tmp/job.ohpc-test.15559 Batch job 362 submitted Job 362 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex16f90 -pep_nev 4 -terse (family=mvapich2) [1776237163.417388] [c1:45677:0] ib_iface.c:1011 UCX ERROR iface=0x283de90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 50) `run_mpi_binary ./ex29 "" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.23753 Batch job 363 submitted Job 363 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./ex29 (family=mvapich2) [1776237166.427269] [c1:45850:0] ib_iface.c:1011 UCX ERROR iface=0x2e22af10: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(943843855) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffcfb05044c, argv=0x7ffcfb050440, required=1, provided=0x7ffcfb04ffbc) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237166.431138] [c2:20159:0] ib_iface.c:1011 UCX ERROR iface=0x1a8e9b60: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428623) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffff5a213bc, argv=0x7ffff5a213b0, required=1, provided=0x7ffff5a20f2c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 39) `run_mpi_binary -t "${TIMEOUT}" ./pddrive "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.1004 Batch job 379 submitted Job 379 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive -r 2 -c 2 g20.rua (family=mvapich2) [1776237251.406530] [c2:21376:0] ib_iface.c:1011 UCX ERROR iface=0x1ff59d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734991) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffcdce6842c, argv=0x7ffcdce68420, required=3, provided=0x7ffcdce6843c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237251.410623] [c1:48442:0] ib_iface.c:1011 UCX ERROR iface=0x3cfccd80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(406972943) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe4da1a1ec, argv=0x7ffe4da1a1e0, required=3, provided=0x7ffe4da1a1fc) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 48) `run_mpi_binary -t "${TIMEOUT}" ./pddrive_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.9720 Batch job 380 submitted Job 380 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive_ABglobal -r 1 -c 1 g20.rua (family=mvapich2) [1776237251.989096] [c2:21458:0] ib_iface.c:1011 UCX ERROR iface=0x3970cd20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc0748961c, argv=0x7ffc07489610) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237251.989621] [c1:48616:0] ib_iface.c:1011 UCX ERROR iface=0x1bc53d80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428367) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc0f653d0c, argv=0x7ffc0f653d00) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 57) `run_mpi_binary -t "${TIMEOUT}" ./pddrive1 "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.10803 Batch job 381 submitted Job 381 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive1 -r 2 -c 2 g20.rua (family=mvapich2) [1776237252.562760] [c1:48792:0] ib_iface.c:1011 UCX ERROR iface=0x16abd80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339864079) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffdc21d7f0c, argv=0x7ffdc21d7f00, required=3, provided=0x7ffdc21d7f1c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237252.577533] [c2:21541:0] ib_iface.c:1011 UCX ERROR iface=0x2d12ad20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339864079) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe215bcedc, argv=0x7ffe215bced0, required=3, provided=0x7ffe215bceec) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 66) `run_mpi_binary -t "${TIMEOUT}" ./pddrive1_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.9084 Batch job 382 submitted Job 382 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive1_ABglobal -r 1 -c 1 g20.rua (family=mvapich2) [1776237253.152976] [c1:48967:0] ib_iface.c:1011 UCX ERROR iface=0x305a7d80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdd01568cc, argv=0x7ffdd01568c0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237253.161263] [c2:21625:0] ib_iface.c:1011 UCX ERROR iface=0x39b68d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc54277a4c, argv=0x7ffc54277a40) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 75) `run_mpi_binary -t "${TIMEOUT}" ./pddrive2 "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.29039 Batch job 383 submitted Job 383 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive2 -r 2 -c 2 g20.rua (family=mvapich2) [1776237253.735389] [c1:49141:0] ib_iface.c:1011 UCX ERROR iface=0x289cfd80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(205646351) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc44c9590c, argv=0x7ffc44c95900, required=3, provided=0x7ffc44c9591c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237253.739945] [c2:21707:0] ib_iface.c:1011 UCX ERROR iface=0x187d0d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(943843855) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe0900e47c, argv=0x7ffe0900e470, required=3, provided=0x7ffe0900e48c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 84) `run_mpi_binary -t "${TIMEOUT}" ./pddrive2_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.24938 Batch job 384 submitted Job 384 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive2_ABglobal -r 1 -c 1 g20.rua (family=mvapich2) [1776237254.294116] [c1:49317:0] ib_iface.c:1011 UCX ERROR iface=0x38cc8d80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428367) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffff8c8066c, argv=0x7ffff8c80660) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237254.297433] [c2:21791:0] ib_iface.c:1011 UCX ERROR iface=0xab8d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734735) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe8f47649c, argv=0x7ffe8f476490) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 93) `run_mpi_binary -t "${TIMEOUT}" ./pddrive3 "-r 2 -c 2 g20.rua" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.32549 Batch job 385 submitted Job 385 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive3 -r 2 -c 2 g20.rua (family=mvapich2) [1776237254.842889] [c2:21873:0] ib_iface.c:1011 UCX ERROR iface=0x3aaed20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(742517263) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff7295f6dc, argv=0x7fff7295f6d0, required=3, provided=0x7fff7295f6ec) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237254.844015] [c1:49491:0] ib_iface.c:1011 UCX ERROR iface=0x233aad80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(272755215) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffde07c5ffc, argv=0x7ffde07c5ff0, required=3, provided=0x7ffde07c600c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 102) `run_mpi_binary -t "${TIMEOUT}" ./pddrive3_ABglobal "-r 1 -c 1 g20.rua" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.6982 Batch job 386 submitted Job 386 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive3_ABglobal -r 1 -c 1 g20.rua (family=mvapich2) [1776237255.414046] [c2:21957:0] ib_iface.c:1011 UCX ERROR iface=0x15897d20: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(205646095) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc9614a7ec, argv=0x7ffc9614a7e0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237255.420912] [c1:49666:0] ib_iface.c:1011 UCX ERROR iface=0x6261d80: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff671274fc, argv=0x7fff671274f0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 115) `run_mpi_binary -t "${TIMEOUT}" ./pddrive4 "g20.rua" "${NODES}" 10' failed job script = /tmp/job.ohpc-test.18847 Batch job 387 submitted Job 387 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./pddrive4 g20.rua (family=mvapich2) [1776237255.965516] [c1:49841:0] ib_iface.c:1011 UCX ERROR iface=0x19410490: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237255.965525] [c1:49842:0] ib_iface.c:1011 UCX ERROR iface=0x366a5490: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237255.965758] [c1:49843:0] ib_iface.c:1011 UCX ERROR iface=0x1e1ce490: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(272755215) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd5223005c, argv=0x7ffd52230050, required=3, provided=0x7ffd5223007c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(608299535) on node 1: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffce3520d6c, argv=0x7ffce3520d60, required=3, provided=0x7ffce3520d8c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(809626127) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff6c2c43bc, argv=0x7fff6c2c43b0, required=3, provided=0x7fff6c2c43dc) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237255.971368] [c2:22041:0] ib_iface.c:1011 UCX ERROR iface=0x2b616380: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237255.971375] [c2:22043:0] ib_iface.c:1011 UCX ERROR iface=0x2ed04380: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237255.971551] [c2:22040:0] ib_iface.c:1011 UCX ERROR iface=0x36c94380: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(272755215) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff92227edc, argv=0x7fff92227ed0, required=3, provided=0x7fff92227efc) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(876734991) on node 9: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc79eb47ec, argv=0x7ffc79eb47e0, required=3, provided=0x7ffc79eb480c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(1010952719) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fffc8f6881c, argv=0x7fffc8f68810, required=3, provided=0x7fffc8f6883c) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 39) `run_mpi_binary -t "${TIMEOUT}" "${EXE}" "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.31889 Batch job 404 submitted Job 404 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./lesson_tpetra_init null (family=mvapich2) [1776237365.925273] [c1:52896:0] ib_iface.c:1011 UCX ERROR iface=0x224fda40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc97dfc4ac, argv=0x7ffc97dfc4a0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237365.933934] [c2:23532:0] ib_iface.c:1011 UCX ERROR iface=0x2b1c58a0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe2cf9122c, argv=0x7ffe2cf91220) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237365.936419] [c1:52897:0] ib_iface.c:1011 UCX ERROR iface=0x3d788a40: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument(from function `run_mpi_binary' in file ../../../common/functions, line 434, in test file rm_execution, line 49) `run_mpi_binary -t "${TIMEOUT}" "${EXE}" "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.12902 Batch job 405 submitted Job 405 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./lesson_tpetra_dataredist null (family=mvapich2) [1776237366.492182] [c2:23616:0] ib_iface.c:1011 UCX ERROR iface=0xd11a940: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237366.490366] [c1:53072:0] ib_iface.c:1011 UCX ERROR iface=0xcd88b50: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734735) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc6deab66c, argv=0x7ffc6deab660) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(474081551) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd5ea0c48c, argv=0x7ffd5ea0c480) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../common/functions, line 434, in test file rm_execution_multi_host, line 27) `run_mpi_binary -t "${TIMEOUT}" ./C_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.6765 Batch job 424 submitted Job 424 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_test 8 (family=mvapich2) [1776237440.400151] [c2:25268:0] ib_iface.c:1011 UCX ERROR iface=0x2e8f7bf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237440.400442] [c2:25266:0] ib_iface.c:1011 UCX ERROR iface=0x33b6abf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237440.404310] [c1:55915:0] ib_iface.c:1011 UCX ERROR iface=0x203ce00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237440.404311] [c1:55916:0] ib_iface.c:1011 UCX ERROR iface=0x34807e00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(1010952463) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff29db487c, argv=0x7fff29db4870) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(205646095) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffef2938abc, argv=0x7ffef2938ab0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(205646095) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fffeb25a1fc, argv=0x7fffeb25a1f0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(876734735) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc97a099cc, argv=0x7ffc97a099c0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../common/functions, line 434, in test file rm_execution_multi_host, line 36) `run_mpi_binary -t "${TIMEOUT}" ./CXX_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.24094 Batch job 425 submitted Job 425 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./CXX_test 8 (family=mvapich2) [1776237440.950821] [c1:56094:0] ib_iface.c:1011 UCX ERROR iface=0x17f94e00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd8803e12c, argv=0x7ffd8803e120) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237440.955039] [c2:25353:0] ib_iface.c:1011 UCX ERROR iface=0x3387fbf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237440.953032] [c1:56092:0] ib_iface.c:1011 UCX ERROR iface=0xc56ae00: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237440.955664] [c2:25352:0] ib_iface.c:1011 UCX ERROR iface=0x34e78bf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339863823) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdd814d7ec, argv=0x7ffdd814d7e0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(205646095) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffce34258fc, argv=0x7ffce34258f0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 56092 RUNNING AT c1 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions(from function `run_mpi_binary' in file ../../common/functions, line 434, in test file rm_execution_multi_host, line 45) `run_mpi_binary -t "${TIMEOUT}" ./F90_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.11802 Batch job 426 submitted Job 426 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F90_test 8 (family=mvapich2) [1776237441.568556] [c2:25441:0] ib_iface.c:1011 UCX ERROR iface=0x13919120: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237441.568726] [c1:56272:0] ib_iface.c:1011 UCX ERROR iface=0xc74b330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237441.568714] [c1:56274:0] ib_iface.c:1011 UCX ERROR iface=0x3a063330: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237441.571317] [c2:25442:0] ib_iface.c:1011 UCX ERROR iface=0x10a4d120: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 25442 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c1: task 0: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion(from function `run_mpi_binary' in file ../../common/functions, line 434, in test file rm_execution_single_host, line 27) `run_mpi_binary -t "${TIMEOUT}" ./C_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.13388 Batch job 421 submitted Job 421 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./C_test 2 (family=mvapich2) [1776237435.396832] [c1:55739:0] ib_iface.c:1011 UCX ERROR iface=0x1dd86190: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe4a01f3bc, argv=0x7ffe4a01f3b0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../common/functions, line 434, in test file rm_execution_single_host, line 36) `run_mpi_binary -t "${TIMEOUT}" ./CXX_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.10256 Batch job 422 submitted Job 422 failed... Reason=NonZeroExitCode [prun] Master compute host = c2 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./CXX_test 2 (family=mvapich2) [1776237435.373714] [c2:25007:0] ib_iface.c:1011 UCX ERROR iface=0x5c74cb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffca39109dc, argv=0x7ffca39109d0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ../../common/functions, line 434, in test file rm_execution_single_host, line 45) `run_mpi_binary -t "${TIMEOUT}" ./F90_test "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.19852 Batch job 423 submitted Job 423 failed... Reason=NonZeroExitCode [prun] Master compute host = c2 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./F90_test 2 (family=mvapich2) [1776237435.920107] [c2:25181:0] ib_iface.c:1011 UCX ERROR iface=0x14818af0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)Warewulf not installed, skipping warewulf-ipmitool checkskipped(from function `run_mpi_binary' in file ./common/functions, line 434, in test file test_imb_mpi1, line 38) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.2289 Batch job 437 submitted Job 437 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm /opt/ohpc/pub/libs/gnu15/mvapich2/imb/2021.11/bin/IMB-MPI1 -off_cache -1 -time 1.5 -npmin 100 -msglog 1:4 PingPong Sendrecv Bcast Allreduce (family=mvapich2) [1776237495.578867] [c1:58114:0] ib_iface.c:1011 UCX ERROR iface=0x29d3700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237495.578869] [c1:58116:0] ib_iface.c:1011 UCX ERROR iface=0x286c1700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237495.578863] [c1:58120:0] ib_iface.c:1011 UCX ERROR iface=0x3f43d700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237495.578864] [c1:58121:0] ib_iface.c:1011 UCX ERROR iface=0x2dab7700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237495.579663] [c1:58119:0] ib_iface.c:1011 UCX ERROR iface=0x1c6ae700: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734991) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffde7a1277c, argv=0x7ffde7a12770, required=0, provided=0x7ffde7a12798) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(138537487) on node 2: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc12ccfebc, argv=0x7ffc12ccfeb0, required=0, provided=0x7ffc12ccfed8) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(474081807) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fffdf364bac, argv=0x7fffdf364ba0, required=0, provided=0x7fffdf364bc8) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(138537487) on node 7: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe5dcb1afc, argv=0x7ffe5dcb1af0, required=0, provided=0x7ffe5dcb1b18) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(339864079) on node 5: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fffcb3b316c, argv=0x7fffcb3b3160, required=0, provided=0x7fffcb3b3188) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237495.583691] [c2:26357:0] ib_iface.c:1011 UCX ERROR iface=0x81c34f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237495.583764] [c2:26359:0] ib_iface.c:1011 UCX ERROR iface=0x331694f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237495.583773] [c2:26361:0] ib_iface.c:1011 UCX ERROR iface=0x340864f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237495.583799] [c2:26356:0] ib_iface.c:1011 UCX ERROR iface=0x3d2d54f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237495.583820] [c2:26355:0] ib_iface.c:1011 UCX ERROR iface=0x1b1e04f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237495.583922] [c2:26360:0] ib_iface.c:1011 UCX ERROR iface=0x2acd64f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408399) on node 10: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc870732cc, argv=0x7ffc870732c0, required=0, provided=0x7ffc870732e8) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(809626127) on node 12: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffebaab069c, argv=0x7ffebaab0690, required=0, provided=0x7ffebaab06b8) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(876734991) on node 14: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff701dd75c, argv=0x7fff701dd750, required=0, provided=0x7fff701dd778) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(541190671) on node 8: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd7d181a1c, argv=0x7ffd7d181a10, required=0, provided=0x7ffd7d181a38) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(4319759) on node 9: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffee219fdcc, argv=0x7ffee219fdc0, required=0, provided=0x7ffee219fde8) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(406972943) on node 13: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc80ff770c, argv=0x7ffc80ff7700, required=0, provided=0x7ffc80ff7728) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file test_imb_mpi2, line 41) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.30089 Batch job 438 submitted Job 438 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm /opt/ohpc/pub/libs/gnu15/mvapich2/imb/2021.11/bin/IMB-EXT -npmin 100 -msglog 1:4 Window (family=mvapich2) [1776237500.356734] [c1:58304:0] ib_iface.c:1011 UCX ERROR iface=0x98a5cd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(4319759) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffdc2c8960c, argv=0x7ffdc2c89600, required=0, provided=0x7ffdc2c89628) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237500.365558] [c2:26451:0] ib_iface.c:1011 UCX ERROR iface=0x299565c0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(406972943) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffd9771da4c, argv=0x7ffd9771da40, required=0, provided=0x7ffd9771da68) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file test_imb_mpi2, line 53) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.19936 Batch job 439 submitted Job 439 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm /opt/ohpc/pub/libs/gnu15/mvapich2/imb/2021.11/bin/IMB-IO -npmin 100 Open_Close (family=mvapich2) [1776237503.361229] [c2:26534:0] ib_iface.c:1011 UCX ERROR iface=0x171a9bd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(272755215) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffc88042a8c, argv=0x7ffc88042a80, required=0, provided=0x7ffc88042aa8) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237503.361861] [c1:58479:0] ib_iface.c:1011 UCX ERROR iface=0x2c4cfdf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237503.372929] [c1:58480:0] ib_iface.c:1011 UCX ERROR iface=0xe33adf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428623) on node 0: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff2dd911ec, argv=0x7fff2dd911e0, required=0, provided=0x7fff2dd91208) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)skipped(from function `run_mpi_binary' in file ./common/functions, line 434, in test file test_imb_mpi3, line 56) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.2871 Batch job 440 submitted Job 440 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm /opt/ohpc/pub/libs/gnu15/mvapich2/imb/2021.11/bin/IMB-RMA -npmin 100 -msglog 1:4 Unidir_put Unidir_get (family=mvapich2) [1776237507.402562] [c2:26618:0] ib_iface.c:1011 UCX ERROR iface=0xd4ce170: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(138537487) on node 6: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7ffe52ca2f1c, argv=0x7ffe52ca2f10, required=0, provided=0x7ffe52ca2f38) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237507.403436] [c1:58657:0] ib_iface.c:1011 UCX ERROR iface=0x3c736280: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299535) on node 3: Fatal error in internal_Init_thread: Other MPI error, error stack: internal_Init_thread(71)......: MPI_Init_thread(argc=0x7fff4083944c, argv=0x7fff40839440, required=0, provided=0x7fff40839468) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 26617 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c1: task 0: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completionskipped(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 23) `run_mpi_binary -t "$TIMEOUT" "$EXE" "-m 512" 2 2' failed job script = /tmp/job.ohpc-test.16648 Batch job 447 submitted Job 447 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm osu_bw -m 512 (family=mvapich2) [1776237562.344082] [c1:59922:0] ib_iface.c:1011 UCX ERROR iface=0x31e1f1d0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe57acd58c, argv=0x7ffe57acd580) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237562.356814] [c2:27252:0] ib_iface.c:1011 UCX ERROR iface=0x2eaf5cf0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(1010952463) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff5ddffb9c, argv=0x7fff5ddffb90) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 36) `run_mpi_binary -t "$TIMEOUT" "$EXE" "-m $MESSAGE_SIZE" 2 2' failed job script = /tmp/job.ohpc-test.5809 Batch job 448 submitted Job 448 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm osu_latency -m 512 (family=mvapich2) [1776237562.875963] [c1:60095:0] ib_iface.c:1011 UCX ERROR iface=0x4299190: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe6d3a019c, argv=0x7ffe6d3a0190) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237562.900732] [c2:27333:0] ib_iface.c:1011 UCX ERROR iface=0x2e30bcb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc480a39cc, argv=0x7ffc480a39c0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 24) `run_mpi_binary -s 1 ./mpi/C_mpi_test $ARGS $NODES "$TASKS"' failed job script = /tmp/job.ohpc-test.31023 Batch job 485 submitted Job 485 failed... Reason=NonZeroExitCode S=C=A=N: Scalasca 2.6.2 trace collection and analysis S=C=A=N: ./scorep_C_mpi_test_8x2_trace experiment archive S=C=A=N: Wed Apr 15 07:23:39 2026: Collect start /opt/ohpc/pub/utils/prun/2.2/prun ./mpi/C_mpi_test 8 [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/C_mpi_test 8 (family=mvapich2) [1776237822.144656] [c2:30378:0] ib_iface.c:1011 UCX ERROR iface=0x244b6cb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(742517007) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff9437425c, argv=0x7fff94374250) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237822.149715] [c2:30380:0] ib_iface.c:1011 UCX ERROR iface=0x3c82ccb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237822.148585] [c1:65857:0] ib_iface.c:1011 UCX ERROR iface=0x1a743e30: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237822.148849] [c1:65856:0] ib_iface.c:1011 UCX ERROR iface=0x3a8cee30: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd0d5c48cc, argv=0x7ffd0d5c48c0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 30380 RUNNING AT c2 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0@c1] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed [proxy:0@c1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [proxy:0@c1] main (proxy/pmip.c:122): demux engine error waiting for event srun: error: c1: task 0: Exited with exit code 7 [mpiexec@c1] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting [mpiexec@c1] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec@c1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion [mpiexec@c1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion S=C=A=N: Wed Apr 15 07:23:43 2026: Collect done (status=0) 4s Abort: missing experiment archive ./scorep_C_mpi_test_8x2_trace(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 38) `run_mpi_binary -s 1 ./mpi/CXX_mpi_test $ARGS $NODES "$TASKS"' failed job script = /tmp/job.ohpc-test.22464 Batch job 486 submitted Job 486 failed... Reason=NonZeroExitCode S=C=A=N: Scalasca 2.6.2 trace collection and analysis S=C=A=N: ./scorep_CXX_mpi_test_8x2_trace experiment archive S=C=A=N: Wed Apr 15 07:23:45 2026: Collect start /opt/ohpc/pub/utils/prun/2.2/prun ./mpi/CXX_mpi_test 8 [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/CXX_mpi_test 8 (family=mvapich2) [1776237826.100954] [c2:30465:0] ib_iface.c:1011 UCX ERROR iface=0x1d548eb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237826.100955] [c2:30468:0] ib_iface.c:1011 UCX ERROR iface=0x1a66aeb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(71428367) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffca54210cc, argv=0x7ffca54210c0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(876734735) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd93c7f54c, argv=0x7ffd93c7f540) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237826.107632] [c1:66155:0] ib_iface.c:1011 UCX ERROR iface=0x324f1030: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237826.108330] [c1:66157:0] ib_iface.c:1011 UCX ERROR iface=0x3065b030: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd69093b4c, argv=0x7ffd69093b40) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(205646095) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc9653afcc, argv=0x7ffc9653afc0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) S=C=A=N: Wed Apr 15 07:23:46 2026: Collect done (status=0) 1s Abort: missing experiment archive ./scorep_CXX_mpi_test_8x2_trace(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 59) `run_mpi_binary ./mpi/main_mpi_c "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.30102 Batch job 463 submitted Job 463 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_mpi_c 8 (family=mvapich2) [1776237708.499348] [c2:28607:0] ib_iface.c:1011 UCX ERROR iface=0x368bff90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe8927276c, argv=0x7ffe89272760) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237708.500373] [c1:62384:0] ib_iface.c:1011 UCX ERROR iface=0x11834110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237708.501568] [c1:62383:0] ib_iface.c:1011 UCX ERROR iface=0x378eb110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(1010952463) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc27fae64c, argv=0x7ffc27fae640) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237708.504968] [c2:28606:0] ib_iface.c:1011 UCX ERROR iface=0x11190f90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(608299279) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffced7df64c, argv=0x7ffced7df640) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(71428367) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd226fc0dc, argv=0x7ffd226fc0d0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237708.507182] [c1:62382:0] ib_iface.c:1011 UCX ERROR iface=0x1cf71110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237708.509531] [c2:28604:0] ib_iface.c:1011 UCX ERROR iface=0x27205f90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(1010952463) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd1e15e05c, argv=0x7ffd1e15e050) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 73) `run_mpi_binary ./mpi/main_mpi_cxx "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.26937 Batch job 464 submitted Job 464 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_mpi_cxx 8 (family=mvapich2) [1776237709.131013] [c1:62562:0] ib_iface.c:1011 UCX ERROR iface=0x1fdbd110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237709.131540] [c1:62561:0] ib_iface.c:1011 UCX ERROR iface=0x1740d110: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd1e80534c, argv=0x7ffd1e805340) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(71428367) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffdc14d4d3c, argv=0x7ffdc14d4d30) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237709.137700] [c2:28693:0] ib_iface.c:1011 UCX ERROR iface=0x80e1f90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237709.137876] [c2:28694:0] ib_iface.c:1011 UCX ERROR iface=0x16a92f90: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339863823) on node 6: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7fff02ab037c, argv=0x7fff02ab0370) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(876734735) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe70f280ec, argv=0x7ffe70f280e0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 87) `run_mpi_binary ./mpi/main_mpi_fort "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.24721 Batch job 465 submitted Job 465 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_mpi_fort 8 (family=mvapich2) [1776237709.763435] [c2:28778:0] ib_iface.c:1011 UCX ERROR iface=0x24032dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237709.763340] [c1:62742:0] ib_iface.c:1011 UCX ERROR iface=0x123ae050: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237709.765313] [c2:28781:0] ib_iface.c:1011 UCX ERROR iface=0x327b1dd0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237709.763986] [c1:62741:0] ib_iface.c:1011 UCX ERROR iface=0xf97c050: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 143) `run_mpi_binary ./mpi/main_hyb_c "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.23342 Batch job 469 submitted Job 469 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_hyb_c 8 (family=mvapich2) [1776237710.936366] [c1:62999:0] ib_iface.c:1011 UCX ERROR iface=0x20fb4800: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(742517007) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe7baec4ec, argv=0x7ffe7baec4e0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237710.941828] [c2:29025:0] ib_iface.c:1011 UCX ERROR iface=0x3733680: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237710.941994] [c2:29026:0] ib_iface.c:1011 UCX ERROR iface=0x2e26a680: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(474081551) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffca8d0b40c, argv=0x7ffca8d0b400) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(205646095) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd018ab8dc, argv=0x7ffd018ab8d0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237710.941825] [c1:63002:0] ib_iface.c:1011 UCX ERROR iface=0x2247a800: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 63002 RUNNING AT c1 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 157) `run_mpi_binary ./mpi/main_hyb_cxx "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.25218 Batch job 470 submitted Job 470 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_hyb_cxx 8 (family=mvapich2) [1776237711.542356] [c1:63181:0] ib_iface.c:1011 UCX ERROR iface=0x1172f470: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339863823) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd5b76ecec, argv=0x7ffd5b76ece0) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237711.544710] [c1:63180:0] ib_iface.c:1011 UCX ERROR iface=0x215c8470: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734735) on node 2: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc2981990c, argv=0x7ffc29819900) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237711.549256] [c2:29112:0] ib_iface.c:1011 UCX ERROR iface=0x336112f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237711.549494] [c2:29115:0] ib_iface.c:1011 UCX ERROR iface=0x3d53d2f0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(876734735) on node 4: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd159fd13c, argv=0x7ffd159fd130) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(876734735) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffe9a976c1c, argv=0x7ffe9a976c10) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error)(from function `run_mpi_binary' in file ./common/functions, line 434, in test file rm_execution, line 171) `run_mpi_binary ./mpi/main_hyb_fort "${ARGS}" "${NODES}" "${TASKS}"' failed job script = /tmp/job.ohpc-test.30182 Batch job 471 submitted Job 471 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi/main_hyb_fort 8 (family=mvapich2) [1776237712.119652] [c2:29200:0] ib_iface.c:1011 UCX ERROR iface=0x17192530: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237712.119794] [c2:29202:0] ib_iface.c:1011 UCX ERROR iface=0x23d3a530: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 5: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) Abort(675408143) on node 7: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237712.123148] [c1:63358:0] ib_iface.c:1011 UCX ERROR iface=0x1049a7b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument [1776237712.123516] [c1:63360:0] ib_iface.c:1011 UCX ERROR iface=0x2c6457b0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(675408143) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 63358 RUNNING AT c1 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestionsdisabled timeout test(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58, in test file test_harness, line 29) `assert_success' failed -- command failed -- status : 1 output (31 lines): job script = /tmp/job.ohpc-test.16460 Batch job 495 submitted Job 495 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./mpi_exit 0 (family=mvapich2) [1776237888.382062] [c1:67689:0] ib_iface.c:1011 UCX ERROR iface=0x1d2af190: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339863823) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffd2acfab2c, argv=0x7ffd2acfab20) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) [1776237888.393156] [c2:31245:0] ib_iface.c:1011 UCX ERROR iface=0x270aacb0: failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64: Invalid argument failed: Invalid argument Abort(339863823) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=0x7ffc3bc7e51c, argv=0x7ffc3bc7e510) failed MPII_Init_thread(288).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(794).........: MPIR_Comm_commit_internal(579): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(668).....: MPIDI_UCX_init_world(277).....: init_worker(48)...............: ucx function returned with failed status(ucx_init.c 48 init_worker Input/output error) --disabled timeout testdisabled timeout testskip experimental charliecloud image builderTemporarily skip sacct checkdisable to invistigate disk fillup