| Duration | 1.06 sec |
|---|---|
| Tests | 5 |
| Failures | 0 |
| Duration | 1.149 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Duration | 16.156 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 15.082 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 6.605 sec |
|---|---|
| Tests | 3 |
| Failures | 2 |
| Test case: | [/apps/hpcg] check if resource manager is defined |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Test case: | [/apps/hpcg] run HPCG on multi nodes under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 6.554 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414, in test file rm_execution_multi_host, line 46) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.2691 Batch job 13 submitted Job 13 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./bin/xhpcg.gnu14.openmpi5 32 32 32 10 (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148439.157437] [c1:7068 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1634050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07068] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148439.161445] [c1:7064 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x191c010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07064] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148439.176549] [c1:7066 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x21ae050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07066] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148439.185260] [c1:7067 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1c94050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07067] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148439.187466] [c1:7063 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1ebe010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148439.187570] [c1:7070 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2365010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07063] pml_ucx.c:313 Error: Failed to create UCP worker [c1:07070] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148439.203650] [c1:7069 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2749050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07069] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148439.208449] [c1:7065 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x23ba010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07065] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148440.477477] [c2:6701 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xeab050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [1729148440.477912] [c2:6698 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x21ab050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [c2:06701] pml_ucx.c:313 Error: Failed to create UCP worker [c2:06698] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148440.492853] [c2:6702 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x240c050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [1729148440.492858] [c2:6699 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1118050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148440.492983] [c2:6697 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x21f4010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148440.493239] [c2:6700 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xc23050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148440.493881] [c2:6695 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x23a4050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:06702] pml_ucx.c:313 Error: Failed to create UCP worker [c2:06699] pml_ucx.c:313 Error: Failed to create UCP worker [c2:06697] pml_ucx.c:313 Error: Failed to create UCP worker [c2:06700] pml_ucx.c:313 Error: Failed to create UCP worker [c2:06695] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148440.515130] [c2:6696 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1713050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:06696] pml_ucx.c:313 Error: Failed to create UCP worker [c2:6696 :0:6696] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x14159e0) ==== backtrace (tid: 6696) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f19c01aac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f19c01aadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f19c01ab0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f19c0654db0] 4 [0x14159e0] ================================= [c1:7070 :0:7070] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20fef60) [c1:7067 :0:7067] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1c27c70) [c1:7068 :0:7068] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x15c5f50) ==== backtrace (tid: 7070) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f1578facc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f1578facdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f1578fad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f1579454db0] 4 [0x20fef60] ================================= [c1:7064 :0:7064] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x16c2430) [c1:7063 :0:7063] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1c5da70) ==== backtrace (tid: 7067) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fdb9443ac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fdb9443adff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fdb9443b0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fdb94254db0] 4 [0x1c27c70] ================================= [c1:7065 :0:7065] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2153f40) ==== backtrace (tid: 7068) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fcbb4c3cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fcbb4c3cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fcbb4c3d0c6] [c1:7069 :0:7069] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x26daf50) 3 /lib64/libc.so.6(+0x54db0) [0x7fcbb5054db0] 4 [0x15c5f50] ================================= ==== backtrace (tid: 7064) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f52013cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f52013cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f52013cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f5201854db0] ==== backtrace (tid: 7063) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fcc4a1cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fcc4a1cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fcc4a1cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fcc4a654db0] 4 [0x1c5da70] ================================= 4 [0x16c2430] ================================= ==== backtrace (tid: 7065) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f8939fcec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f8939fcedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f8939fcf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f893a454db0] 4 [0x2153f40] ================================= [c2:6697 :0:6697] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1ef4890) ==== backtrace (tid: 7069) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f6617d9fc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f6617d9fdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f6617da00c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f6618254db0] 4 [0x26daf50] ================================= [c2:6699 :0:6699] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xd35a00) [c2:6702 :0:6702] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x211a100) ==== backtrace (tid: 6697) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f493aa3cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f493aa3cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f493aa3d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f493ae54db0] 4 [0x1ef4890] ================================= ==== backtrace (tid: 6699) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f49e143bc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f49e143bdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f49e143c0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f49e1254db0] 4 [0xd35a00] ================================= [c2:6700 :0:6700] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x9259e0) [c2:6701 :0:6701] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xbb8740) ==== backtrace (tid: 6702) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f928b7cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f928b7cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f928b7cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f928bc54db0] 4 [0x211a100] ================================= [c1:7066 :0:7066] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x213ff50) ==== backtrace (tid: 6701) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f76adbcec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f76adbcedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f76adbcf0c6] ==== backtrace (tid: 6700) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fc589f8fc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fc589f8fdff] 3 /lib64/libc.so.6(+0x54db0) [0x7f76ae054db0] 4 [0xbb8740] ================================= 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fc589f900c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fc58a454db0] 4 [0x9259e0] ================================= ==== backtrace (tid: 7066) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f4634fcec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f4634fcedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f4634fcf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f4635454db0] 4 [0x213ff50] ================================= [c2:6695 :0:6695] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20a69e0) [c2:6698 :0:6698] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1eb8740) ==== backtrace (tid: 6695) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fa936e3cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fa936e3cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fa936e3d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fa937254db0] 4 [0x20a69e0] ================================= ==== backtrace (tid: 6698) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f779fe3cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f779fe3cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f779fe3d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f77a0254db0] 4 [0x1eb8740] ================================= -------------------------------------------------------------------------- prterun noticed that process rank 9 with PID 6696 on node c2 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
| Test case: | [/apps/hpcg] log HPCG multi node results (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 0.032 sec |
| Failed | None |
(in test file rm_execution_multi_host, line 55) `mv $run_yaml $wrk_yaml' failed Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory Moving to HPCG.32x32x32.P16.gnu14.openmpi5.yaml mv: missing destination file operand after 'HPCG.32x32x32.P16.gnu14.openmpi5.yaml' Try 'mv --help' for more information.
| Duration | 3.41 sec |
|---|---|
| Tests | 3 |
| Failures | 2 |
| Test case: | [/apps/hpcg] check if resource manager is defined |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Test case: | [/apps/hpcg] run HPCG on single node under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 3.359 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414,
in test file rm_execution_single_host, line 46)
`run_mpi_binary -t $CMD_TIMEOUT ${EXE} "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.5240
Batch job 12 submitted
Job 12 failed...
Reason=NonZeroExitCode
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./bin/xhpcg.gnu14.openmpi5 32 32 32 10 (family=openmpi5)
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148435.264956] [c1:6975 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1da6010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148435.265445] [c1:6976 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xbf7010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148435.265635] [c1:6977 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x15d7010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06975] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148435.266366] [c1:6972 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x175f010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06977] pml_ucx.c:313 Error: Failed to create UCP worker
[c1:06972] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148435.267776] [c1:6979 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x20e1050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148435.268652] [c1:6974 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xe81010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06979] pml_ucx.c:313 Error: Failed to create UCP worker
[c1:06976] pml_ucx.c:313 Error: Failed to create UCP worker
[c1:06974] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148435.272403] [c1:6978 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1b4c010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148435.272621] [c1:6973 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2695050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06973] pml_ucx.c:313 Error: Failed to create UCP worker
[c1:06978] pml_ucx.c:313 Error: Failed to create UCP worker
[c1:6975 :0:6975] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1b3ff40)
[c1:6973 :0:6973] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2439f30)
==== backtrace (tid: 6975) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f1eac6b9c44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f1eac6b9dff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f1eac6ba0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f1eaca54db0]
4 [0x1b3ff40]
=================================
[c1:6972 :0:6972] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x14f8f40)
[c1:6978 :0:6978] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18e5f40)
[c1:6974 :0:6974] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc1eb70)
[c1:6977 :0:6977] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1370f40)
==== backtrace (tid: 6973) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f08f49acc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f08f49acdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f08f49ad0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f08f4e54db0]
4 [0x2439f30]
=================================
==== backtrace (tid: 6972) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f1ef2cfac44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f1ef2cfadff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f1ef2cfb0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f1ef3054db0]
4 [0x14f8f40]
=================================
[c1:6979 :0:6979] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2070f10)
==== backtrace (tid: 6978) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f592f982c44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f592f982dff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f592f9830c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f592fe54db0]
4 [0x18e5f40]
=================================
==== backtrace (tid: 6974) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ff2d4e3cc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ff2d4e3cdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ff2d4e3d0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7ff2d5254db0]
4 [0xc1eb70]
=================================
==== backtrace (tid: 6977) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f913decbc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f913decbdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f913decc0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f913e254db0]
4 [0x1370f40]
=================================
==== backtrace (tid: 6979) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f35c2bacc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f35c2bacdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f35c2bad0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f35c3054db0]
4 [0x2070f10]
=================================
[c1:6976 :0:6976] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x994b70)
==== backtrace (tid: 6976) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f55765acc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f55765acdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f55765ad0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f5576a54db0]
4 [0x994b70]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 7 with PID 6979 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------
| Test case: | [/apps/hpcg] log HPCG single node results (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 0.032 sec |
| Failed | None |
(in test file rm_execution_single_host, line 55) `mv $run_yaml $wrk_yaml' failed Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory Moving to HPCG.32x32x32.P8.gnu14.openmpi5.yaml mv: missing destination file operand after 'HPCG.32x32x32.P8.gnu14.openmpi5.yaml' Try 'mv --help' for more information.
| Duration | 5.536 sec |
|---|---|
| Tests | 2 |
| Failures | 1 |
| Test case: | [Apps/miniFE] run miniFE on multi nodes under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 5.484 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common-test/functions, line 414, in test file rm_execution_multi_host, line 46) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.10855 Batch job 17 submitted Job 17 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./src/miniFE.x.gnu14.openmpi5 nx=256 ny=256 nz=256 verify_solution=0 (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.063058] [c2:6874 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2732010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:06874] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.081351] [c2:6873 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x12d1050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:06873] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.085046] [c2:6876 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x26b7050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:06876] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.123071] [c2:6875 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1c14050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:06875] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.129756] [c2:6872 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1b04050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:06872] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.143533] [c2:6878 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x287a050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:06878] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.174593] [c1:7421 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1338050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07421] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.183985] [c2:6871 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1094050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [c2:06871] pml_ucx.c:313 Error: Failed to create UCP worker [1729148499.187541] [c2:6877 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1c60050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:06877] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.203717] [c1:7420 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x13c0010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [c1:07420] pml_ucx.c:313 Error: Failed to create UCP worker [1729148499.205479] [c1:7416 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x18aa010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07416] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.215657] [c1:7418 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1788050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07418] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.219945] [c1:7422 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1b7d010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07422] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.222807] [c1:7415 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x271c050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.223850] [c1:7419 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1e57050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07415] pml_ucx.c:313 Error: Failed to create UCP worker [c1:07419] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148499.289596] [c1:7417 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xc5b010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07417] pml_ucx.c:313 Error: Failed to create UCP worker [c1:7418 :0:7418] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1719f50) [c1:7415 :0:7415] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x24c0f50) [c1:7421 :0:7421] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x12ca920) [c1:7416 :0:7416] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1643ef0) [c1:7420 :0:7420] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1159f40) ==== backtrace (tid: 7418) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0d3e58fc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0d3e58fdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f0d3e5900c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0d3ea54db0] 4 [0x1719f50] ================================= [c1:7417 :0:7417] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x9faa70) ==== backtrace (tid: 7415) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ffbfa23cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ffbfa23cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ffbfa23d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7ffbfa654db0] 4 [0x24c0f50] ================================= [c1:7419 :0:7419] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1bfbf50) ==== backtrace (tid: 7421) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f03926fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f03926fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f03926fb0c6] ==== backtrace (tid: 7417) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f08f0ed8c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f08f0ed8dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f08f0ed90c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f08f1254db0] 4 [0x9faa70] ================================= 3 /lib64/libc.so.6(+0x54db0) [0x7f0392a54db0] 4 [0x12ca920] ================================= ==== backtrace (tid: 7416) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fc4022fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fc4022fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fc4022fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fc402654db0] 4 [0x1643ef0] ================================= ==== backtrace (tid: 7420) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f531cbacc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f531cbacdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f531cbad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f531d054db0] 4 [0x1159f40] ================================= [c1:7422 :0:7422] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x191cf40) ==== backtrace (tid: 7419) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f049d4cbc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f049d4cbdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f049d4cc0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f049d854db0] 4 [0x1bfbf50] ================================= ==== backtrace (tid: 7422) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f6ab423cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f6ab423cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f6ab423d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f6ab4654db0] 4 [0x191cf40] ================================= [c2:6871 :0:6871] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xda2130) [c2:6872 :0:6872] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18ae510) [c2:6875 :0:6875] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x19bfec0) [c2:6877 :0:6877] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x187da40) ==== backtrace (tid: 6871) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f059e98fc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f059e98fdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f059e9900c6] [c2:6876 :0:6876] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x23c4740) 3 /lib64/libc.so.6(+0x54db0) [0x7f059ee54db0] 4 [0xda2130] ================================= [c2:6874 :0:6874] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2431ee0) [c2:6873 :0:6873] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xfd0e70) ==== backtrace (tid: 6872) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f71581cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f71581cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f71581cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f7158654db0] 4 [0x18ae510] ================================= [c2:6878 :0:6878] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2497a40) ==== backtrace (tid: 6877) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ff04bfcec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ff04bfcedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ff04bfcf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7ff04c454db0] 4 [0x187da40] ================================= ==== backtrace (tid: 6875) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f9d28182c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f9d28182dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f9d281830c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f9d28654db0] 4 [0x19bfec0] ================================= ==== backtrace (tid: 6876) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fb8306fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fb8306fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fb8306fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fb830a54db0] 4 [0x23c4740] ================================= ==== backtrace (tid: 6873) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0d6bb8fc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0d6bb8fdff] ==== backtrace (tid: 6874) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fa790facc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fa790facdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fa790fad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fa791454db0] 4 [0x2431ee0] ==== backtrace (tid: 6878) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ffab523cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ffab523cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ffab523d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7ffab5654db0] 4 [0x2497a40] ================================= 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f0d6bb900c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0d6c054db0] 4 [0xfd0e70] ================================= ================================= -------------------------------------------------------------------------- prterun noticed that process rank 4 with PID 7419 on node c1 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
| Duration | 3.404 sec |
|---|---|
| Tests | 2 |
| Failures | 1 |
| Test case: | [Apps/miniFE] run miniFE on single node under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 3.351 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common-test/functions, line 414, in test file rm_execution_single_host, line 46) `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed job script = /tmp/job.ohpc-test.5903 Batch job 16 submitted Job 16 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./src/miniFE.x.gnu14.openmpi5 nx=100 ny=100 nz=100 verify_solution=1 (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148494.940802] [c1:7325 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x235d010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148494.941818] [c1:7327 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1286010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07325] pml_ucx.c:313 Error: Failed to create UCP worker [c1:07327] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148494.944160] [c1:7324 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1b64010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148494.944445] [c1:7329 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x11b8010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07324] pml_ucx.c:313 Error: Failed to create UCP worker [c1:07329] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148494.949195] [c1:7328 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x16cb010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148494.949930] [c1:7330 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xe19010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07328] pml_ucx.c:313 Error: Failed to create UCP worker [c1:07330] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148494.955672] [c1:7326 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x260e010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07326] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729148494.991460] [c1:7331 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2658050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:07331] pml_ucx.c:313 Error: Failed to create UCP worker [c1:7329 :0:7329] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xf5bb10) ==== backtrace (tid: 7329) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f825c568c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f825c568dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f825c5690c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f825c854db0] 4 [0xf5bb10] ================================= [c1:7331 :0:7331] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2355a80) ==== backtrace (tid: 7331) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fe0b1b68c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fe0b1b68dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fe0b1b690c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fe0b1e54db0] 4 [0x2355a80] ================================= [c1:7325 :0:7325] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1efd030) [c1:7330 :0:7330] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xb05f60) ==== backtrace (tid: 7325) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7efd783cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7efd783cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7efd783cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7efd78854db0] 4 [0x1efd030] ================================= [c1:7324 :0:7324] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1b7eb30) ==== backtrace (tid: 7330) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0989ad8c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0989ad8dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f0989ad90c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0989e54db0] 4 [0xb05f60] ================================= [c1:7326 :0:7326] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x22faf60) [c1:7328 :0:7328] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x146ed10) [c1:7327 :0:7327] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1029b10) ==== backtrace (tid: 7324) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f24d0cfac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f24d0cfadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f24d0cfb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f24d1054db0] 4 [0x1b7eb30] ================================= ==== backtrace (tid: 7326) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f08b6efac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f08b6efadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f08b6efb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f08b7254db0] 4 [0x22faf60] ================================= ==== backtrace (tid: 7328) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7faafac3dc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7faafac3ddff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7faafac3e0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7faafaa54db0] 4 [0x146ed10] ================================= ==== backtrace (tid: 7327) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0ba2dacc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0ba2dacdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f0ba2dad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0ba3254db0] 4 [0x1029b10] ================================= -------------------------------------------------------------------------- prterun noticed that process rank 7 with PID 7331 on node c1 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
| Duration | 4.372 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Duration | 0.141 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.966 sec |
|---|---|
| Tests | 6 |
| Failures | 0 |
| Test case: | [Compilers] C binary runs under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.152 sec |
| Failed | None |
None
| Test case: | [Compilers] C++ binary runs under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.161 sec |
| Failed | None |
None
| Test case: | [Compilers] Fortran binary runs under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.165 sec |
| Failed | None |
None
| Test case: | [Compilers] C openmp binary runs under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.159 sec |
| Failed | None |
None
| Duration | 0.412 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 5.03 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [/dev-tools/autotools] running autoreconf |
|---|---|
| Outcome: | Passed |
| Duration: | 3.184 sec |
| Failed | None |
None
| Test case: | [/dev-tools/autotools] run generated configure |
|---|---|
| Outcome: | Passed |
| Duration: | 1.654 sec |
| Failed | None |
None
| Duration | 1.978 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Duration | 0.284 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [HWLOC] Verify HWLOC module is loaded and matches rpm version (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.163 sec |
| Failed | None |
None
| Test case: | [HWLOC] Verify module HWLOC_DIR is defined and exists (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [HWLOC] Verify module HWLOC_LIB is defined and exists (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Test case: | [HWLOC] Verify dynamic library available in HWLOC_LIB (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [HWLOC] Verify static library is not present in HWLOC_LIB (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 0.059 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 6.547 sec |
|---|---|
| Tests | 1 |
| Failures | 1 |
| Test case: | [dev-tools/py3-mpi4py] python hello world (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 6.547 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414,
in test file hipy, line 35)
`run_mpi_binary "${_python} helloworld.py" $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.27534
Batch job 27 submitted
Job 27 failed...
Reason=NonZeroExitCode
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- python3 helloworld.py 8 (family=openmpi5)
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148539.188319] [c2:6997 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x55eee2009050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06997] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148539.205549] [c2:6995 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x562053135010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06995] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148539.225360] [c2:6996 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x559de04cd010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06996] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148539.235651] [c2:6994 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x55d7cd062050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06994] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148539.269468] [c1:7686 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x55c812c90010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07686] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148539.279982] [c1:7687 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x55f1ce1a6050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07687] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148539.283822] [c1:7685 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x563f1fe42050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148539.284868] [c1:7684 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x55bf1ecf5fd0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07685] pml_ucx.c:313 Error: Failed to create UCP worker
[c1:07684] pml_ucx.c:313 Error: Failed to create UCP worker
[c1:7685 :0:7685] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x563f1fdd5de0)
[c1:7687 :0:7687] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x55f1ce139c30)
==== backtrace (tid: 7685) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f3056ddbc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f3056ddbdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f3056ddc0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f3057854db0]
4 [0x563f1fdd5de0]
=================================
==== backtrace (tid: 7687) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f2f93f2ec44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f2f93f2edff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f2f93f2f0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f2f94a54db0]
4 [0x55f1ce139c30]
=================================
[c1:7684 :0:7684] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x55bf1e83a110)
==== backtrace (tid: 7684) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f5f1c06dc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f5f1c06ddff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f5f1c06e0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f5f1cc54db0]
4 [0x55bf1e83a110]
=================================
[c2:6996 :0:6996] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x559de0277d50)
==== backtrace (tid: 6996) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fb5f5f4bc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fb5f5f4bdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fb5f5f4c0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7fb5f6a54db0]
4 [0x559de0277d50]
=================================
[c2:6994 :0:6994] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x55d7cce0ac10)
[c2:6995 :0:6995] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x562052edaeb0)
==== backtrace (tid: 6995) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0e3414bc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0e3414bdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f0e3414c0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f0e34c54db0]
4 [0x562052edaeb0]
=================================
[c2:6997 :0:6997] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x55eee1db49f0)
==== backtrace (tid: 6994) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f5bf365ac44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f5bf365adff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f5bf365b0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f5bf3454db0]
4 [0x55d7cce0ac10]
=================================
==== backtrace (tid: 6997) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f492c1dbc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f492c1dbdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f492c1dc0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f492cc54db0]
4 [0x55eee1db49f0]
=================================
[c1:7686 :0:7686] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x55c81297ae10)
==== backtrace (tid: 7686) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ff0335dbc44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ff0335dbdff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ff0335dc0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7ff034054db0]
4 [0x55c81297ae10]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 6 with PID 6996 on node c2 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------
| Duration | 0.06 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.081 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [Numpy] Verify NUMPY modules can be loaded and match rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.018 sec |
| Failed | None |
None
| Test case: | [Numpy] Verify module NUMPY_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Duration | 0.06 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.063 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 3.458 sec |
|---|---|
| Tests | 8 |
| Failures | 0 |
| Test case: | [Valgrind] Verify valgrind module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.163 sec |
| Failed | None |
None
| Test case: | [Valgrind] Verify module VALGRIND_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [Valgrind] Verify availability of valgrind binary (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.033 sec |
| Failed | None |
None
| Test case: | [Valgrind] Verify availability of man page (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.054 sec |
| Failed | None |
None
| Test case: | [Valgrind] Verify module VALGRIND_INC is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [Valgrind] Verify header file is present in VALGRIND_INC (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Duration | 12.55 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [R] Verify R module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.219 sec |
| Failed | None |
None
| Test case: | [R] Verify R_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [R] Verify availability of R executable -> R (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.031 sec |
| Failed | None |
None
| Test case: | [R] Verify availability of R executable -> Rscript (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.032 sec |
| Failed | None |
None
| Duration | 0.33 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/ADIOS2] Verify ADIOS2 module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.195 sec |
| Failed | None |
None
| Test case: | [libs/ADIOS2] Verify module ADIOS2_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.024 sec |
| Failed | None |
None
| Test case: | [libs/ADIOS2] Verify module ADIOS2_BIN is defined and exists |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/ADIOS2] Verify module ADIOS2_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/ADIOS2] Verify (dynamic) library available in ADIOS2_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.025 sec |
| Failed | None |
None
| Duration | 11.025 sec |
|---|---|
| Tests | 2 |
| Failures | 2 |
| Test case: | [libs/ADIOS2] MPI C binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 5.511 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414,
in test file rm_execution, line 24)
`run_mpi_binary ./${binary} "" $NODES $TASKS ' failed
job script = /tmp/job.ohpc-test.22878
Batch job 33 submitted
Job 33 failed...
Reason=NonZeroExitCode
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./arrays_write (family=openmpi5)
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148599.275003] [c2:7139 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1a2e010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148599.277949] [c2:7141 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x199c010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07139] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148599.280243] [c2:7142 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x24f1010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148599.280565] [c2:7140 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x18a2010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07141] pml_ucx.c:313 Error: Failed to create UCP worker
[c2:07140] pml_ucx.c:313 Error: Failed to create UCP worker
[c2:07142] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148599.397496] [c1:7878 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x25c7010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07878] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148599.422480] [c1:7879 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x22f9010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07879] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148599.430420] [c1:7877 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1e17010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07877] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148599.438523] [c1:7876 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1eae010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07876] pml_ucx.c:313 Error: Failed to create UCP worker
[c1:7876 :0:7876] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1a69eb0)
[c1:7877 :0:7877] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1bb6f40)
==== backtrace (tid: 7876) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f1029c20c44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f1029c20dff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f1029c210c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f102a854db0]
4 [0x1a69eb0]
=================================
==== backtrace (tid: 7877) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f4854e8ec44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f4854e8edff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f4854e8f0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f4855a54db0]
4 [0x1bb6f40]
=================================
[c1:7878 :0:7878] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2366f40)
==== backtrace (tid: 7878) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ff3829e1c44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ff3829e1dff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ff3829e20c6]
3 /lib64/libc.so.6(+0x54db0) [0x7ff383654db0]
4 [0x2366f40]
=================================
[c2:7141 :0:7141] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x169cd90)
[c1:7879 :0:7879] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2092f40)
[c2:7139 :0:7139] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x172ed90)
[c2:7140 :0:7140] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x159fa00)
[c2:7142 :0:7142] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x21ee970)
==== backtrace (tid: 7141) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f6f9668ec44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f6f9668edff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f6f9668f0c6]
==== backtrace (tid: 7139) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fa6fac3ac44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fa6fac3adff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fa6fac3b0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7fa6faa54db0]
4 [0x172ed90]
=================================
==== backtrace (tid: 7140) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f09d1df1c44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f09d1df1dff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f09d1df20c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f09d2a54db0]
4 [0x159fa00]
=================================
3 /lib64/libc.so.6(+0x54db0) [0x7f6f97254db0]
4 [0x169cd90]
=================================
==== backtrace (tid: 7142) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f765cb3ec44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f765cb3edff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f765cb3f0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7f765d854db0]
4 [0x21ee970]
=================================
==== backtrace (tid: 7879) ====
0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ff7ba88ec44]
1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ff7ba88edff]
2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ff7ba88f0c6]
3 /lib64/libc.so.6(+0x54db0) [0x7ff7bb454db0]
4 [0x2092f40]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 7876 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------
| Test case: | [libs/ADIOS2] MPI F90 binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 5.514 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414,
in test file rm_execution, line 34)
`run_mpi_binary ./${binary} "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.26592
Batch job 34 submitted
Job 34 failed...
Reason=NonZeroExitCode
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./scalars_write (family=openmpi5)
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148605.180878] [c2:7214 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x289f010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07214] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148605.186080] [c2:7211 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x195b050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148605.188030] [c2:7213 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x12c4050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07213] pml_ucx.c:313 Error: Failed to create UCP worker
[c2:07211] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148605.230934] [c2:7212 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x16db050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07212] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148605.265050] [c1:7944 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1509010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07944] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148605.278312] [c1:7945 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x24e1050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07945] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148605.309157] [c1:7943 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xf93050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07943] pml_ucx.c:313 Error: Failed to create UCP worker
[create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22
[1729148605.313508] [c1:7942 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x25f1050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07942] pml_ucx.c:313 Error: Failed to create UCP worker
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x7f707b893cda in ???
#1 0x7f707b892d75 in ???
#2 0x7f707ac54daf in ???
#3 0x2584c70 in ???
#0 0x7f7a6af18cda in ???
#1 0x7f7a6af17d75 in ???
#2 0x7f7a6a254daf in ???
#3 0xf24f50 in ???
#0 0x7f5dee4bfcda in ???
#1 0x7f5dee4bed75 in ???
#2 0x7f5ded854daf in ???
#0 0x7fc77ece2cda in ???
#1 0x7fc77ece1d75 in ???
#2 0x7fc77e054daf in ???
#3 0x2476ef0 in ???
#3 0x12a4b40 in ???
#0 0x7fceec68dcda in ???
#1 0x7fceec68cd75 in ???
#2 0x7fceeba54daf in ???
#3 0x2568ba0 in ???
#0 0x7efee0da0cda in ???
#1 0x7efee0d9fd75 in ???
#2 0x7efee0054daf in ???
#3 0x150bf30 in ???
#0 0x7fa5eed79cda in ???
#1 0x7fa5eed78d75 in ???
#2 0x7fa5ee054daf in ???
#3 0xfd2100 in ???
#0 0x7f1b8f027cda in ???
#1 0x7f1b8f026d75 in ???
#2 0x7f1b8e254daf in ???
#3 0x1668740 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 7942 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------
| Duration | 0.34 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/ADIOS2] Verify ADIOS2 module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.205 sec |
| Failed | None |
None
| Test case: | [libs/ADIOS2] Verify module ADIOS2_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/ADIOS2] Verify module ADIOS2_BIN is defined and exists |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/ADIOS2] Verify module ADIOS2_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/ADIOS2] Verify (dynamic) library available in ADIOS2_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.026 sec |
| Failed | None |
None
| Duration | 0.683 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [Boost/Accumulators] min-test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.178 sec |
| Failed | None |
None
| Test case: | [Boost/Accumulators] max-test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.169 sec |
| Failed | None |
None
| Duration | 0.684 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [Boost/Accumulators] min-test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.171 sec |
| Failed | None |
None
| Test case: | [Boost/Accumulators] max-test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.176 sec |
| Failed | None |
None
| Duration | 3.276 sec |
|---|---|
| Tests | 8 |
| Failures | 0 |
| Test case: | [BOOST] Verify BOOST module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.198 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify dynamic library available in BOOST_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify static library is not present in BOOST_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_INC is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 3.092 sec |
|---|---|
| Tests | 8 |
| Failures | 0 |
| Test case: | [BOOST] Verify BOOST module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.208 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify dynamic library available in BOOST_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify static library is not present in BOOST_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_INC is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 0.645 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [Boost/Multi Array] access-test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.157 sec |
| Failed | None |
None
| Test case: | [Boost/Multi Array] iterators-test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.161 sec |
| Failed | None |
None
| Duration | 0.682 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [Boost/Multi Array] access-test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.171 sec |
| Failed | None |
None
| Test case: | [Boost/Multi Array] iterators-test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.175 sec |
| Failed | None |
None
| Duration | 1.604 sec |
|---|---|
| Tests | 11 |
| Failures | 0 |
| Test case: | [Boost/Program Options] cmdline_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.19 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] exception_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.169 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] options_description_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.169 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] parsers_test on master host(gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.029 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] parsers_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.157 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] positional_options_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.156 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] required_test on master host(gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.028 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] required_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.182 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] unicode_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.176 sec |
| Failed | None |
None
| Duration | 1.6 sec |
|---|---|
| Tests | 11 |
| Failures | 0 |
| Test case: | [Boost/Program Options] cmdline_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.182 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] exception_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.177 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] options_description_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.176 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] parsers_test on master host(gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.03 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] parsers_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.176 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] positional_options_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.173 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] required_test on master host(gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.029 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] required_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.174 sec |
| Failed | None |
None
| Test case: | [Boost/Program Options] unicode_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.16 sec |
| Failed | None |
None
| Duration | 1.928 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [Boost/Random] test_piecewise_linear under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.173 sec |
| Failed | None |
None
| Test case: | [Boost/Random] test_discrete under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.409 sec |
| Failed | None |
None
| Duration | 1.967 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [Boost/Random] test_piecewise_linear under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.178 sec |
| Failed | None |
None
| Test case: | [Boost/Random] test_discrete under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.431 sec |
| Failed | None |
None
| Duration | 0.993 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.987 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.327 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [BOOST] Verify BOOST module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.205 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify dynamic library available in BOOST_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify static library is not present in BOOST_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 0.327 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [BOOST] Verify BOOST module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.202 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify module BOOST_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify dynamic library available in BOOST_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [BOOST] Verify static library is not present in BOOST_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 15.725 sec |
|---|---|
| Tests | 6 |
| Failures | 0 |
| Test case: | [Boost/MPI] all_gather_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.269 sec |
| Failed | None |
None
| Test case: | [Boost/MPI] all_reduce_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.326 sec |
| Failed | None |
None
| Test case: | [Boost/MPI] all_to_all_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.262 sec |
| Failed | None |
None
| Test case: | [Boost/MPI] groups_test under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.267 sec |
| Failed | None |
None
| Duration | 31.879 sec |
|---|---|
| Tests | 6 |
| Failures | 6 |
| Test case: | [Boost/MPI] all_gather_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 7.634 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414, in test file rm_execution, line 21) `run_mpi_binary ./$test atest 2 16' failed job script = /tmp/job.ohpc-test.16163 Batch job 93 submitted Job 93 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./all_gather_test atest (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.095896] [c2:7538 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x11f0050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.096960] [c2:7544 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x260f570 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07544] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.098892] [c2:7539 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x11c6050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07538] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07539] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.101206] [c2:7543 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1713050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.101978] [c2:7542 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x28b6570 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07543] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07542] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.114445] [c2:7541 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2b1d0a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07541] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.144966] [c2:7545 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x210e050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [1729149058.144959] [c2:7540 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xef2050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07545] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07540] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.250492] [c1:9318 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x204e010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.253039] [c1:9312 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1037010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09312] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09318] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.256455] [c1:9311 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x10de010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09311] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.258247] [c1:9313 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2814010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.259668] [c1:9316 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x164c010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09313] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09316] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.264882] [c1:9317 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xd19010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.265134] [c1:9314 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xd63010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09317] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09314] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149058.281334] [c1:9315 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1fec050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09315] pml_ucx.c:313 Error: Failed to create UCP worker [c1:9315 :0:9315] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1d9def0) ==== backtrace (tid: 9315) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fd34eb68c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fd34eb68dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fd34eb690c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fd34ee54db0] 4 [0x1d9def0] ================================= [c1:9313 :0:9313] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x25adf40) [c1:9316 :0:9316] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x13e5f20) ==== backtrace (tid: 9316) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fdc0ff82c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fdc0ff82dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fdc0ff830c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fdc10454db0] 4 [0x13e5f20] ================================= ==== backtrace (tid: 9313) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7eff2e03cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7eff2e03cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7eff2e03d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7eff2e454db0] 4 [0x25adf40] ================================= [c1:9314 :0:9314] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xafcef0) [c1:9317 :0:9317] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xab8ef0) [c1:9312 :0:9312] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xdd0f40) ==== backtrace (tid: 9314) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f5be8a3cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f5be8a3cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f5be8a3d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f5be8e54db0] 4 [0xafcef0] ================================= ==== backtrace (tid: 9317) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fd06b4fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fd06b4fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fd06b4fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fd06b854db0] 4 [0xab8ef0] ================================= [c1:9311 :0:9311] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xe77f40) ==== backtrace (tid: 9312) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f58ba8fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f58ba8fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f58ba8fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f58bac54db0] 4 [0xdd0f40] ================================= ==== backtrace (tid: 9311) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fba0c8fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fba0c8fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fba0c8fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fba0cc54db0] 4 [0xe77f40] ================================= [c2:7539 :0:7539] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xfd1950) [c2:7538 :0:7538] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xf9b9f0) [c2:7545 :0:7545] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1eb9c10) [c2:7541 :0:7541] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x26abb40) [c2:7543 :0:7543] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x14be780) [c2:7540 :0:7540] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc9d9f0) [c2:7544 :0:7544] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2333b40) ==== backtrace (tid: 7539) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fd99103cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fd99103cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fd99103d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fd991454db0] 4 [0xfd1950] ================================= [c2:7542 :0:7542] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x25dab40) ==== backtrace (tid: 7538) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f7fe26d8c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f7fe26d8dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f7fe26d90c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f7fe2a54db0] 4 [0xf9b9f0] ================================= ==== backtrace (tid: 7545) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0d0ff68c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0d0ff68dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f0d0ff690c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0d10254db0] 4 [0x1eb9c10] ================================= ==== backtrace (tid: 7541) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f44dfccbc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f44dfccbdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f44dfccc0c6] ==== backtrace (tid: 7543) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f5d2f4fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f5d2f4fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f5d2f4fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f5d2f854db0] 3 /lib64/libc.so.6(+0x54db0) [0x7f44e0054db0] 4 [0x26abb40] ================================= 4 [0x14be780] ================================= ==== backtrace (tid: 7540) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fae1aac9c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fae1aac9dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fae1aaca0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fae1ae54db0] 4 [0xc9d9f0] ================================= ==== backtrace (tid: 7544) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f8471e3cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f8471e3cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f8471e3d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f8472254db0] 4 [0x2333b40] ================================= ==== backtrace (tid: 7542) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f310efacc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f310efacdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f310efad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f310f454db0] 4 [0x25dab40] ================================= [c1:9318 :0:9318] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1de7f40) ==== backtrace (tid: 9318) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fce22c3cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fce22c3cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fce22c3d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fce23054db0] 4 [0x1de7f40] ================================= -------------------------------------------------------------------------- prterun noticed that process rank 2 with PID 9313 on node c1 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
| Test case: | [Boost/MPI] all_reduce_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 4.422 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414, in test file rm_execution, line 30) `run_mpi_binary ./$test atest 2 16' failed job script = /tmp/job.ohpc-test.2944 Batch job 94 submitted Job 94 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./all_reduce_test atest (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.074956] [c2:7658 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x17bc050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [1729149065.075183] [c2:7659 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xb3c050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07659] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [c2:07658] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.078519] [c2:7663 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x24fe050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [1729149065.078728] [c2:7662 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xd91570 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07663] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07662] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.095673] [c2:7664 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1fef570 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07664] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.102268] [c2:7660 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x24ae0a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07660] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.114965] [c2:7661 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1b25050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07661] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.120238] [c2:7657 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xe0b010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07657] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.165619] [c1:9415 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2588010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09415] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.172747] [c1:9411 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xc85010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09411] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.180260] [c1:9410 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xf7b010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09410] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.187126] [c1:9417 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2274010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09417] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.191435] [c1:9414 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x265b050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09414] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.201404] [c1:9413 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1b82010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09413] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.211622] [c1:9412 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x23af010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09412] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149065.231276] [c1:9416 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x17df010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09416] pml_ucx.c:313 Error: Failed to create UCP worker [c1:9416 :0:9416] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x157eef0) ==== backtrace (tid: 9416) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fbf0befac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fbf0befadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fbf0befb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fbf0c254db0] 4 [0x157eef0] ================================= [c1:9411 :0:9411] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xa1ef40) [c1:9412 :0:9412] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2148f40) [c1:9413 :0:9413] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x191bef0) [c1:9414 :0:9414] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x25ecf50) [c1:9410 :0:9410] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xd14f40) ==== backtrace (tid: 9411) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f71bb3cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f71bb3cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f71bb3cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f71bb854db0] 4 [0xa1ef40] ================================= ==== backtrace (tid: 9412) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f6a9e03cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f6a9e03cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f6a9e03d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f6a9e454db0] 4 [0x2148f40] ================================= ==== backtrace (tid: 9413) ==== [c1:9417 :0:9417] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2013a70) 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fb6da2fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fb6da2fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fb6da2fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fb6da654db0] 4 [0x191bef0] ================================= ==== backtrace (tid: 9410) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fe17a43cc44] ==== backtrace (tid: 9414) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fa22f168c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fa22f168dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fa22f1690c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fa22f454db0] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fe17a43cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fe17a43d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fe17a854db0] 4 [0xd14f40] ================================= 4 [0x25ecf50] ================================= ==== backtrace (tid: 9417) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f4d4983bc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f4d4983bdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f4d4983c0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f4d49654db0] 4 [0x2013a70] ================================= [c2:7662 :0:7662] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xab5b40) [c2:7661 :0:7661] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18d09f0) [c2:7663 :0:7663] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x22a99f0) [c2:7660 :0:7660] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x203cb40) [c2:7658 :0:7658] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1567bd0) [c2:7659 :0:7659] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x947950) ==== backtrace (tid: 7662) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f99efd82c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f99efd82dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f99efd830c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f99f0254db0] 4 [0xab5b40] ================================= ==== backtrace (tid: 7663) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f3629368c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f3629368dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f36293690c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f3629654db0] ==== backtrace (tid: 7661) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fdbfeefac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fdbfeefadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fdbfeefb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fdbff254db0] 4 [0x18d09f0] ================================= 4 [0x22a99f0] ================================= ==== backtrace (tid: 7660) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f4f7cfacc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f4f7cfacdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f4f7cfad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f4f7d454db0] 4 [0x203cb40] ================================= ==== backtrace (tid: 7658) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fb9f2ccbc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fb9f2ccbdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fb9f2ccc0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fb9f3054db0] 4 [0x1567bd0] ================================= ==== backtrace (tid: 7659) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fc347368c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fc347368dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fc3473690c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fc347654db0] 4 [0x947950] ================================= [c2:7664 :0:7664] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1d13b40) [c2:7657 :0:7657] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xbb8f20) ==== backtrace (tid: 7657) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fcabb240c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fcabb240dff] ==== backtrace (tid: 7664) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f83b42d8c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f83b42d8dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f83b42d90c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f83b4654db0] 4 [0x1d13b40] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fcabb2410c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fcabb054db0] 4 [0xbb8f20] ================================= ================================= [c1:9415 :0:9415] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2327ef0) ==== backtrace (tid: 9415) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7faec043cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7faec043cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7faec043d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7faec0854db0] 4 [0x2327ef0] ================================= -------------------------------------------------------------------------- prterun noticed that process rank 15 with PID 7661 on node c2 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
| Test case: | [Boost/MPI] all_to_all_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 4.419 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414, in test file rm_execution, line 39) `run_mpi_binary ./$test atest 2 16' failed job script = /tmp/job.ohpc-test.30190 Batch job 95 submitted Job 95 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./all_to_all_test atest (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.710925] [c2:7762 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1f22050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.715502] [c2:7759 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x13da050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07762] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07759] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.727060] [c2:7766 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x148f050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07766] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.765438] [c2:7760 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1c0d050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07760] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.767159] [c2:7761 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x16dc050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [1729149069.767291] [c2:7764 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xc0e050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07761] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.770599] [c2:7763 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x12ad050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07763] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07764] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.790660] [c2:7765 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xefa050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07765] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.814504] [c1:9506 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1c88010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.818240] [c1:9507 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x256b010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09506] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09507] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.825818] [c1:9505 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x288e920 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09505] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.841701] [c1:9509 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1c49050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [c1:09509] pml_ucx.c:313 Error: Failed to create UCP worker [1729149069.843322] [c1:9511 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xf14010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09511] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.856446] [c1:9508 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x229e050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09508] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.862097] [c1:9512 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xba5010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149069.862514] [c1:9510 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x180b010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09512] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09510] pml_ucx.c:313 Error: Failed to create UCP worker [c1:9510 :0:9510] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x15aaa70) [c1:9512 :0:9512] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x944a70) ==== backtrace (tid: 9510) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f4fe1968c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f4fe1968dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f4fe19690c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f4fe1c54db0] 4 [0x15aaa70] ================================= [c1:9508 :0:9508] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x222df10) ==== backtrace (tid: 9512) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f68bc7cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f68bc7cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f68bc7cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f68bcc54db0] 4 [0x944a70] ================================= ==== backtrace (tid: 9508) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f27eeefac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f27eeefadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f27eeefb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f27ef254db0] 4 [0x222df10] ================================= [c1:9509 :0:9509] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1bdaf50) [c1:9511 :0:9511] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xcb3a70) ==== backtrace (tid: 9509) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f394963cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f394963cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f394963d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f3949454db0] 4 [0x1bdaf50] ================================= [c1:9505 :0:9505] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2538680) ==== backtrace (tid: 9511) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f286cefac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f286cefadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f286cefb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f286d254db0] 4 [0xcb3a70] ================================= [c1:9507 :0:9507] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2304f40) ==== backtrace (tid: 9505) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f58f0bcec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f58f0bcedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f58f0bcf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f58f1054db0] 4 [0x2538680] ================================= ==== backtrace (tid: 9507) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f47da9acc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f47da9acdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f47da9ad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f47dae54db0] 4 [0x2304f40] ================================= [c2:7764 :0:7764] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x91bad0) [c2:7766 :0:7766] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x119cbc0) [c2:7762 :0:7762] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1bd3ee0) [c2:7760 :0:7760] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x191abc0) [c2:7759 :0:7759] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x10e7ad0) [c2:7763 :0:7763] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xfba740) [c2:7765 :0:7765] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc08640) [c2:7761 :0:7761] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x13e9ad0) ==== backtrace (tid: 7764) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f62a35cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f62a35cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f62a35cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f62a3a54db0] 4 [0x91bad0] ================================= ==== backtrace (tid: 7762) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fc93843cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fc93843cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fc93843d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fc938854db0] 4 [0x1bd3ee0] ================================= ==== backtrace (tid: 7766) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fb6e79cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fb6e79cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fb6e79cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fb6e7e54db0] 4 [0x119cbc0] ================================= ==== backtrace (tid: 7760) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0436382c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0436382dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f04363830c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0436854db0] 4 [0x191abc0] ================================= ==== backtrace (tid: 7759) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fcd35fcec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fcd35fcedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fcd35fcf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fcd36454db0] 4 [0x10e7ad0] ================================= ==== backtrace (tid: 7763) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fc5551cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fc5551cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fc5551cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fc555654db0] 4 [0xfba740] ================================= ==== backtrace (tid: 7765) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fb28d9cec44] ==== backtrace (tid: 7761) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ff613968c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ff613968dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ff6139690c6] 3 /lib64/libc.so.6(+0x54db0) [0x7ff613c54db0] 4 [0x13e9ad0] ================================= 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fb28d9cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fb28d9cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fb28de54db0] 4 [0xc08640] ================================= [c1:9506 :0:9506] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1a21f40) ==== backtrace (tid: 9506) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f47812cbc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f47812cbdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f47812cc0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f4781654db0] 4 [0x1a21f40] ================================= -------------------------------------------------------------------------- prterun noticed that process rank 5 with PID 9510 on node c1 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
| Test case: | [Boost/MPI] groups_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 4.426 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414, in test file rm_execution, line 48) `run_mpi_binary ./$test atest 2 16' failed job script = /tmp/job.ohpc-test.14346 Batch job 96 submitted Job 96 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./groups_test atest (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.057977] [c2:7867 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1b27050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07867] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.070493] [c2:7869 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2513010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.071811] [c2:7864 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x28b7050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07869] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07864] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.079929] [c2:7866 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1afb010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.080811] [c2:7865 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x23ed050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07866] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07865] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.082717] [c2:7862 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x20ee050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07862] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.093824] [c2:7868 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x280a010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07868] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.106914] [c2:7863 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2633050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07863] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.171390] [c1:9604 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x26cc010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09604] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.179256] [c1:9602 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x11ce010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.180535] [c1:9606 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x208a010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09602] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09606] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.192822] [c1:9601 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x9ec010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09601] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.194509] [c1:9599 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xe68010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.195003] [c1:9600 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1ab9010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09600] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09599] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.198736] [c1:9605 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1985010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09605] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149074.206277] [c1:9603 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2111050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09603] pml_ucx.c:313 Error: Failed to create UCP worker [c1:9605 :0:9605] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1724ef0) [c1:9603 :0:9603] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1ec2ee0) ==== backtrace (tid: 9605) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f18752fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f18752fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f18752fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f1875654db0] 4 [0x1724ef0] ================================= ==== backtrace (tid: 9603) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f4be1efac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f4be1efadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f4be1efb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f4be2254db0] 4 [0x1ec2ee0] ================================= [c1:9600 :0:9600] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1852f40) [c1:9599 :0:9599] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc07f40) ==== backtrace (tid: 9600) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f33812fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f33812fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f33812fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f3381654db0] 4 [0x1852f40] ================================= ==== backtrace (tid: 9599) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f031b5acc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f031b5acdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f031b5ad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f031ba54db0] 4 [0xc07f40] ================================= [c1:9601 :0:9601] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x785f40) ==== backtrace (tid: 9601) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fcf891acc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fcf891acdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fcf891ad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fcf89654db0] 4 [0x785f40] ================================= [c1:9604 :0:9604] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x246bef0) ==== backtrace (tid: 9604) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fbf43afac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fbf43afadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fbf43afb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fbf43e54db0] 4 [0x246bef0] ================================= [c1:9602 :0:9602] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xf73f60) ==== backtrace (tid: 9602) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0cb3ed8c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0cb3ed8dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f0cb3ed90c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0cb4254db0] 4 [0xf73f60] ================================= [c2:7864 :0:7864] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x265fc70) [c2:7862 :0:7862] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1e999f0) [c2:7867 :0:7867] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1744de0) ==== backtrace (tid: 7864) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f02494d8c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f02494d8dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f02494d90c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0249854db0] 4 [0x265fc70] ================================= ==== backtrace (tid: 7867) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ff9dcdcec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ff9dcdcedff] ==== backtrace (tid: 7862) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f052903cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f052903cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f052903d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0529454db0] 4 [0x1e999f0] ================================= 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ff9dcdcf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7ff9dd254db0] 4 [0x1744de0] ================================= [c2:7868 :0:7868] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2429e00) [c2:7865 :0:7865] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20a8c80) [c2:7863 :0:7863] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x22eec80) [c2:7866 :0:7866] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x17fba70) [c2:7869 :0:7869] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x22c0f00) ==== backtrace (tid: 7868) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f790b1acc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f790b1acdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f790b1ad0c6] ==== backtrace (tid: 7865) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ff7041cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ff7041cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ff7041cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7ff704654db0] 4 [0x20a8c80] ================================= ==== backtrace (tid: 7863) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ff6ee18fc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ff6ee18fdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ff6ee1900c6] 3 /lib64/libc.so.6(+0x54db0) [0x7ff6ee654db0] 4 [0x22eec80] ================================= 3 /lib64/libc.so.6(+0x54db0) [0x7f790b654db0] 4 [0x2429e00] ================================= ==== backtrace (tid: 7866) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f8b237cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f8b237cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f8b237cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f8b23c54db0] 4 [0x17fba70] ================================= ==== backtrace (tid: 7869) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f3c8ced8c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f3c8ced8dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f3c8ced90c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f3c8d254db0] 4 [0x22c0f00] ================================= [c1:9606 :0:9606] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1e29a70) ==== backtrace (tid: 9606) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fb0b42d8c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fb0b42d8dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fb0b42d90c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fb0b4654db0] 4 [0x1e29a70] ================================= -------------------------------------------------------------------------- prterun noticed that process rank 8 with PID 7862 on node c2 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
| Test case: | [Boost/MPI] ring_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 5.493 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414, in test file rm_execution, line 66) `run_mpi_binary ./$test atest 2 16' failed job script = /tmp/job.ohpc-test.1991 Batch job 97 submitted Job 97 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./ring_test atest (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.068994] [c2:7961 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x14f8050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.069987] [c2:7966 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1268050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [1729149079.070142] [c2:7964 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x14c3050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07966] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [c2:07964] pml_ucx.c:313 Error: Failed to create UCP worker [1729149079.071676] [c2:7965 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x12c4010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.072316] [c2:7967 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1d6d050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07961] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07965] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07967] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.086060] [c2:7963 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xeb9010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.086957] [c2:7962 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1cdc010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07963] pml_ucx.c:313 Error: Failed to create UCP worker [c2:07962] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.128589] [c2:7960 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x233d050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:07960] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.186440] [c1:9694 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xd64010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09694] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.188685] [c1:9693 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xdf9010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09693] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.190914] [c1:9697 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x167b010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.191429] [c1:9698 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2882010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.191767] [c1:9692 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1df2010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09697] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09692] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09698] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.203481] [c1:9691 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1ce1010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09691] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.210560] [c1:9696 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2043010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09696] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149079.230352] [c1:9695 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1e44050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09695] pml_ucx.c:313 Error: Failed to create UCP worker [c1:9695 :0:9695] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1dd5f50) ==== backtrace (tid: 9695) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f254d0fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f254d0fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f254d0fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f254d454db0] 4 [0x1dd5f50] ================================= [c1:9691 :0:9691] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1a80a70) [c1:9696 :0:9696] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1de2ef0) [c1:9693 :0:9693] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xb92f40) [c1:9694 :0:9694] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xaffc40) [c1:9697 :0:9697] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x141aef0) ==== backtrace (tid: 9691) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f4bc058fc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f4bc058fdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f4bc05900c6] ==== backtrace (tid: 9696) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f03ac99fc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f03ac99fdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f03ac9a00c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f03ace54db0] 4 [0x1de2ef0] ================================= 3 /lib64/libc.so.6(+0x54db0) [0x7f4bc0a54db0] 4 [0x1a80a70] ================================= [c1:9692 :0:9692] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1aceec0) ==== backtrace (tid: 9693) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fd3fe1cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fd3fe1cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fd3fe1cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fd3fe654db0] 4 [0xb92f40] ================================= ==== backtrace (tid: 9694) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fbe6eafac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fbe6eafadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fbe6eafb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fbe6ee54db0] 4 [0xaffc40] ================================= ==== backtrace (tid: 9697) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0df90fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0df90fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f0df90fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0df9454db0] 4 [0x141aef0] ================================= ==== backtrace (tid: 9692) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f75df9cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f75df9cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f75df9cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f75dfe54db0] 4 [0x1aceec0] ================================= [c2:7967 :0:7967] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1b18cf0) [c2:7966 :0:7966] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x10139f0) [c2:7962 :0:7962] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18fbe40) ==== backtrace (tid: 7967) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f22ee9cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f22ee9cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f22ee9cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f22eee54db0] 4 [0x1b18cf0] ================================= ==== backtrace (tid: 7966) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fea4fd9fc44] ==== backtrace (tid: 7962) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f12eb9cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f12eb9cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f12eb9cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f12ebe54db0] 4 [0x18fbe40] ================================= [c2:7961 :0:7961] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x12a0f60) 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fea4fd9fdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fea4fda00c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fea50254db0] 4 [0x10139f0] ================================= [c2:7963 :0:7963] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xad8e40) [c2:7964 :0:7964] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x126e9f0) [c2:7960 :0:7960] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20e8780) [c2:7965 :0:7965] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xee3e40) ==== backtrace (tid: 7961) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f2e695cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f2e695cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f2e695cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f2e69a54db0] 4 [0x12a0f60] ================================= ==== backtrace (tid: 7963) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f994aafac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f994aafadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f994aafb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f994ae54db0] 4 [0xad8e40] ================================= ==== backtrace (tid: 7964) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fc023bcec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fc023bcedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fc023bcf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fc024054db0] 4 [0x126e9f0] ==== backtrace (tid: 7960) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f5fc15cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f5fc15cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f5fc15cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f5fc1a54db0] 4 [0x20e8780] ================================= ================================= ==== backtrace (tid: 7965) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f88d31cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f88d31cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f88d31cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f88d3654db0] 4 [0xee3e40] ================================= [c1:9698 :0:9698] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2621a70) ==== backtrace (tid: 9698) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f1ae5cfac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f1ae5cfadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f1ae5cfb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f1ae6054db0] 4 [0x2621a70] ================================= -------------------------------------------------------------------------- prterun noticed that process rank 13 with PID 7962 on node c2 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
| Test case: | [Boost/MPI] pointer_test under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 5.485 sec |
| Failed | None |
(from function `run_mpi_binary' in file ./common/functions, line 414, in test file rm_execution, line 75) `run_mpi_binary ./$test atest 2 16' failed job script = /tmp/job.ohpc-test.1260 Batch job 98 submitted Job 98 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./pointer_test atest (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.103416] [c2:8060 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1ec9010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [c2:08060] pml_ucx.c:313 Error: Failed to create UCP worker [1729149085.105043] [c2:8061 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x1dee050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:08061] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.111857] [c2:8057 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2483050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.112611] [c2:8062 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2238050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:08057] pml_ucx.c:313 Error: Failed to create UCP worker [c2:08062] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.115592] [c2:8063 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2944050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [c2:08063] pml_ucx.c:313 Error: Failed to create UCP worker [1729149085.116923] [c2:8064 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x2211010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:08064] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.128762] [c2:8058 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x9a7010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:08058] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.131530] [c2:8059 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xfb9050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c2:08059] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.172526] [c1:9789 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x20d1010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09789] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.176417] [c1:9784 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xea5010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09784] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.187415] [c1:9783 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xa2a010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09783] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.191002] [c1:9788 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x158d010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09788] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.199862] [c1:9787 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x128f050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09787] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.203512] [c1:9786 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xa1d010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [1729149085.203684] [c1:9790 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0xd88010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09786] pml_ucx.c:313 Error: Failed to create UCP worker [c1:09790] pml_ucx.c:313 Error: Failed to create UCP worker [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [1729149085.218032] [c1:9785 :0] ib_iface.c:1110 UCX ERROR qedr0: iface 0x23ad010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument [c1:09785] pml_ucx.c:313 Error: Failed to create UCP worker [c1:9785 :0:9785] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2146f40) ==== backtrace (tid: 9785) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fd384facc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fd384facdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fd384fad0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fd385454db0] 4 [0x2146f40] ================================= [c1:9786 :0:9786] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7bed10) ==== backtrace (tid: 9786) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fd1a7ecbc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fd1a7ecbdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fd1a7ecc0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fd1a8254db0] 4 [0x7bed10] ================================= [c1:9790 :0:9790] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xb21f40) [c1:9787 :0:9787] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1033f30) ==== backtrace (tid: 9790) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fbea323cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fbea323cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fbea323d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fbea3654db0] 4 [0xb21f40] ================================= ==== backtrace (tid: 9787) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f1a70782c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f1a70782dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f1a707830c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f1a70c54db0] 4 [0x1033f30] ================================= [c1:9783 :0:9783] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7c3f40) [c1:9784 :0:9784] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc3ef40) ==== backtrace (tid: 9783) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f5ed4a3cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f5ed4a3cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f5ed4a3d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f5ed4e54db0] 4 [0x7c3f40] ================================= [c1:9788 :0:9788] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x132ef30) ==== backtrace (tid: 9784) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fdc2783cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fdc2783cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fdc2783d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fdc27c54db0] 4 [0xc3ef40] ================================= ==== backtrace (tid: 9788) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7faeac782c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7faeac782dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7faeac7830c6] 3 /lib64/libc.so.6(+0x54db0) [0x7faeacc54db0] 4 [0x132ef30] ================================= [c2:8060 :0:8060] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1bd0740) [c2:8058 :0:8058] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x6ae740) [c2:8061 :0:8061] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1a9f730) [c2:8062 :0:8062] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1f452d0) [c2:8064 :0:8064] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1f18740) [c2:8057 :0:8057] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x222e9f0) [c2:8059 :0:8059] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc6a730) [c2:8063 :0:8063] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x26512d0) ==== backtrace (tid: 8060) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fba9c83cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fba9c83cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fba9c83d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fba9cc54db0] 4 [0x1bd0740] ================================= ==== backtrace (tid: 8061) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f9d5343ac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f9d5343adff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f9d5343b0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f9d53254db0] 4 [0x1a9f730] ================================= ==== backtrace (tid: 8058) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f45692d8c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f45692d8dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f45692d90c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f4569654db0] 4 [0x6ae740] ================================= ==== backtrace (tid: 8062) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fa3213cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fa3213cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fa3213cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fa321854db0] 4 [0x1f452d0] ================================= ==== backtrace (tid: 8064) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7ffae97cec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7ffae97cedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7ffae97cf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7ffae9c54db0] 4 [0x1f18740] ================================= ==== backtrace (tid: 8057) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f4dac6fac44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f4dac6fadff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f4dac6fb0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f4daca54db0] 4 [0x222e9f0] ================================= ==== backtrace (tid: 8059) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fad396aec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fad396aedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fad396af0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fad39a54db0] 4 [0xc6a730] ================================= ==== backtrace (tid: 8063) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f8551e3cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f8551e3cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f8551e3d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f8552254db0] 4 [0x26512d0] ================================= [c1:9789 :0:9789] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1e72f30) ==== backtrace (tid: 9789) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fd36303bc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fd36303bdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fd36303c0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fd362e54db0] 4 [0x1e72f30] ================================= -------------------------------------------------------------------------- prterun noticed that process rank 2 with PID 9785 on node c1 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
| Duration | 0.292 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [FFTW] Verify FFTW module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.17 sec |
| Failed | None |
None
| Test case: | [FFTW] Verify module FFTW_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [FFTW] Verify module FFTW_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [FFTW] Verify dynamic library available in FFTW_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [FFTW] Verify static library is not present in FFTW_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 0.305 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [FFTW] Verify FFTW module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.177 sec |
| Failed | None |
None
| Test case: | [FFTW] Verify module FFTW_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [FFTW] Verify module FFTW_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [FFTW] Verify dynamic library available in FFTW_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [FFTW] Verify static library is not present in FFTW_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 3.851 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 7.067 sec |
|---|---|
| Tests | 3 |
| Failures | 1 |
| Test case: | [libs/FFTW] Serial C binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.338 sec |
| Failed | None |
None
| Test case: | [libs/FFTW] MPI C binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Failed |
| Duration: | 6.375 sec |
| Failed | None |
(from function `assert_success' in file ../../../common/test_helper_functions.bash, line 58, in test file rm_execution, line 40) `assert_success' failed -- command failed -- status : 1 output (365 lines): job script = /tmp/job.ohpc-test.30186 Batch job 103 submitted Job 103 failed... Reason=NonZeroExitCode [prun] Master compute host = c1 [prun] Resource manager = slurm [prun] Launch cmd = mpirun -- ./C_mpi_test 8 (family=openmpi5) [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 22 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [create_qp:2752]create qp: failed on ibv_cmd_create_qp with 95 [c1:9966 :0:9966] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2496b90) [c1:9969 :0:9969] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1794b80) ==== backtrace (tid: 9969) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f61574b4c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f61574b4dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f61574b50c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f6157a54db0] 4 [0x1794b80] ================================= ==== backtrace (tid: 9966) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fd5e083cc44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fd5e083cdff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fd5e083d0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fd5e0c54db0] 4 [0x2496b90] ================================= [c1:9967 :0:9967] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2357c30) [c1:9968 :0:9968] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1d03db0) ==== backtrace (tid: 9967) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f9b6e8c4c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f9b6e8c4dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f9b6e8c50c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f9b6ee54db0] 4 [0x2357c30] ================================= ==== backtrace (tid: 9968) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fd368f61c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7fd368f61dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7fd368f620c6] 3 /lib64/libc.so.6(+0x54db0) [0x7fd369454db0] 4 [0x1d03db0] ================================= [c2:8185 :0:8185] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18a8810) [c2:8186 :0:8186] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1921f20) [c2:8183 :0:8183] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x19d1810) [c2:8184 :0:8184] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xd5ff00) ==== backtrace (tid: 8185) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f44e9bcec44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f44e9bcedff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f44e9bcf0c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f44ea054db0] 4 [0x18a8810] ================================= ==== backtrace (tid: 8186) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f0df3cf3c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f0df3cf3dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f0df3cf40c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f0df4254db0] 4 [0x1921f20] ================================= ==== backtrace (tid: 8183) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f2f7e2d1c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f2f7e2d1dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f2f7e2d20c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f2f7e854db0] 4 [0x19d1810] ================================= ==== backtrace (tid: 8184) ==== 0 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f5eb52a7c44] 1 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x31dff) [0x7f5eb52a7dff] 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x320c6) [0x7f5eb52a80c6] 3 /lib64/libc.so.6(+0x54db0) [0x7f5eb5854db0] 4 [0xd5ff00] ================================= -------------------------------------------------------------------------- prterun noticed that process rank 1 with PID 9967 on node c1 exited on signal 11 (Segmentation fault). -------------------------------------------------------------------------- --
| Duration | 34.38 sec |
|---|---|
| Tests | 50 |
| Failures | 0 |
| Test case: | [libs/GSL] run test_gsl_histogram (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.04 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run block under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.284 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run bspline under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.531 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run cblas under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.171 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run cdf under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.171 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run cheb under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.163 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run combination under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.169 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run complex under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.157 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run const under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.148 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run deriv under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.158 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run dht under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.219 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run diff under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.168 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run eigen under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.336 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run err under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.157 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run fft under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.508 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run fit under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.168 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run histogram under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.222 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run ieee-utils under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.17 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run integration under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.038 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run interpolation under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.173 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run linalg under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.277 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run matrix under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.794 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run min under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.166 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run monte under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.434 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run multifit under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.825 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run multilarge under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.193 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run multimin under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.158 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run multiroots under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.163 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run multiset under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.156 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run ntuple under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.153 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run ode-initval under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.428 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run ode-initval2 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 5.153 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run permutation under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.146 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run poly under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.166 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run qrng under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.167 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run randist under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.01 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run rng under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.227 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run roots under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.16 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run rstat under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.097 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run siman under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.968 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run sort under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.234 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run spblas under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.318 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run specfunc under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.676 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run splinalg under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.324 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run spmatrix under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.188 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run statistics under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.16 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run sum under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.161 sec |
| Failed | None |
None
| Test case: | [libs/GSL] run sys under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.147 sec |
| Failed | None |
None
| Duration | 0.287 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/GSL] Verify GSL module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.165 sec |
| Failed | None |
None
| Test case: | [libs/GSL] Verify module GSL_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/GSL] Verify module GSL_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/GSL] Verify dynamic library available in GSL_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/GSL] Verify static library is not present in GSL_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 46.174 sec |
|---|---|
| Tests | 12 |
| Failures | 0 |
| Test case: | [libs/HYPRE] 2 PE structured test binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 5.483 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE PCG with SMG preconditioner binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.416 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE Semi-Structured PCG binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.356 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE Three-part stencil binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 6.556 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE FORTRAN PCG with PFMG preconditioner binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.287 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE -Delta u = 1 binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.409 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE convection-reaction-diffusion binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 5.489 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE FORTRAN 2-D Laplacian binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 6.554 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE Semi-Structured convection binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
skipped
| Test case: | [libs/HYPRE] 2 PE biharmonic binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
skipped
| Duration | 3.698 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/HYPRE] Verify HYPRE module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.198 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify module HYPRE_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify module HYPRE_BIN is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify module HYPRE_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify dynamic library available in HYPRE_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify static library is not present in HYPRE_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify module HYPRE_INC is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 24.738 sec |
|---|---|
| Tests | 12 |
| Failures | 0 |
| Test case: | [libs/HYPRE] 2 PE structured test binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.278 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE PCG with SMG preconditioner binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.276 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE Semi-Structured PCG binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.276 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE Three-part stencil binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.342 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE FORTRAN PCG with PFMG preconditioner binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.277 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE -Delta u = 1 binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.409 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE convection-reaction-diffusion binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.206 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE FORTRAN 2-D Laplacian binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.273 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] 2 PE Semi-Structured convection binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
skipped
| Test case: | [libs/HYPRE] 2 PE biharmonic binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
skipped
| Duration | 2.604 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/HYPRE] Verify HYPRE module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.19 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify module HYPRE_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify module HYPRE_BIN is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify module HYPRE_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify dynamic library available in HYPRE_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify static library is not present in HYPRE_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/HYPRE] Verify module HYPRE_INC is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 3.54 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Duration | 0.358 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/Metis] Verify METIS module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.171 sec |
| Failed | None |
None
| Test case: | [libs/Metis] Verify module METIS_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/Metis] Verify module METIS_BIN is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/Metis] Verify availability of m2gmetis binary (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.044 sec |
| Failed | None |
None
| Test case: | [libs/Metis] Verify module METIS_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/Metis] Verify dynamic library available in METIS_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/Metis] Verify static library is not present in METIS_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 11.407 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [libs/MFEM] p_laplace MPI binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.31 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] p_laplace_perf MPI binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.395 sec |
| Failed | None |
None
| Duration | 23.348 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [libs/MFEM] p_laplace MPI binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 7.735 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] p_laplace_perf MPI binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 6.649 sec |
| Failed | None |
None
| Duration | 0.349 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/MFEM] Verify MFEM module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.215 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] Verify module MFEM_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.024 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] Verify module MFEM_BIN is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.024 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] Verify module MFEM_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] Verify (dynamic) library available in MFEM_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Duration | 0.351 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/MFEM] Verify MFEM module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.215 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] Verify module MFEM_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.024 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] Verify module MFEM_BIN is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.024 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] Verify module MFEM_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/MFEM] Verify (dynamic) library available in MFEM_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Duration | 23.146 sec |
|---|---|
| Tests | 5 |
| Failures | 0 |
| Test case: | [libs/Mumps] C (double precision) runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 5.482 sec |
| Failed | None |
None
| Test case: | [libs/Mumps] Fortran (single precision) runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.42 sec |
| Failed | None |
None
| Test case: | [libs/Mumps] Fortran (double precision) runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.41 sec |
| Failed | None |
None
| Duration | 13.498 sec |
|---|---|
| Tests | 5 |
| Failures | 0 |
| Test case: | [libs/Mumps] C (double precision) runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.272 sec |
| Failed | None |
None
| Test case: | [libs/Mumps] Fortran (single precision) runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.341 sec |
| Failed | None |
None
| Test case: | [libs/Mumps] Fortran (double precision) runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.278 sec |
| Failed | None |
None
| Duration | 0.386 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.184 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify availability of nc-config binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.051 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 0.395 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.195 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.05 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 0.392 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.194 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.049 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 4.143 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/NetCDF] ncdump availability (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.232 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] verify nc4/hdf5 available for C interface (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.062 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] verify nc4 available for Fortran interface (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.039 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] verify nc4 available for C++ interface (gnu14/openmpi5) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
option no longer supported
| Test case: | [libs/NetCDF] C write/read (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.425 sec |
| Failed | None |
None
| Duration | 0.374 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.177 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify availability of nc-config binary (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.049 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 0.378 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.184 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.049 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 0.376 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.182 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.049 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 1.624 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/NetCDF] ncdump availability (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.065 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] verify nc4/hdf5 available for C interface (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.056 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] verify nc4 available for Fortran interface (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.038 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] verify nc4 available for C++ interface (gnu14) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
option no longer supported
| Test case: | [libs/NetCDF] C write/read (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.542 sec |
| Failed | None |
None
| Duration | 0.381 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.182 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify availability of nc-config binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.051 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 0.387 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.189 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.049 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 0.38 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.184 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.049 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 3.349 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/NetCDF] ncdump availability (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.18 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] verify nc4/hdf5 available for C interface (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.059 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] verify nc4 available for Fortran interface (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.038 sec |
| Failed | None |
None
| Test case: | [libs/NetCDF] verify nc4 available for C++ interface (gnu14/mpich) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
option no longer supported
| Test case: | [libs/NetCDF] C write/read (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.164 sec |
| Failed | None |
None
| Duration | 3.473 sec |
|---|---|
| Tests | 11 |
| Failures | 0 |
| Test case: | [libs/openblas/dblat1] dblat1 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.169 sec |
| Failed | None |
None
| Test case: | [libs/openblas/xccblat1] xccblat1 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.17 sec |
| Failed | None |
None
| Test case: | [libs/openblas/xzcblat1] xzcblat1 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.159 sec |
| Failed | None |
None
| Test case: | [libs/openblas/xscblat2] xscblat2 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.321 sec |
| Failed | None |
None
| Test case: | [libs/openblas/xdcblat2] xdcblat2 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.327 sec |
| Failed | None |
None
| Test case: | [libs/openblas/xccblat2] xccblat2 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.578 sec |
| Failed | None |
None
| Test case: | [libs/openblas/xzcblat2] xzcblat2 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.473 sec |
| Failed | None |
None
| Test case: | [libs/openblas/xscblat3] xscblat3 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.209 sec |
| Failed | None |
None
| Test case: | [libs/openblas/xdcblat3] xdcblat3 under resource manager (slurm/gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.225 sec |
| Failed | None |
None
| Duration | 0.243 sec |
|---|---|
| Tests | 5 |
| Failures | 0 |
| Test case: | [libs/OpenBLAS] Verify OPENBLAS module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.167 sec |
| Failed | None |
None
| Test case: | [libs/OpenBLAS] Verify module OPENBLAS_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/OpenBLAS] Verify module OPENBLAS_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.018 sec |
| Failed | None |
None
| Duration | 2.215 sec |
|---|---|
| Tests | 2 |
| Failures | 0 |
| Duration | 0.336 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/opencoarrays] Verify opencoarrays module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.176 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify module OPENCOARRAYS_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify module OPENCOARRAYS_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify dynamic library available in OPENCOARRAYS_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify static library is not present in OPENCOARRAYS_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify module OPENCOARRAYS_INC is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify header file is present in OPENCOARRAYS_INC (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Duration | 4.334 sec |
|---|---|
| Tests | 2 |
| Failures | 0 |
| Duration | 0.345 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/opencoarrays] Verify opencoarrays module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.18 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify module OPENCOARRAYS_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify module OPENCOARRAYS_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify dynamic library available in OPENCOARRAYS_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify static library is not present in OPENCOARRAYS_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify module OPENCOARRAYS_INC is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/opencoarrays] Verify header file is present in OPENCOARRAYS_INC (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 6.846 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 2.623 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/PETSc] Verify PETSC module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.205 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify module PETSC_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify module PETSC_BIN is defined and exists |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify module PETSC_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify dynamic library available in PETSC_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify static library is not present in PETSC_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify module PETSC_INC is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 16.51 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 3.719 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/PETSc] Verify PETSC module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.207 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify module PETSC_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify module PETSC_BIN is defined and exists |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify module PETSC_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify dynamic library available in PETSC_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify static library is not present in PETSC_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/PETSc] Verify module PETSC_INC is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 0.316 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [HDF5] Verify HDF5 module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.189 sec |
| Failed | None |
None
| Test case: | [HDF5] Verify module HDF5_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [HDF5] Verify module HDF5_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [HDF5] Verify dynamic library available in HDF5_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [HDF5] Verify static library is not present in HDF5_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 1.569 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.284 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/PLASMA] Verify plasma module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.165 sec |
| Failed | None |
None
| Test case: | [libs/PLASMA] Verify module PLASMA_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PLASMA] Verify module PLASMA_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PLASMA] Verify dynamic library available in PLASMA_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PLASMA] Verify static library is not present in PLASMA_LIB (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Duration | 0.269 sec |
|---|---|
| Tests | 5 |
| Failures | 0 |
| Test case: | [PNETCDF] Verify PNETCDF module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.186 sec |
| Failed | None |
None
| Test case: | [PNETCDF] Verify module PNETCDF_DIR is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [PNETCDF] Verify module PNETCDF_LIB is defined and exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 9.044 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [libs/PNETCDF] Parallel Fortran binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.261 sec |
| Failed | None |
None
| Test case: | [libs/PNETCDF] Parallel Fortran binary 2 runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.197 sec |
| Failed | None |
None
| Duration | 12.273 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [libs/PNETCDF] Parallel Fortran binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.334 sec |
| Failed | None |
None
| Test case: | [libs/PNETCDF] Parallel Fortran binary 2 runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.271 sec |
| Failed | None |
None
| Duration | 6.795 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.359 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/PTScotch] Verify PTSCOTCH module is loaded and matches rpm version (gnu14-mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.183 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify PTSCOTCH_DIR is defined and directory exists (gnu14-mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify module PTSCOTCH_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify dynamic library available in PTSCOTCH_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify static library is not present in PTSCOTCH_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify module PTSCOTCH_INC is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify header file is present in PTSCOTCH_INC (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 13.211 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.367 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/PTScotch] Verify PTSCOTCH module is loaded and matches rpm version (gnu14-openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.187 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify PTSCOTCH_DIR is defined and directory exists (gnu14-openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify module PTSCOTCH_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify dynamic library available in PTSCOTCH_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify static library is not present in PTSCOTCH_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify module PTSCOTCH_INC is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/PTScotch] Verify header file is present in PTSCOTCH_INC (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 11.21 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.272 sec |
| Failed | None |
None
| Test case: | [libs/ScaLAPACK/PDSCAEX] DPCGESV under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.267 sec |
| Failed | None |
None
| Duration | 0.266 sec |
|---|---|
| Tests | 5 |
| Failures | 0 |
| Test case: | [libs/ScaLAPACK] Verify SCALAPACK module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.183 sec |
| Failed | None |
None
| Test case: | [libs/ScaLAPACK] Verify module SCALAPACK_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/ScaLAPACK] Verify module SCALAPACK_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 21.887 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.406 sec |
| Failed | None |
None
| Test case: | [libs/ScaLAPACK/PDSCAEX] DPCGESV under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 5.47 sec |
| Failed | None |
None
| Duration | 0.266 sec |
|---|---|
| Tests | 5 |
| Failures | 0 |
| Test case: | [libs/ScaLAPACK] Verify SCALAPACK module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.181 sec |
| Failed | None |
None
| Test case: | [libs/ScaLAPACK] Verify module SCALAPACK_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [libs/ScaLAPACK] Verify module SCALAPACK_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 0.498 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.345 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/Scotch] Verify SCOTCH module is loaded and matches rpm version (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.171 sec |
| Failed | None |
None
| Test case: | [libs/Scotch] Verify SCOTCH_DIR is defined and directory exists (gnu14) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/Scotch] Verify module SCOTCH_LIB is defined and exists (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/Scotch] Verify dynamic library available in SCOTCH_LIB (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Test case: | [libs/Scotch] Verify static library is not present in SCOTCH_LIB (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/Scotch] Verify module SCOTCH_INC is defined and exists (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/Scotch] Verify header file is present in SCOTCH_INC (gnu14/) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 10.207 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [libs/SLEPc] F90 SVD test binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.289 sec |
| Failed | None |
None
| Test case: | [libs/SLEPc] C SVD of the Lauchli matrix binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.286 sec |
| Failed | None |
None
| Duration | 0.321 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/slepc] Verify slepc module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.195 sec |
| Failed | None |
None
| Test case: | [libs/slepc] Verify module SLEPC_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [libs/slepc] Verify module SLEPC_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Test case: | [libs/slepc] Verify dynamic library available in SLEPC_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/slepc] Verify static library is not present in SLEPC_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 16.663 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [libs/SLEPc] F90 SVD test binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.435 sec |
| Failed | None |
None
| Test case: | [libs/SLEPc] C SVD of the Lauchli matrix binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.433 sec |
| Failed | None |
None
| Duration | 0.33 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/slepc] Verify slepc module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.201 sec |
| Failed | None |
None
| Test case: | [libs/slepc] Verify module SLEPC_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.024 sec |
| Failed | None |
None
| Test case: | [libs/slepc] Verify module SLEPC_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/slepc] Verify dynamic library available in SLEPC_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/slepc] Verify static library is not present in SLEPC_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 48.484 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/superLU_dist] PDGSSVX with full (default) options (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 5.503 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] pdgssvx_ABglobal with full (default) options (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 5.501 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] vary RHS (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.44 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] vary RHS ABglobal (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 6.575 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] reuse permutation vector (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 5.498 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] reuse permutation vector ABglobal (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 6.583 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] reuse symbolic factorization (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.441 sec |
| Failed | None |
None
| Duration | 0.326 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/superlu_dist] Verify SUPERLU_DIST module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.2 sec |
| Failed | None |
None
| Test case: | [libs/superlu_dist] Verify module SUPERLU_DIST_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/superlu_dist] Verify module SUPERLU_DIST_LIB is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/superlu_dist] Verify dynamic library available in SUPERLU_DIST_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/superlu_dist] Verify static library is not present in SUPERLU_DIST_LIB (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Duration | 23.732 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/superLU_dist] PDGSSVX with full (default) options (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.216 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] pdgssvx_ABglobal with full (default) options (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.347 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] vary RHS (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 1.211 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] vary RHS ABglobal (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.284 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] reuse permutation vector (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.352 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] reuse permutation vector ABglobal (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.343 sec |
| Failed | None |
None
| Test case: | [libs/superLU_dist] reuse symbolic factorization (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.282 sec |
| Failed | None |
None
| Duration | 0.318 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [libs/superlu_dist] Verify SUPERLU_DIST module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.194 sec |
| Failed | None |
None
| Test case: | [libs/superlu_dist] Verify module SUPERLU_DIST_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/superlu_dist] Verify module SUPERLU_DIST_LIB is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/superlu_dist] Verify dynamic library available in SUPERLU_DIST_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [libs/superlu_dist] Verify static library is not present in SUPERLU_DIST_LIB (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.02 sec |
| Failed | None |
None
| Duration | 31.001 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/Trilinos] Kokkos-MemorySpace runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.362 sec |
| Failed | None |
None
| Test case: | [libs/Trilinos] Tpetra-InitMPI runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 7.643 sec |
| Failed | None |
None
| Test case: | [libs/Trilinos] Tpetra-DataRedistribution runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 6.571 sec |
| Failed | None |
None
| Test case: | [libs/Trilinos] Epetra-DataRedistribution runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.423 sec |
| Failed | None |
None
| Test case: | [libs/Trilinos] Epetra-Galeri runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
skip Epetra-Galeri in 2.x
| Test case: | [libs/Trilinos] Epetra-Ifpack runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
skip Epetra-Ifpack in 2.x
| Test case: | [libs/Trilinos] Teuchos-ParameterList runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.359 sec |
| Failed | None |
None
| Duration | 19.144 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [libs/Trilinos] Kokkos-MemorySpace runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.274 sec |
| Failed | None |
None
| Test case: | [libs/Trilinos] Tpetra-InitMPI runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.278 sec |
| Failed | None |
None
| Test case: | [libs/Trilinos] Tpetra-DataRedistribution runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.34 sec |
| Failed | None |
None
| Test case: | [libs/Trilinos] Epetra-DataRedistribution runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.278 sec |
| Failed | None |
None
| Test case: | [libs/Trilinos] Epetra-Galeri runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
skip Epetra-Galeri in 2.x
| Test case: | [libs/Trilinos] Epetra-Ifpack runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Skipped |
| Duration: | 0.0 sec |
| Failed | None |
| Skipped | None |
None
skip Epetra-Ifpack in 2.x
| Test case: | [libs/Trilinos] Teuchos-ParameterList runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.28 sec |
| Failed | None |
None
| Duration | 5.651 sec |
|---|---|
| Tests | 8 |
| Failures | 0 |
| Duration | 1.821 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Test case: | [modules] env variable passes through () |
|---|---|
| Outcome: | Passed |
| Duration: | 0.395 sec |
| Failed | None |
None
| Test case: | [modules] loaded module passes through () |
|---|---|
| Outcome: | Passed |
| Duration: | 0.485 sec |
| Failed | None |
None
| Duration | 1.884 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 8.706 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 7.635 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.54 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 3.431 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 11.773 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Test case: | [MPI] C binary runs on two nodes under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 6.41 sec |
| Failed | None |
None
| Duration | 7.553 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Test case: | [MPI] C binary runs on single node under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.255 sec |
| Failed | None |
None
| Duration | 0.597 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.157 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 1.986 sec |
|---|---|
| Tests | 6 |
| Failures | 0 |
| Test case: | [Dimemas] Verify dimemas module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.195 sec |
| Failed | None |
None
| Test case: | [Dimemas] Verify module DIMEMAS_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.033 sec |
| Failed | None |
None
| Test case: | [Dimemas] Verify module DIMEMAS_BIN is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.033 sec |
| Failed | None |
None
| Test case: | [Dimemas] Verify availability of prv2dim binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.046 sec |
| Failed | None |
None
| Duration | 1.997 sec |
|---|---|
| Tests | 6 |
| Failures | 0 |
| Test case: | [Dimemas] Verify dimemas module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.202 sec |
| Failed | None |
None
| Test case: | [Dimemas] Verify module DIMEMAS_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.033 sec |
| Failed | None |
None
| Test case: | [Dimemas] Verify module DIMEMAS_BIN is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.03 sec |
| Failed | None |
None
| Test case: | [Dimemas] Verify availability of prv2dim binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.046 sec |
| Failed | None |
None
| Duration | 0.225 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.232 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.361 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [OMB] Verify OMB module is loaded and matches rpm version (gnu14-mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.176 sec |
| Failed | None |
None
| Test case: | [OMB] Verify OMB_DIR is defined and directory exists (gnu14-mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Test case: | [OMB] Verify osu_bw binary is available (gnu14-mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [OMB] Verify osu_latency binary is available (gnu14-mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [OMB] Verify osu_allgather binary is available (gnu14-mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.033 sec |
| Failed | None |
None
| Duration | 0.372 sec |
|---|---|
| Tests | 7 |
| Failures | 0 |
| Test case: | [OMB] Verify OMB module is loaded and matches rpm version (gnu14-openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.183 sec |
| Failed | None |
None
| Test case: | [OMB] Verify OMB_DIR is defined and directory exists (gnu14-openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.019 sec |
| Failed | None |
None
| Test case: | [OMB] Verify osu_bw binary is available (gnu14-openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [OMB] Verify osu_latency binary is available (gnu14-openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [OMB] Verify osu_allgather binary is available (gnu14-openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Duration | 16.369 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.503 sec |
|---|---|
| Tests | 11 |
| Failures | 0 |
| Test case: | [perf-tools/Scalasca] Verify scalasca module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.189 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify module SCALASCA_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.021 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify module SCALASCA_BIN is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scalasca binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scan binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scout.mpi binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scout.omp binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scout.hyb binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scout.ser binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Duration | 23.976 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 0.52 sec |
|---|---|
| Tests | 11 |
| Failures | 0 |
| Test case: | [perf-tools/Scalasca] Verify scalasca module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.192 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify module SCALASCA_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.023 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify module SCALASCA_BIN is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scalasca binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scan binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.036 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scout.mpi binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scout.omp binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.036 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scout.hyb binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Scalasca] Verify availability of scout.ser binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Duration | 1.272 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [perf-tools/Score-P] MPI C binary includes expected Score-P instrumenter symbols (/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.099 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] MPI C++ binary includes expected Score-P instrumenter symbols (/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.1 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] MPI Fortran binary includes expected Score-P instrumenter symbols (/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.099 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial C OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.119 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial C++ OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.12 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial Fortran OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.12 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary includes expected Score-P instrumenter symbols (/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.204 sec |
| Failed | None |
None
| Duration | 25.301 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [perf-tools/Score-P] MPI C binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.566 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] MPI C++ binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.493 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] MPI Fortran binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 3.789 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial C OpenMP binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.414 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial C++ OpenMP binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.375 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial Fortran OpenMP binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.395 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary runs under resource manager (slurm/gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 2.514 sec |
| Failed | None |
None
| Duration | 0.732 sec |
|---|---|
| Tests | 14 |
| Failures | 0 |
| Test case: | [perf-tools/Score-P] Verify scorep module is loaded and matches rpm version (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.192 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify module SCOREP_DIR is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify module SCOREP_BIN is defined and exists (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-backend-info binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-config binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-info binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-preload-init binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-score binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-wrapper binary (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep wrapper binaries (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.153 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of user guide for scorep (gnu14/mpich) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.033 sec |
| Failed | None |
None
| Duration | 1.353 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [perf-tools/Score-P] MPI C binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.107 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] MPI C++ binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.109 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] MPI Fortran binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.107 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial C OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.122 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial C++ OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.122 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial Fortran OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.123 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.221 sec |
| Failed | None |
None
| Duration | 42.481 sec |
|---|---|
| Tests | 9 |
| Failures | 0 |
| Test case: | [perf-tools/Score-P] MPI C binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.643 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] MPI C++ binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 5.721 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] MPI Fortran binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 7.012 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial C OpenMP binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.436 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial C++ OpenMP binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.441 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Serial Fortran OpenMP binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.411 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary runs under resource manager (slurm/gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 4.676 sec |
| Failed | None |
None
| Duration | 0.783 sec |
|---|---|
| Tests | 14 |
| Failures | 0 |
| Test case: | [perf-tools/Score-P] Verify scorep module is loaded and matches rpm version (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.191 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify module SCOREP_DIR is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify module SCOREP_BIN is defined and exists (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.022 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-backend-info binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-config binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-info binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-preload-init binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.034 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-score binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep-wrapper binary (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of scorep wrapper binaries (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.202 sec |
| Failed | None |
None
| Test case: | [perf-tools/Score-P] Verify availability of user guide for scorep (gnu14/openmpi5) |
|---|---|
| Outcome: | Passed |
| Duration: | 0.035 sec |
| Failed | None |
None
| Duration | 4.47 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 8.728 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |
| Duration | 3.473 sec |
|---|---|
| Tests | 5 |
| Failures | 0 |
| Duration | 9.925 sec |
|---|---|
| Tests | 4 |
| Failures | 0 |
| Duration | 0.915 sec |
|---|---|
| Tests | 3 |
| Failures | 0 |