results

Test Suite: InstallTests.computes_installed-20241019093110

Results

Duration	1.057 sec
Tests	5
Failures	2

Tests

InstallTests.computes_installed

Test case:	test_01_numcomputes_greater_than_zero
Outcome:	Passed
Duration:	0.0 sec
Failed	None

None

Test case:	test_02_koomie_cf_available
Outcome:	Passed
Duration:	0.0 sec
Failed	None

None

Test case:	test_03_nonzero_results_from_uptime
Outcome:	Passed
Duration:	1.055 sec
Failed	None

None

Test case:	test_04_correct_number_of_hosts_booted
Outcome:	Failed
Duration:	0.001 sec
Failed	5 != 2

Traceback (most recent call last):
  File "computes_installed.py", line 59, in test_04_correct_number_of_hosts_booted
    self.assertEqual(num_lines, self.num_computes)
AssertionError: 5 != 2

Test case:	test_05_verify_boot_times_are_reasonable
Outcome:	Failed
Duration:	0.0 sec
Failed	could not convert string to float: 'Permission'

Traceback (most recent call last):
  File "computes_installed.py", line 74, in test_05_verify_boot_times_are_reasonable
    if float(vals[1]) >= uptimeThreshold:
ValueError: could not convert string to float: 'Permission'

Test Suite: sms_installed.bats

Results

Duration	0.051 sec
Tests	2
Failures	0

Tests

sms_installed.bats

Test case:	Verify hostname matches expectations
Outcome:	Passed
Duration:	0.025 sec
Failed	None

None

Test case:	Base OS check
Outcome:	Passed
Duration:	0.026 sec
Failed	None

None

Test Suite: clustershell

Results

Duration	1.026 sec
Tests	2
Failures	1

Tests

clustershell

Test case:	[clush] check for OS-provided RPM
Outcome:	Passed
Duration:	0.045 sec
Failed	None

None

Test case:	[clush] clush -Sg compute
Outcome:	Failed
Duration:	0.981 sec
Failed	None

(from function `assert_success' in file ./common/test_helper_functions.bash, line 58,
 in test file clustershell, line 26)
  `assert_success' failed

-- command failed --
status : 255
output (3 lines):
  c1: root@c1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,hostbased).
  clush: c1: exited with exit code 255
  c2: c2
--

Test Suite: conman

Results

Duration	0.072 sec
Tests	2
Failures	0

Tests

conman

Test case:	[conman] check for RPM
Outcome:	Passed
Duration:	0.042 sec
Failed	None

None

Test case:	[conman] query conmand conman
Outcome:	Passed
Duration:	0.03 sec
Failed	None

None

Test Suite: genders

Results

Duration	0.069 sec
Tests	2
Failures	0

Tests

genders

Test case:	[genders] check for RPM
Outcome:	Passed
Duration:	0.042 sec
Failed	None

None

Test case:	[genders] check node attributes
Outcome:	Passed
Duration:	0.027 sec
Failed	None

None

Test Suite: nhc

Results

Duration	1.231 sec
Tests	3
Failures	0

Tests

nhc

Test case:	[nhc] check for RPM
Outcome:	Passed
Duration:	0.042 sec
Failed	None

None

Test case:	[nhc] generate config file
Outcome:	Passed
Duration:	0.827 sec
Failed	None

None

Test case:	[nhc] service failure detection and restart
Outcome:	Passed
Duration:	0.362 sec
Failed	None

None

Test Suite: slurm-plugins

Results

Duration	4.656 sec
Tests	4
Failures	0

Tests

slurm-plugins

Test case:	[slurm] check for jobcomp_elasticsearch plugin
Outcome:	Passed
Duration:	0.026 sec
Failed	None

None

Test case:	[slurm] check for job_submit_lua plugin
Outcome:	Passed
Duration:	0.026 sec
Failed	None

None

Test case:	[slurm] check for --x11 option
Outcome:	Passed
Duration:	0.038 sec
Failed	None

None

Test case:	[slurm] check for sview rpm availability
Outcome:	Passed
Duration:	4.566 sec
Failed	None

None

Test Suite: lmod

Results

Duration	0.449 sec
Tests	1
Failures	0

Tests

lmod

Test case:	[lmod] test that the setup function passed
Outcome:	Passed
Duration:	0.449 sec
Failed	None

None

Test Suite: spack

Results

Duration	62.12 sec
Tests	4
Failures	0

Tests

spack

Test case:	[spack] check for RPM
Outcome:	Passed
Duration:	0.583 sec
Failed	None

None

Test case:	[spack] add compiler
Outcome:	Passed
Duration:	2.089 sec
Failed	None

None

Test case:	[spack] test build
Outcome:	Passed
Duration:	57.925 sec
Failed	None

None

Test case:	[spack] test module refresh
Outcome:	Passed
Duration:	1.523 sec
Failed	None

None

Test Suite: build

Results

Duration	6.33 sec
Tests	1
Failures	0

Tests

build

Test case:	[/apps/hpcg] build HPCG executable (gnu14/mpich)
Outcome:	Passed
Duration:	6.33 sec
Failed	None

None

Test Suite: rm_execution_multi_host

Results

Duration	953.168 sec
Tests	3
Failures	0

Tests

rm_execution_multi_host

Test case:	[/apps/hpcg] check if resource manager is defined
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[/apps/hpcg] run HPCG on multi nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	953.09 sec
Failed	None

None

Test case:	[/apps/hpcg] log HPCG multi node results (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.059 sec
Failed	None

None

Test Suite: rm_execution_single_host

Results

Duration	13.967 sec
Tests	3
Failures	0

Tests

rm_execution_single_host

Test case:	[/apps/hpcg] check if resource manager is defined
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[/apps/hpcg] run HPCG on single node under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	13.89 sec
Failed	None

None

Test case:	[/apps/hpcg] log HPCG single node results (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.058 sec
Failed	None

None

Test Suite: build

Results

Duration	4.334 sec
Tests	1
Failures	0

Tests

build

Test case:	[/apps/hpcg] build HPCG executable (gnu14/openmpi5)
Outcome:	Passed
Duration:	4.334 sec
Failed	None

None

Test Suite: rm_execution_multi_host

Results

Duration	6.571 sec
Tests	3
Failures	2

Tests

rm_execution_multi_host

Test case:	[/apps/hpcg] check if resource manager is defined
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[/apps/hpcg] run HPCG on multi nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.519 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution_multi_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.19458
Batch job 13 submitted

Job 13 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./bin/xhpcg.gnu14.openmpi5 32 32 32 10 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.875181] [c2:4440 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2551d60 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04440] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.883609] [c2:4447 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x210dcc0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04447] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.889690] [c2:4439 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1176d60 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04439] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.893167] [c2:4442 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e66d80 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04442] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.912383] [c2:4444 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x256bd80 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.912670] [c2:4445 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x15ffd10 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.928109] [c2:4443 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x198cd70 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729345917.928112] [c2:4441 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1785da0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04445] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:04444] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:04441] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:04443] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.765193] [c1:2759 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16b61a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02759] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.773122] [c1:2755 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x27131a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02755] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.794483] [c1:2756 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x15d1cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02756] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.799000] [c1:2758 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x173d1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02758] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.807887] [c1:2760 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x27e01a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.808993] [c1:2761 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c61cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02760] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.809991] [c1:2757 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x17c61a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02761] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02757] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345917.821808] [c1:2754 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x231fcb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02754] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:2754 :0:2754] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1eb4ee0)
[c1:2757 :0:2757] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x149ba20)
==== backtrace (tid:   2754) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f77632662bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f77632665ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f7763266854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f7763917910]
 4  [0x1eb4ee0]
=================================
==== backtrace (tid:   2757) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f8e6bda02bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f8e6bda05ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f8e6bda0854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f8e6c451910]
 4  [0x149ba20]
=================================
[c1:2760 :0:2760] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x24b5a20)
[c1:2761 :0:2761] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x17f7500)
[c1:2756 :0:2756] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1166ee0)
==== backtrace (tid:   2760) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc4c55872bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc4c55875ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc4c5587854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc4c5c38910]
 4  [0x24b5a20]
=================================
==== backtrace (tid:   2761) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fabb3d6e2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fabb3d6e5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fabb3d6e854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fabb441f910]
 4  [0x17f7500]
=================================
==== backtrace (tid:   2756) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fe6624b22bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fe6624b25ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fe6624b2854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fe662b63910]
 4  [0x1166ee0]
=================================
[c1:2758 :0:2758] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1430890)
[c1:2755 :0:2755] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x24aa9e0)
==== backtrace (tid:   2758) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f42ed80c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f42ed80c5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f42ed80c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f42edebd910]
 4  [0x1430890]
=================================
[c1:2759 :0:2759] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x138ba20)
==== backtrace (tid:   2755) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f72c04a82bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f72c04a85ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f72c04a8854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f72c0b59910]
 4  [0x24aa9e0]
=================================
==== backtrace (tid:   2759) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7ff20b2312bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7ff20b2315ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7ff20b231854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7ff20b8e2910]
 4  [0x138ba20]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 2754 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[/apps/hpcg] log HPCG multi node results (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	0.033 sec
Failed	None

(in test file rm_execution_multi_host, line 55)
  `mv $run_yaml $wrk_yaml' failed
Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg
ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory
Moving  to HPCG.32x32x32.P16.gnu14.openmpi5.yaml
mv: missing destination file operand after 'HPCG.32x32x32.P16.gnu14.openmpi5.yaml'
Try 'mv --help' for more information.

Test Suite: rm_execution_single_host

Results

Duration	3.393 sec
Tests	3
Failures	2

Tests

rm_execution_single_host

Test case:	[/apps/hpcg] check if resource manager is defined
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[/apps/hpcg] run HPCG on single node under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	3.341 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution_single_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT ${EXE} "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.4272
Batch job 12 submitted

Job 12 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./bin/xhpcg.gnu14.openmpi5 32 32 32 10 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345912.984884] [c1:2692 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1609050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729345912.984884] [c1:2695 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x27cb050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345912.986110] [c1:2693 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1a83050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345913.000234] [c1:2691 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1390050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729345913.000237] [c1:2696 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xaf7050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729345913.000240] [c1:2694 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2318050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345913.000401] [c1:2698 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x147c050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02692] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02695] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02693] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02691] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02696] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345913.020102] [c1:2697 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x21cd050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02694] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02698] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02697] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:2695 :0:2695] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x24c05e0)
[c1:2698 :0:2698] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1173e10)
[c1:2692 :0:2692] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x12fce20)
[c1:2691 :0:2691] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x10851e0)
[c1:2697 :0:2697] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1ec21e0)
[c1:2694 :0:2694] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20c4730)
==== backtrace (tid:   2698) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4996ea12bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4996ea15ec]
==== backtrace (tid:   2695) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f39dfcf32bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f39dfcf35ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f39dfcf3854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f39e03a4910]
 4  [0x24c05e0]
[c1:2696 :0:2696] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7ec1e0)
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4996ea1854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f4997552910]
 4  [0x1173e10]
=================================
=================================
==== backtrace (tid:   2692) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f354c37d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f354c37d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f354c37d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f354ca2e910]
 4  [0x12fce20]
=================================
==== backtrace (tid:   2691) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fbab3c0a2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fbab3c0a5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fbab3c0a854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fbab42bb910]
 4  [0x10851e0]
=================================
==== backtrace (tid:   2697) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f586e8752bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f586e8755ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f586e875854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f586ef26910]
 4  [0x1ec21e0]
=================================
==== backtrace (tid:   2694) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f247e44d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f247e44d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f247e44d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f247eafe910]
 4  [0x20c4730]
=================================
==== backtrace (tid:   2696) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f7f732fd2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f7f732fd5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f7f732fd854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f7f739ae910]
 4  [0x7ec1e0]
=================================
[c1:2693 :0:2693] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x181bf40)
==== backtrace (tid:   2693) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc7a304f2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc7a304f5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc7a304f854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc7a3700910]
 4  [0x181bf40]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 4 with PID 2695 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[/apps/hpcg] log HPCG single node results (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	0.033 sec
Failed	None

(in test file rm_execution_single_host, line 55)
  `mv $run_yaml $wrk_yaml' failed
Finding latest HPCG-Bencmark-*.yaml in /home/ohpc-test/tests/apps/hpcg
ls: cannot access 'HPCG-Benchmark-*.yaml': No such file or directory
Moving  to HPCG.32x32x32.P8.gnu14.openmpi5.yaml
mv: missing destination file operand after 'HPCG.32x32x32.P8.gnu14.openmpi5.yaml'
Try 'mv --help' for more information.

Test Suite: build

Results

Duration	9.088 sec
Tests	1
Failures	0

Tests

build

Test case:	[Apps/miniFE] build MiniFE executable (gnu14/mpich)
Outcome:	Passed
Duration:	9.088 sec
Failed	None

None

Test Suite: rm_execution_multi_host

Results

Duration	16.079 sec
Tests	2
Failures	0

Tests

rm_execution_multi_host

Test case:	[Apps/miniFE] run miniFE on multi nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	16.025 sec
Failed	None

None

Test case:	[Apps/miniFE] log miniFE multi node results (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.054 sec
Failed	None

None

Test Suite: rm_execution_single_host

Results

Duration	3.363 sec
Tests	2
Failures	0

Tests

rm_execution_single_host

Test case:	[Apps/miniFE] run miniFE on single node under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.31 sec
Failed	None

None

Test case:	[Apps/miniFE] log miniFE single node results (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.053 sec
Failed	None

None

Test Suite: build

Results

Duration	6.409 sec
Tests	1
Failures	0

Tests

build

Test case:	[Apps/miniFE] build MiniFE executable (gnu14/openmpi5)
Outcome:	Passed
Duration:	6.409 sec
Failed	None

None

Test Suite: rm_execution_multi_host

Results

Duration	4.45 sec
Tests	2
Failures	1

Tests

rm_execution_multi_host

Test case:	[Apps/miniFE] run miniFE on multi nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.398 sec
Failed	None

(from function `run_mpi_binary' in file ./common-test/functions, line 414,
 in test file rm_execution_multi_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.427
Batch job 17 submitted

Job 17 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./src/miniFE.x.gnu14.openmpi5 nx=256 ny=256 nz=256 verify_solution=0 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.877239] [c2:4579 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xe98df0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04579] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.880174] [c2:4578 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1abee00 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04578] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.908682] [c2:4581 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x13a5d20 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.909376] [c2:4576 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x7cdd60 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04581] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.913258] [c2:4574 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2561ef0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.915108] [c2:4577 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c43f90 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.923091] [c2:4575 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x218adc0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04575] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:04576] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:04577] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:04574] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.932660] [c2:4580 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x159fcd0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04580] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.848921] [c1:3048 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x223a1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03048] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.880566] [c1:3041 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1364050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.887479] [c1:3047 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc421a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03047] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.891596] [c1:3045 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x12d5050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03045] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:03041] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.927643] [c1:3046 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25201a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03046] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.944305] [c1:3043 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x19f61a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03043] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.947528] [c1:3042 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x23641a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345972.951235] [c1:3044 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2ac6cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03044] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:03042] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:3044 :0:3044] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x265bee0)
==== backtrace (tid:   3044) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f3a2bd8a2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f3a2bd8a5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f3a2bd8a854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f3a2c43b910]
 4  [0x265bee0]
=================================
[c1:3042 :0:3042] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20399e0)
[c1:3043 :0:3043] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x16cba20)
[c1:3041 :0:3041] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x105d380)
[c1:3046 :0:3046] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20b3550)
==== backtrace (tid:   3042) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fba33fb32bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fba33fb35ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fba33fb3854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fba34664910]
 4  [0x20399e0]
=================================
==== backtrace (tid:   3043) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fdb1c56a2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fdb1c56a5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fdb1c56a854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fdb1cc1b910]
 4  [0x16cba20]
=================================
[c1:3048 :0:3048] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1f0f9e0)
==== backtrace (tid:   3046) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4cfd67c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4cfd67c5ec]
==== backtrace (tid:   3041) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5d20a8c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5d20a8c5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5d20a8c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5d2113d910]
 4  [0x105d380]
=================================
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4cfd67c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f4cfdd2d910]
 4  [0x20b3550]
=================================
==== backtrace (tid:   3048) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f1e0b38e2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f1e0b38e5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f1e0b38e854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f1e0ba3f910]
 4  [0x1f0f9e0]
=================================
[c1:3047 :0:3047] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x917a20)
[c1:3045 :0:3045] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xfce380)
==== backtrace (tid:   3047) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fb951ff22bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fb951ff25ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fb951ff2854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fb9526a3910]
 4  [0x917a20]
=================================
==== backtrace (tid:   3045) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f7d322332bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f7d322335ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f7d32233854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f7d328e4910]
 4  [0xfce380]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 3 with PID 3044 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[Apps/miniFE] log miniFE multi node results (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.052 sec
Failed	None

None

Test Suite: rm_execution_single_host

Results

Duration	2.333 sec
Tests	2
Failures	1

Tests

rm_execution_single_host

Test case:	[Apps/miniFE] run miniFE on single node under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	2.281 sec
Failed	None

(from function `run_mpi_binary' in file ./common-test/functions, line 414,
 in test file rm_execution_single_host, line 46)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.10782
Batch job 16 submitted

Job 16 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./src/miniFE.x.gnu14.openmpi5 nx=100 ny=100 nz=100 verify_solution=1 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345969.575537] [c1:2983 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x153f010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345969.586598] [c1:2981 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x10da050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02981] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02983] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345969.607530] [c1:2984 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2430050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02984] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345969.612166] [c1:2979 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x23fa010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345969.613042] [c1:2985 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16d2050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02985] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345969.623789] [c1:2980 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b7c010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345969.628124] [c1:2982 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1868010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02982] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729345969.631744] [c1:2986 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2882010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:02979] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02986] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:02980] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:2985 :0:2985] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x13cbe20)
[c1:2986 :0:2986] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x25552f0)
[c1:2982 :0:2982] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x17acdd0)
[c1:2979 :0:2979] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x21f4d40)
==== backtrace (tid:   2985) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f3cc36f22bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f3cc36f25ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f3cc36f2854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f3cc3da3910]
 4  [0x13cbe20]
=================================
==== backtrace (tid:   2986) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f97a645c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f97a645c5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f97a645c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f97a6b0d910]
 4  [0x25552f0]
=================================
==== backtrace (tid:   2982) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f55010be2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f55010be5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f55010be854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f550176f910]
 4  [0x17acdd0]
=================================
==== backtrace (tid:   2979) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f3141f0d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f3141f0d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f3141f0d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f31425be910]
 4  [0x21f4d40]
=================================
[c1:2980 :0:2980] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x184f2f0)
==== backtrace (tid:   2980) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7ff4771692bc]
[c1:2984 :0:2984] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2125f40)
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7ff4771695ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7ff477169854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7ff47781a910]
 4  [0x184f2f0]
=================================
[c1:2983 :0:2983] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1316d60)
[c1:2981 :0:2981] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xdcff40)
==== backtrace (tid:   2984) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f92f81772bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f92f81775ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f92f8177854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f92f8828910]
 4  [0x2125f40]
=================================
==== backtrace (tid:   2983) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f8afae2d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f8afae2d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f8afae2d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f8afb4de910]
 4  [0x1316d60]
=================================
==== backtrace (tid:   2981) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f85822cc2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f85822cc5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f85822cc854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f858297d910]
 4  [0xdcff40]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 6 with PID 2985 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[Apps/miniFE] log miniFE single node results (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.052 sec
Failed	None

None

Test Suite: os_distribution

Results

Duration	0.016 sec
Tests	1
Failures	0

Tests

os_distribution

Test case:	[BOS] OS distribution matches (local)
Outcome:	Passed
Duration:	0.016 sec
Failed	None

None

Test Suite: computes

Results

Duration	1.768 sec
Tests	4
Failures	2

Tests

computes

Test case:	[BOS] OS distribution matches (1 active computes)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

disable BOS_RELEASE check

Test case:	[BOS] consistent kernel (1 active computes)
Outcome:	Failed
Duration:	0.591 sec
Failed	None

(from function `assert_equal' in file ./common/test_helper_functions.bash, line 38,
 in test file computes, line 38)
  `assert_equal $kernel $sms_kernel' failed
output expected: Permission
output observed: 5.14.21-150500.53-default

Test case:	[BOS] increased locked memory limits
Outcome:	Failed
Duration:	0.591 sec
Failed	None

(from function `assert_equal' in file ./common/test_helper_functions.bash, line 38,
 in test file computes, line 50)
  `assert_equal $limit unlimited' failed
output expected: Permission
output observed: unlimited

Test case:	[BOS] syslog forwarding
Outcome:	Passed
Duration:	0.586 sec
Failed	None

None

Test Suite: debugger

Results

Duration	0.035 sec
Tests	2
Failures	0

Tests

debugger

Test case:	[Compilers] debugger man page (gnu14)
Outcome:	Passed
Duration:	0.017 sec
Failed	None

None

Test case:	[Compilers] debugger availability (gnu14)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test Suite: man_pages

Results

Duration	0.217 sec
Tests	3
Failures	0

Tests

man_pages

Test case:	[Compilers] C compiler man/help page (gnu14)
Outcome:	Passed
Duration:	0.074 sec
Failed	None

None

Test case:	[Compilers] C++ compiler man/help page (gnu14)
Outcome:	Passed
Duration:	0.076 sec
Failed	None

None

Test case:	[Compilers] Fortran compiler man/help page (gnu14)
Outcome:	Passed
Duration:	0.067 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.719 sec
Tests	6
Failures	0

Tests

rm_execution

Test case:	[Compilers] C binary runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.127 sec
Failed	None

None

Test case:	[Compilers] C++ binary runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.104 sec
Failed	None

None

Test case:	[Compilers] Fortran binary runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.112 sec
Failed	None

None

Test case:	[Compilers] C openmp binary runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.122 sec
Failed	None

None

Test case:	[Compilers] C++ openmp binary runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.128 sec
Failed	None

None

Test case:	[Compilers] Fortran openmp binary runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.126 sec
Failed	None

None

Test Suite: version_match

Results

Duration	0.418 sec
Tests	3
Failures	0

Tests

version_match

Test case:	[Compilers] compiler module loaded (gnu14)
Outcome:	Passed
Duration:	0.129 sec
Failed	None

None

Test case:	[Compilers] compiler module version available (gnu14)
Outcome:	Passed
Duration:	0.122 sec
Failed	None

None

Test case:	[Compilers] C, C++, and Fortran versions match module (gnu14)
Outcome:	Passed
Duration:	0.167 sec
Failed	None

None

Test Suite: test_autotools

Results

Duration	4.731 sec
Tests	4
Failures	0

Tests

test_autotools

Test case:	[/dev-tools/autotools] running autoreconf
Outcome:	Passed
Duration:	2.953 sec
Failed	None

None

Test case:	[/dev-tools/autotools] run generated configure
Outcome:	Passed
Duration:	1.605 sec
Failed	None

None

Test case:	[/dev-tools/autotools] run make on generated Makefile
Outcome:	Passed
Duration:	0.15 sec
Failed	None

None

Test case:	[/dev-tools/autotools] run compiled binary
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test Suite: test_cmake

Results

Duration	1.725 sec
Tests	4
Failures	0

Tests

test_cmake

Test case:	[/dev-tools/cmake] running cmake --system-information
Outcome:	Passed
Duration:	0.696 sec
Failed	None

None

Test case:	[/dev-tools/cmake] run cmake
Outcome:	Passed
Duration:	0.353 sec
Failed	None

None

Test case:	[/dev-tools/cmake] run make on generated Makefile
Outcome:	Passed
Duration:	0.655 sec
Failed	None

None

Test case:	[/dev-tools/cmake] run compiled binary
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: EasyBuild

Results

Duration	10.958 sec
Tests	3
Failures	0

Tests

EasyBuild

Test case:	[EasyBuild] check for RPM
Outcome:	Passed
Duration:	0.092 sec
Failed	None

None

Test case:	[EasyBuild] test executable
Outcome:	Passed
Duration:	1.22 sec
Failed	None

None

Test case:	[EasyBuild] quick test install of bzip2
Outcome:	Passed
Duration:	9.646 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.372 sec
Tests	2
Failures	0

Tests

rm_execution

Test case:	[dev-tools/hwloc] lstopo runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.21 sec
Failed	None

None

Test case:	[dev-tools/hwloc] hwloc_hello runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.162 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.29 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[HWLOC] Verify HWLOC module is loaded and matches rpm version (gnu14/)
Outcome:	Passed
Duration:	0.165 sec
Failed	None

None

Test case:	[HWLOC] Verify module HWLOC_DIR is defined and exists (gnu14/)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[HWLOC] Verify module HWLOC_LIB is defined and exists (gnu14/)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[HWLOC] Verify dynamic library available in HWLOC_LIB (gnu14/)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[HWLOC] Verify static library is not present in HWLOC_LIB (gnu14/)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[HWLOC] Verify module HWLOC_INC is defined and exists (gnu14/)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[HWLOC] Verify header file is present in HWLOC_INC (gnu14/)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: hipy

Results

Duration	2.256 sec
Tests	1
Failures	0

Tests

hipy

Test case:	[dev-tools/py3-mpi4py] python hello world (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.256 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.058 sec
Tests	3
Failures	0

Tests

test_module

Test case:	[mpi4py] Verify module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[mpi4py] Verify module MPI4PY_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[mpi4py] Verify PYTHONPATH is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test Suite: hipy

Results

Duration	4.399 sec
Tests	1
Failures	1

Tests

hipy

Test case:	[dev-tools/py3-mpi4py] python hello world (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.399 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file hipy, line 35)
  `run_mpi_binary "${_python} helloworld.py" $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.20737
Batch job 27 submitted

Job 27 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- python3 helloworld.py 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346012.905419] [c2:4658 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x55a3f2d25140 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04658] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346012.917005] [c2:4659 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x55f38e1dd280 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04659] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346012.936458] [c2:4660 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x5600e8de31a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04660] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346012.944791] [c2:4661 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x557718094260 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04661] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346012.871363] [c1:3295 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x55a5a892e050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03295] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346012.925446] [c1:3297 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x56217ebfa1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03297] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346012.976541] [c1:3294 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x560d6c34c1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03294] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346012.992765] [c1:3296 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x561324131b70 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03296] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:3294 :0:3294] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x560d6c03fef0)
==== backtrace (tid:   3294) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f91918ef2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f91918ef5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f91918ef854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f91928bb910]
 4  [0x560d6c03fef0]
=================================
[c1:3297 :0:3297] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x56217e990b20)
[c1:3295 :0:3295] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x55a5a86d5960)
[c1:3296 :0:3296] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x561323d19930)
==== backtrace (tid:   3297) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f3f79b8d2bc]
==== backtrace (tid:   3295) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f6401c522bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f6401c525ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f6401c52854]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f3f79b8d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f3f79b8d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f3f7ab59910]
 4  [0x56217e990b20]
=================================
 3  /lib64/libpthread.so.0(+0x16910) [0x7f6402c1e910]
 4  [0x55a5a86d5960]
=================================
==== backtrace (tid:   3296) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f287f27c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f287f27c5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f287f27c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2880248910]
 4  [0x561323d19930]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 3294 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.059 sec
Tests	3
Failures	0

Tests

test_module

Test case:	[mpi4py] Verify module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[mpi4py] Verify module MPI4PY_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[mpi4py] Verify PYTHONPATH is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test Suite: MM

Results

Duration	1.622 sec
Tests	1
Failures	0

Tests

MM

Test case:	[dev-tools/py3-numpy] Numpy Matrix Multiply
Outcome:	Passed
Duration:	1.622 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.075 sec
Tests	4
Failures	0

Tests

test_module

Test case:	[Numpy] Verify NUMPY modules can be loaded and match rpm version (gnu14)
Outcome:	Passed
Duration:	0.017 sec
Failed	None

None

Test case:	[Numpy] Verify module NUMPY_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[Numpy] Verify module NUMPY_BIN is defined and exists
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[Numpy] Verify PYTHONPATH is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test Suite: springs

Results

Duration	0.364 sec
Tests	1
Failures	0

Tests

springs

Test case:	[dev-tools/py3-scipy] Coupled Spring-Mass System (/gnu14/mpich)
Outcome:	Passed
Duration:	0.364 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.059 sec
Tests	3
Failures	0

Tests

test_module

Test case:	[scipy] Verify SCIPY module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[scipy] Verify module SCIPY_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.024 sec
Failed	None

None

Test case:	[scipy] Verify PYTHONPATH is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.017 sec
Failed	None

None

Test Suite: springs

Results

Duration	0.36 sec
Tests	1
Failures	0

Tests

springs

Test case:	[dev-tools/py3-scipy] Coupled Spring-Mass System (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.36 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.059 sec
Tests	3
Failures	0

Tests

test_module

Test case:	[scipy] Verify SCIPY module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[scipy] Verify module SCIPY_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[scipy] Verify PYTHONPATH is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	1.285 sec
Tests	2
Failures	0

Tests

rm_execution

Test case:	[Valgrind] Callgrind execution under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.373 sec
Failed	None

None

Test case:	[Valgrind] Memcheck execution under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.912 sec
Failed	None

None

Test Suite: test_module

Results

Duration	1.46 sec
Tests	8
Failures	0

Tests

test_module

Test case:	[Valgrind] Verify valgrind module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.163 sec
Failed	None

None

Test case:	[Valgrind] Verify module VALGRIND_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[Valgrind] Verify availability of valgrind binary (gnu14)
Outcome:	Passed
Duration:	0.028 sec
Failed	None

None

Test case:	[Valgrind] Verify availability of man page (gnu14)
Outcome:	Passed
Duration:	0.077 sec
Failed	None

None

Test case:	[Valgrind] Verify module VALGRIND_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[Valgrind] Verify header file is present in VALGRIND_INC (gnu14)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[Valgrind] Callgrind compile/test (gnu14)
Outcome:	Passed
Duration:	0.329 sec
Failed	None

None

Test case:	[Valgrind] Memcheck compile/test (gnu14)
Outcome:	Passed
Duration:	0.805 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	12.493 sec
Tests	1
Failures	0

Tests

rm_execution

Test case:	[R] Running Rscript bench.R under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	12.493 sec
Failed	None

None

Test Suite: test_module

Results

Duration	12.741 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[R] Verify R module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.221 sec
Failed	None

None

Test case:	[R] Verify R_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[R] Verify availability of R executable -> R (gnu14)
Outcome:	Passed
Duration:	0.028 sec
Failed	None

None

Test case:	[R] Verify availability of R executable -> Rscript (gnu14)
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test case:	[R] Verify module R_SHARE is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[R] Running bench.R test
Outcome:	Passed
Duration:	11.906 sec
Failed	None

None

Test case:	[R] Verify ability to compile C code for R and execute
Outcome:	Passed
Duration:	0.512 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	4.523 sec
Tests	2
Failures	0

Tests

rm_execution

Test case:	[libs/ADIOS2] MPI C binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.263 sec
Failed	None

None

Test case:	[libs/ADIOS2] MPI F90 binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.26 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.327 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/ADIOS2] Verify ADIOS2 module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.202 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify module ADIOS2_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify module ADIOS2_BIN is defined and exists
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify module ADIOS2_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify (dynamic) library available in ADIOS2_LIB (gnu14)
Outcome:	Passed
Duration:	0.026 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify module ADIOS2_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify header file is present in ADIOS2_INC (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	9.879 sec
Tests	2
Failures	2

Tests

rm_execution

Test case:	[libs/ADIOS2] MPI C binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.409 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 24)
  `run_mpi_binary ./${binary} "" $NODES $TASKS ' failed
job script = /tmp/job.ohpc-test.6637
Batch job 33 submitted

Job 33 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./arrays_write  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346069.016687] [c2:4838 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26f0450 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729346069.016690] [c2:4837 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xbc2600 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729346069.016842] [c2:4836 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xeaa430 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04838] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:04837] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:04836] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346069.041936] [c2:4835 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb734e0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04835] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346069.189947] [c1:3531 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x20c2010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03531] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346069.206913] [c1:3533 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x187ccb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03533] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346069.211174] [c1:3534 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1524cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03534] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346069.244847] [c1:3532 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1786010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03532] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:3532 :0:3532] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x16e9890)
==== backtrace (tid:   3532) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f9a1997d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f9a1997d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f9a1997d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f9a1a885910]
 4  [0x16e9890]
=================================
[c1:3531 :0:3531] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1e906c0)
[c1:3533 :0:3533] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x12b0f90)
[c1:3534 :0:3534] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xf58f90)
==== backtrace (tid:   3533) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f488b37d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f488b37d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f488b37d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f488c285910]
 4  [0x12b0f90]
=================================
==== backtrace (tid:   3531) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2d180552bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2d180555ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2d18055854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2d18f5d910]
 4  [0x1e906c0]
=================================
==== backtrace (tid:   3534) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5ca4d4f2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5ca4d4f5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5ca4d4f854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5ca5c57910]
 4  [0xf58f90]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 3532 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/ADIOS2] MPI F90 binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.47 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 34)
  `run_mpi_binary ./${binary} "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.1270
Batch job 34 submitted

Job 34 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./scalars_write  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346073.868700] [c2:4879 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x19c9530 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346073.872405] [c2:4880 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2479530 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04880] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346073.875348] [c2:4881 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e24530 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04881] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346073.883808] [c2:4882 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x24f0e20 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:04879] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:04882] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346074.156126] [c1:3625 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x13bc010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346074.168637] [c1:3623 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f73010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03623] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:03625] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346074.192443] [c1:3624 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1872010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03624] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346074.219788] [c1:3626 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x23e01a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:03626] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7effd5fde862 in ???
#1  0x7effd5fdd8f5 in ???
#2  0x7effd5342dbf in ???
#3  0x2178c80 in ???
#0  0x7f217b0b2862 in ???
#1  0x7f217b0b18f5 in ???
#2  0x7f217a416dbf in ???
#3  0x1301a00 in ???
#0  0x7f35458db862 in ???
#1  0x7f35458da8f5 in ???
#2  0x7f3544c3fdbf in ???
#3  0x1e9af40 in ???
#0  0x7fb758ec9862 in ???
#1  0x7fb758ec88f5 in ???
#2  0x7fb75822ddbf in ???
#3  0x160ae80 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 3 with PID 3626 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.34 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/ADIOS2] Verify ADIOS2 module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.21 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify module ADIOS2_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify module ADIOS2_BIN is defined and exists
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify module ADIOS2_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify (dynamic) library available in ADIOS2_LIB (gnu14)
Outcome:	Passed
Duration:	0.026 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify module ADIOS2_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/ADIOS2] Verify header file is present in ADIOS2_INC (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.49 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[Boost/Accumulators] min-test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.144 sec
Failed	None

None

Test case:	[Boost/Accumulators] max-test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.113 sec
Failed	None

None

Test case:	[Boost/Accumulators] skewness-test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.121 sec
Failed	None

None

Test case:	[Boost/Accumulators] variance-test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.112 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.485 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[Boost/Accumulators] min-test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.145 sec
Failed	None

None

Test case:	[Boost/Accumulators] max-test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.113 sec
Failed	None

None

Test case:	[Boost/Accumulators] skewness-test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.118 sec
Failed	None

None

Test case:	[Boost/Accumulators] variance-test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.109 sec
Failed	None

None

Test Suite: test_module

Results

Duration	3.168 sec
Tests	8
Failures	0

Tests

test_module

Test case:	[BOOST] Verify BOOST module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.224 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Verify dynamic library available in BOOST_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[BOOST] Verify static library is not present in BOOST_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[BOOST] Verify header file is present in BOOST_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Test interactive build/exec of f_test.cpp (gnu14/mpich)
Outcome:	Passed
Duration:	2.82 sec
Failed	None

None

Test Suite: test_module

Results

Duration	2.913 sec
Tests	8
Failures	0

Tests

test_module

Test case:	[BOOST] Verify BOOST module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.216 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Verify dynamic library available in BOOST_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[BOOST] Verify static library is not present in BOOST_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[BOOST] Verify header file is present in BOOST_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[BOOST] Test interactive build/exec of f_test.cpp (gnu14/openmpi5)
Outcome:	Passed
Duration:	2.57 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.463 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[Boost/Multi Array] access-test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.137 sec
Failed	None

None

Test case:	[Boost/Multi Array] iterators-test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.11 sec
Failed	None

None

Test case:	[Boost/Multi Array] resize-test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.11 sec
Failed	None

None

Test case:	[Boost/Multi Array] idxgen1-test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.106 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.517 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[Boost/Multi Array] access-test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.138 sec
Failed	None

None

Test case:	[Boost/Multi Array] iterators-test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.113 sec
Failed	None

None

Test case:	[Boost/Multi Array] resize-test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.134 sec
Failed	None

None

Test case:	[Boost/Multi Array] idxgen1-test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.132 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	1.328 sec
Tests	1
Failures	0

Tests

rm_execution

Test case:	[Boost/ublas] bench1_ublas binary under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.328 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	1.326 sec
Tests	1
Failures	0

Tests

rm_execution

Test case:	[Boost/ublas] bench1_ublas binary under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	1.326 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	1.197 sec
Tests	11
Failures	0

Tests

rm_execution

Test case:	[Boost/Program Options] cmdline_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.146 sec
Failed	None

None

Test case:	[Boost/Program Options] exception_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.108 sec
Failed	None

None

Test case:	[Boost/Program Options] options_description_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.132 sec
Failed	None

None

Test case:	[Boost/Program Options] parsers_test on master host(gnu14/openmpi5)
Outcome:	Passed
Duration:	0.028 sec
Failed	None

None

Test case:	[Boost/Program Options] parsers_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.13 sec
Failed	None

None

Test case:	[Boost/Program Options] positional_options_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.133 sec
Failed	None

None

Test case:	[Boost/Program Options] required_test on master host(gnu14/openmpi5)
Outcome:	Passed
Duration:	0.029 sec
Failed	None

None

Test case:	[Boost/Program Options] required_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.133 sec
Failed	None

None

Test case:	[Boost/Program Options] unicode_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.108 sec
Failed	None

None

Test case:	[Boost/Program Options] unrecognized_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.137 sec
Failed	None

None

Test case:	[Boost/Program Options] variable_map_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.113 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	1.109 sec
Tests	11
Failures	0

Tests

rm_execution

Test case:	[Boost/Program Options] cmdline_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.154 sec
Failed	None

None

Test case:	[Boost/Program Options] exception_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.11 sec
Failed	None

None

Test case:	[Boost/Program Options] options_description_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.111 sec
Failed	None

None

Test case:	[Boost/Program Options] parsers_test on master host(gnu14/mpich)
Outcome:	Passed
Duration:	0.027 sec
Failed	None

None

Test case:	[Boost/Program Options] parsers_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.113 sec
Failed	None

None

Test case:	[Boost/Program Options] positional_options_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.106 sec
Failed	None

None

Test case:	[Boost/Program Options] required_test on master host(gnu14/mpich)
Outcome:	Passed
Duration:	0.026 sec
Failed	None

None

Test case:	[Boost/Program Options] required_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.132 sec
Failed	None

None

Test case:	[Boost/Program Options] unicode_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.109 sec
Failed	None

None

Test case:	[Boost/Program Options] unrecognized_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.109 sec
Failed	None

None

Test case:	[Boost/Program Options] variable_map_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.112 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	1.792 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[Boost/Random] test_piecewise_linear under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.138 sec
Failed	None

None

Test case:	[Boost/Random] test_discrete under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.387 sec
Failed	None

None

Test case:	[Boost/Random] test_random_device under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.135 sec
Failed	None

None

Test case:	[Boost/Random] test_random_number_generator under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.132 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	1.75 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[Boost/Random] test_piecewise_linear under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	1.134 sec
Failed	None

None

Test case:	[Boost/Random] test_discrete under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.39 sec
Failed	None

None

Test case:	[Boost/Random] test_random_device under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.114 sec
Failed	None

None

Test case:	[Boost/Random] test_random_number_generator under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.112 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.18 sec
Tests	1
Failures	0

Tests

rm_execution

Test case:	[Boost] regress under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.18 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.791 sec
Tests	3
Failures	0

Tests

rm_execution

Test case:	[Boost] bad_expression_test under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.519 sec
Failed	None

None

Test case:	[Boost] named_subexpressions_test under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.111 sec
Failed	None

None

Test case:	[Boost] recursion_test under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.161 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.837 sec
Tests	3
Failures	0

Tests

rm_execution

Test case:	[Boost] bad_expression_test under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.558 sec
Failed	None

None

Test case:	[Boost] named_subexpressions_test under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.113 sec
Failed	None

None

Test case:	[Boost] recursion_test under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.166 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.334 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[BOOST] Verify BOOST module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.209 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[BOOST] Verify dynamic library available in BOOST_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Verify static library is not present in BOOST_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[BOOST] Verify header file is present in BOOST_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.338 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[BOOST] Verify BOOST module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.216 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Verify dynamic library available in BOOST_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Verify static library is not present in BOOST_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Verify module BOOST_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[BOOST] Verify header file is present in BOOST_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	15.638 sec
Tests	6
Failures	0

Tests

rm_execution

Test case:	[Boost/MPI] all_gather_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.255 sec
Failed	None

None

Test case:	[Boost/MPI] all_reduce_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.25 sec
Failed	None

None

Test case:	[Boost/MPI] all_to_all_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.31 sec
Failed	None

None

Test case:	[Boost/MPI] groups_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.255 sec
Failed	None

None

Test case:	[Boost/MPI] ring_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.312 sec
Failed	None

None

Test case:	[Boost/MPI] pointer_test under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.256 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	31.648 sec
Tests	6
Failures	6

Tests

rm_execution

Test case:	[Boost/MPI] all_gather_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.521 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 21)
  `run_mpi_binary ./$test atest 2 16' failed
job script = /tmp/job.ohpc-test.31048
Batch job 93 submitted

Job 93 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./all_gather_test atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346500.942141] [c2:5266 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2131500 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05266] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346500.946978] [c2:5262 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e754c0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05262] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346500.980399] [c2:5267 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x22165f0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346500.980850] [c2:5264 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x7fc490 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346500.981768] [c2:5268 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25754e0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346500.984539] [c2:5269 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x24965e0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05269] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346500.989944] [c2:5265 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2676500 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05265] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346500.991704] [c2:5263 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc14500 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05263] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05268] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05264] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05267] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346501.197205] [c1:4960 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x22a01a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729346501.197207] [c1:4956 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x822010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:04956] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:04960] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346501.206874] [c1:4959 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16fe1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:04959] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346501.226483] [c1:4958 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xff7010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346501.227101] [c1:4961 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x185a050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:04961] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:04958] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346501.244328] [c1:4963 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1dc7050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346501.257019] [c1:4962 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x120d050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:04963] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:04962] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346501.312407] [c1:4957 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x141e1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:04957] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:4960 :0:4960] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1f75a20)
[c1:4957 :0:4957] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x10f3a20)
[c1:4959 :0:4959] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x12b3a70)
[c1:4956 :0:4956] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x766e10)
[c1:4961 :0:4961] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x15f8df0)
==== backtrace (tid:   4960) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2b24e572bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2b24e575ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2b24e57854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2b25514910]
 4  [0x1f75a20]
=================================
==== backtrace (tid:   4957) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc0cd2c32bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc0cd2c35ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc0cd2c3854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc0cd980910]
 4  [0x10f3a20]
=================================
[c1:4962 :0:4962] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xfabdf0)
[c1:4963 :0:4963] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1b64dc0)
[c1:4958 :0:4958] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xccccc0)
==== backtrace (tid:   4959) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f03b3b802bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f03b3b805ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f03b3b80854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f03b423d910]
 4  [0x12b3a70]
=================================
==== backtrace (tid:   4956) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f0eccd972bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f0eccd975ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f0eccd97854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f0ecd454910]
 4  [0x766e10]
=================================
==== backtrace (tid:   4961) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f760c5082bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f760c5085ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f760c508854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f760cbc5910]
 4  [0x15f8df0]
=================================
==== backtrace (tid:   4962) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fb98cedd2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fb98cedd5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fb98cedd854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fb98d59a910]
 4  [0xfabdf0]
==== backtrace (tid:   4963) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fcdc83142bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fcdc83145ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fcdc8314854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fcdc89d1910]
 4  [0x1b64dc0]
=================================
=================================
==== backtrace (tid:   4958) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fe5ab77a2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fe5ab77a5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fe5ab77a854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fe5abe37910]
 4  [0xccccc0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 4 with PID 4960 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[Boost/MPI] all_reduce_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.452 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 30)
  `run_mpi_binary ./$test atest 2 16' failed
job script = /tmp/job.ohpc-test.7253
Batch job 94 submitted

Job 94 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./all_reduce_test atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346507.868797] [c2:5329 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x9575e0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729346507.868957] [c2:5327 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25504c0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346507.869403] [c2:5331 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b4e4d0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05331] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346507.882459] [c2:5328 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc6a5f0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05327] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346507.890017] [c2:5330 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x130c580 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05330] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05328] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346507.898419] [c2:5332 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xfa3500 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05329] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346507.905327] [c2:5334 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x182d4d0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05332] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346507.918672] [c2:5333 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1713580 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05333] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05334] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346508.144285] [c1:5025 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fec050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729346508.144451] [c1:5024 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1877050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05025] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05024] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346508.187731] [c1:5023 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x13941a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05023] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346508.209777] [c1:5019 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1374010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346508.221428] [c1:5021 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x254c010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05019] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346508.228450] [c1:5022 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25c5050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346508.232490] [c1:5026 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x15561a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05026] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05021] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346508.243200] [c1:5020 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x24511a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05020] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05022] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:5020 :0:5020] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2126a20)
==== backtrace (tid:   5020) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f067b3642bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f067b3645ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f067b364854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f067ba21910]
 4  [0x2126a20]
=================================
[c1:5026 :0:5026] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1247380)
==== backtrace (tid:   5026) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2ce5c122bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2ce5c125ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2ce5c12854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2ce62cf910]
 4  [0x1247380]
=================================
[c1:5019 :0:5019] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x12b8e10)
[c1:5022 :0:5022] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2363df0)
[c1:5023 :0:5023] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1069a20)
[c1:5024 :0:5024] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1615df0)
==== backtrace (tid:   5019) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4c969972bc]
[c1:5025 :0:5025] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1d8adf0)
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4c969975ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4c96997854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f4c97054910]
 4  [0x12b8e10]
=================================
==== backtrace (tid:   5022) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4a26aed2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4a26aed5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4a26aed854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f4a271aa910]
 4  [0x2363df0]
=================================
[c1:5021 :0:5021] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2221cc0)
==== backtrace (tid:   5024) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fdf422662bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fdf422665ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fdf42266854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fdf42923910]
==== backtrace (tid:   5023) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f7535e642bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f7535e645ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f7535e64854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f7536521910]
 4  [0x1069a20]
=================================
 4  [0x1615df0]
=================================
==== backtrace (tid:   5025) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fb172cb52bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fb172cb55ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fb172cb5854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fb173372910]
 4  [0x1d8adf0]
=================================
==== backtrace (tid:   5021) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f9ee8c4b2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f9ee8c4b5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f9ee8c4b854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f9ee9308910]
 4  [0x2221cc0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 5020 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[Boost/MPI] all_to_all_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.388 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 39)
  `run_mpi_binary ./$test atest 2 16' failed
job script = /tmp/job.ohpc-test.28358
Batch job 95 submitted

Job 95 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./all_to_all_test atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346512.852264] [c2:5383 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb44480 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05383] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346512.855818] [c2:5389 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x17d24d0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05389] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346512.877637] [c2:5387 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x13504d0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05387] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346512.913108] [c2:5390 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc6b4b0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346512.913795] [c2:5386 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x22975a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729346512.913961] [c2:5384 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb555c0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346512.915378] [c2:5388 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1db0480 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346512.928521] [c2:5385 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b4c4d0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05386] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05384] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05390] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05388] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05385] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346513.104798] [c1:5082 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b241a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346513.109247] [c1:5084 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x15fc010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346513.110857] [c1:5087 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xbf9010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05084] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05082] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05087] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346513.131816] [c1:5083 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25ae1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05083] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346513.149851] [c1:5085 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x158a1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05085] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346513.168394] [c1:5086 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1238010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346513.173595] [c1:5088 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26b5050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05086] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05088] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346513.213195] [c1:5081 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x23d9010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05081] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:5081 :0:5081] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20cccd0)
==== backtrace (tid:   5081) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f64ffab82bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f64ffab85ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f64ffab8854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f6500175910]
 4  [0x20cccd0]
=================================
[c1:5088 :0:5088] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x224cfb0)
[c1:5086 :0:5086] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x115dc50)
[c1:5087 :0:5087] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xb1ec50)
[c1:5085 :0:5085] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1322e80)
[c1:5083 :0:5083] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x22838e0)
==== backtrace (tid:   5088) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f512f2622bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f512f2625ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f512f262854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f512f91f910]
 4  [0x224cfb0]
=================================
[c1:5084 :0:5084] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1521c50)
[c1:5082 :0:5082] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18bce80)
==== backtrace (tid:   5086) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f21513cf2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f21513cf5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f21513cf854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2151a8c910]
 4  [0x115dc50]
=================================
==== backtrace (tid:   5087) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f19c528c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f19c528c5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f19c528c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f19c5949910]
 4  [0xb1ec50]
=================================
==== backtrace (tid:   5085) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f85719d32bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f85719d35ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f85719d3854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f8572090910]
 4  [0x1322e80]
=================================
==== backtrace (tid:   5083) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f24c39de2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f24c39de5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f24c39de854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f24c409b910]
 4  [0x22838e0]
=================================
==== backtrace (tid:   5084) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2089b9a2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2089b9a5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2089b9a854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f208a257910]
 4  [0x1521c50]
=================================
==== backtrace (tid:   5082) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f173301b2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f173301b5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f173301b854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f17336d8910]
 4  [0x18bce80]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 5081 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[Boost/MPI] groups_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.448 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 48)
  `run_mpi_binary ./$test atest 2 16' failed
job script = /tmp/job.ohpc-test.9007
Batch job 96 submitted

Job 96 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./groups_test atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346517.859516] [c2:5447 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1327580 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05447] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346517.869344] [c2:5444 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e90490 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346517.869861] [c2:5446 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x988440 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05446] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346517.874561] [c2:5443 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1bd6570 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346517.878822] [c2:5440 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x99a460 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05440] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05444] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346517.886662] [c2:5441 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x22fc480 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346517.888352] [c2:5445 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x111e490 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05441] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05443] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05445] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346517.929071] [c2:5442 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x7b5490 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05442] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346518.115147] [c1:5149 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x19c0010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346518.116709] [c1:5146 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x170a1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05149] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346518.122680] [c1:5151 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x12cc010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346518.124654] [c1:5144 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xdcc010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05151] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05146] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05144] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346518.152137] [c1:5145 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1bf91a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05145] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346518.162509] [c1:5148 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x27c61a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346518.163816] [c1:5150 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x23ef010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05148] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05150] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346518.185668] [c1:5147 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xe1f010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05147] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:5147 :0:5147] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x9b1d40)
==== backtrace (tid:   5147) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2a64baa2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2a64baa5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2a64baa854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2a65267910]
 4  [0x9b1d40]
=================================
[c1:5150 :0:5150] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1f81d40)
[c1:5148 :0:5148] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x249ba20)
[c1:5144 :0:5144] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xb66c60)
==== backtrace (tid:   5150) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f75ab38d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f75ab38d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f75ab38d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f75aba4a910]
 4  [0x1f81d40]
=================================
[c1:5146 :0:5146] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x13dfa20)
==== backtrace (tid:   5148) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f214e6fb2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f214e6fb5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f214e6fb854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f214edb8910]
 4  [0x249ba20]
=================================
==== backtrace (tid:   5144) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fbe432c52bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fbe432c55ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fbe432c5854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fbe43982910]
 4  [0xb66c60]
=================================
==== backtrace (tid:   5146) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f60fc9552bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f60fc9555ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f60fc955854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f60fd012910]
[c1:5145 :0:5145] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18cea20)
 4  [0x13dfa20]
=================================
[c1:5149 :0:5149] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1552d40)
==== backtrace (tid:   5145) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4319fe62bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4319fe65ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4319fe6854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f431a6a3910]
 4  [0x18cea20]
=================================
==== backtrace (tid:   5149) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4c4fb182bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4c4fb185ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4c4fb18854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f4c501d5910]
 4  [0x1552d40]
=================================
[c1:5151 :0:5151] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1064e70)
==== backtrace (tid:   5151) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f37b35102bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f37b35105ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f37b3510854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f37b3bcd910]
 4  [0x1064e70]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 3 with PID 5147 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[Boost/MPI] ring_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.391 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 66)
  `run_mpi_binary ./$test atest 2 16' failed
job script = /tmp/job.ohpc-test.17806
Batch job 97 submitted

Job 97 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ring_test atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346522.869562] [c2:5497 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x19b44a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05497] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346522.872151] [c2:5502 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x8524f0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05502] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346522.880313] [c2:5498 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x23a75d0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05498] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346522.883453] [c2:5500 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x147b4f0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05500] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346522.917809] [c2:5499 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x177f5d0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05499] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346522.929593] [c2:5504 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x14c54a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05504] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346522.939416] [c2:5503 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc444f0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729346522.939614] [c2:5501 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x161b4c0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05501] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05503] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346523.105766] [c1:5206 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x118f010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05206] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346523.111703] [c1:5211 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x136e050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346523.119314] [c1:5212 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x24a8050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05212] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346523.122752] [c1:5213 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xd0f1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05211] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346523.135938] [c1:5209 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16c31a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05213] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346523.151235] [c1:5210 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25cb1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05210] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05209] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346523.168507] [c1:5208 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x212b010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05208] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346523.211506] [c1:5207 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc1f1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05207] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:5207 :0:5207] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x8f4a20)
==== backtrace (tid:   5207) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fb71a3722bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fb71a3725ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fb71a372854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fb71aa2f910]
 4  [0x8f4a20]
=================================
[c1:5212 :0:5212] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2246df0)
[c1:5209 :0:5209] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1278a60)
[c1:5210 :0:5210] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x22a0a20)
[c1:5206 :0:5206] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x10d3e10)
[c1:5211 :0:5211] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x110cdf0)
[c1:5208 :0:5208] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1e00cc0)
==== backtrace (tid:   5212) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fb28661d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fb28661d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fb28661d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fb286cda910]
 4  [0x2246df0]
=================================
[c1:5213 :0:5213] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xa00380)
==== backtrace (tid:   5209) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4e98c1c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4e98c1c5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4e98c1c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f4e992d9910]
 4  [0x1278a60]
=================================
==== backtrace (tid:   5210) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5a90e002bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5a90e005ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5a90e00854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5a914bd910]
 4  [0x22a0a20]
=================================
==== backtrace (tid:   5206) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc52686e2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc52686e5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc52686e854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc526f2b910]
 4  [0x10d3e10]
=================================
==== backtrace (tid:   5211) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f9ba4d182bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f9ba4d185ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f9ba4d18854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f9ba53d5910]
 4  [0x110cdf0]
=================================
==== backtrace (tid:   5208) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fcdf57cd2bc]
==== backtrace (tid:   5213) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f51689312bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f51689315ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5168931854]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fcdf57cd5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fcdf57cd854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fcdf5e8a910]
 4  [0x1e00cc0]
=================================
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5168fee910]
 4  [0xa00380]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 5207 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[Boost/MPI] pointer_test under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.448 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 75)
  `run_mpi_binary ./$test atest 2 16' failed
job script = /tmp/job.ohpc-test.9565
Batch job 98 submitted

Job 98 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pointer_test atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346527.847601] [c2:5560 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x23ef570 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05560] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346527.857012] [c2:5559 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x9f2550 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346527.861704] [c2:5558 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f05510 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729346527.861707] [c2:5554 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x23bd630 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05559] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346527.877501] [c2:5555 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xde2550 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05554] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05558] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:05555] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346527.885094] [c2:5557 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ef2550 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05557] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346527.897394] [c2:5556 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1eb5520 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05556] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346527.905052] [c2:5553 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x17ac640 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05553] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346528.130722] [c1:5274 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x27a7cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05274] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346528.147167] [c1:5275 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x12dccb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05275] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346528.152565] [c1:5270 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x29511a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05270] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346528.161758] [c1:5268 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1a75010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05268] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346528.164539] [c1:5272 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xf5b1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05272] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346528.176443] [c1:5269 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x19fb1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346528.193588] [c1:5271 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1373cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05271] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:05269] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346528.220808] [c1:5273 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x182ccb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:05273] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:5273 :0:5273] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x13a3c90)
==== backtrace (tid:   5273) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f156c6cb2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f156c6cb5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f156c6cb854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f156cd88910]
 4  [0x13a3c90]
=================================
[c1:5271 :0:5271] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xeeac90)
[c1:5268 :0:5268] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x19bace0)
[c1:5269 :0:5269] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x15b07b0)
[c1:5275 :0:5275] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xf9dad0)
==== backtrace (tid:   5271) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc7205462bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc7205465ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc720546854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc720c03910]
 4  [0xeeac90]
=================================
[c1:5272 :0:5272] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xb107a0)
[c1:5270 :0:5270] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2626a20)
==== backtrace (tid:   5268) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f51acc872bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f51acc875ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f51acc87854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f51ad344910]
 4  [0x19bace0]
=================================
[c1:5274 :0:5274] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x231ec90)
==== backtrace (tid:   5269) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f1e874412bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f1e874415ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f1e87441854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f1e87afe910]
 4  [0x15b07b0]
=================================
==== backtrace (tid:   5275) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fa026be02bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fa026be05ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fa026be0854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fa02729d910]
 4  [0xf9dad0]
=================================
==== backtrace (tid:   5272) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f709b76c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f709b76c5ec]
==== backtrace (tid:   5270) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f510f1b52bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f510f1b55ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f510f1b5854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f510f872910]
 4  [0x2626a20]
=================================
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f709b76c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f709be29910]
 4  [0xb107a0]
=================================
==== backtrace (tid:   5274) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4282bf12bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4282bf15ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4282bf1854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f42832ae910]
 4  [0x231ec90]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 5 with PID 5273 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.303 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[FFTW] Verify FFTW module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.179 sec
Failed	None

None

Test case:	[FFTW] Verify module FFTW_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[FFTW] Verify module FFTW_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[FFTW] Verify dynamic library available in FFTW_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[FFTW] Verify static library is not present in FFTW_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[FFTW] Verify module FFTW_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[FFTW] Verify header file is present in FFTW_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.315 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[FFTW] Verify FFTW module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.185 sec
Failed	None

None

Test case:	[FFTW] Verify module FFTW_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.024 sec
Failed	None

None

Test case:	[FFTW] Verify module FFTW_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[FFTW] Verify dynamic library available in FFTW_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[FFTW] Verify static library is not present in FFTW_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[FFTW] Verify module FFTW_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[FFTW] Verify header file is present in FFTW_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	3.749 sec
Tests	3
Failures	0

Tests

rm_execution

Test case:	[libs/FFTW] Serial C binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.263 sec
Failed	None

None

Test case:	[libs/FFTW] MPI C binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.236 sec
Failed	None

None

Test case:	[libs/FFTW] Serial Fortran binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.25 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	6.29 sec
Tests	3
Failures	1

Tests

rm_execution

Test case:	[libs/FFTW] Serial C binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.331 sec
Failed	None

None

Test case:	[libs/FFTW] MPI C binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.65 sec
Failed	None

(from function `assert_success' in file ../../../common/test_helper_functions.bash, line 58,
 in test file rm_execution, line 40)
  `assert_success' failed

-- command failed --
status : 1
output (333 lines):
  job script = /tmp/job.ohpc-test.6547
  Batch job 103 submitted

  Job 103 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = mpirun -- ./C_mpi_test 8 (family=openmpi5)
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [c1:5446 :0:5446] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1210310)
  [c1:5444 :0:5444] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xf96a10)
  [c1:5445 :0:5445] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xdef310)
  ==== backtrace (tid:   5446) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f84cd0182bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f84cd0185ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f84cd018854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7f84cd817910]
   4  [0x1210310]
  =================================
  ==== backtrace (tid:   5444) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fb97519a2bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fb97519a5ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fb97519a854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7fb975999910]
   4  [0xf96a10]
  =================================
  ==== backtrace (tid:   5445) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f0d333de2bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f0d333de5ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f0d333de854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7f0d33bdd910]
   4  [0xdef310]
  =================================
  [c1:5443 :0:5443] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x17f8a10)
  ==== backtrace (tid:   5443) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f0c55ec32bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f0c55ec35ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f0c55ec3854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7f0c566c2910]
   4  [0x17f8a10]
  =================================
  --------------------------------------------------------------------------
  prterun noticed that process rank 3 with PID 5446 on node c1 exited on
  signal 11 (Segmentation fault).
  --------------------------------------------------------------------------
--

Test case:	[libs/FFTW] Serial Fortran binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.309 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	37.564 sec
Tests	50
Failures	0

Tests

rm_execution

Test case:	[libs/GSL] run test_gsl_histogram (gnu14)
Outcome:	Passed
Duration:	0.038 sec
Failed	None

None

Test case:	[libs/GSL] run block under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.232 sec
Failed	None

None

Test case:	[libs/GSL] run bspline under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.467 sec
Failed	None

None

Test case:	[libs/GSL] run cblas under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.11 sec
Failed	None

None

Test case:	[libs/GSL] run cdf under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.114 sec
Failed	None

None

Test case:	[libs/GSL] run cheb under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.127 sec
Failed	None

None

Test case:	[libs/GSL] run combination under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.103 sec
Failed	None

None

Test case:	[libs/GSL] run complex under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.11 sec
Failed	None

None

Test case:	[libs/GSL] run const under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.104 sec
Failed	None

None

Test case:	[libs/GSL] run deriv under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.105 sec
Failed	None

None

Test case:	[libs/GSL] run dht under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.159 sec
Failed	None

None

Test case:	[libs/GSL] run diff under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.102 sec
Failed	None

None

Test case:	[libs/GSL] run eigen under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.296 sec
Failed	None

None

Test case:	[libs/GSL] run err under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.127 sec
Failed	None

None

Test case:	[libs/GSL] run fft under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.476 sec
Failed	None

None

Test case:	[libs/GSL] run fit under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.107 sec
Failed	None

None

Test case:	[libs/GSL] run histogram under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.167 sec
Failed	None

None

Test case:	[libs/GSL] run ieee-utils under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.122 sec
Failed	None

None

Test case:	[libs/GSL] run integration under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	2.898 sec
Failed	None

None

Test case:	[libs/GSL] run interpolation under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.127 sec
Failed	None

None

Test case:	[libs/GSL] run linalg under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	1.222 sec
Failed	None

None

Test case:	[libs/GSL] run matrix under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.877 sec
Failed	None

None

Test case:	[libs/GSL] run min under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.129 sec
Failed	None

None

Test case:	[libs/GSL] run monte under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	1.382 sec
Failed	None

None

Test case:	[libs/GSL] run multifit under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	1.77 sec
Failed	None

None

Test case:	[libs/GSL] run multilarge under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.155 sec
Failed	None

None

Test case:	[libs/GSL] run multimin under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.112 sec
Failed	None

None

Test case:	[libs/GSL] run multiroots under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.131 sec
Failed	None

None

Test case:	[libs/GSL] run multiset under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.123 sec
Failed	None

None

Test case:	[libs/GSL] run ntuple under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.117 sec
Failed	None

None

Test case:	[libs/GSL] run ode-initval under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.415 sec
Failed	None

None

Test case:	[libs/GSL] run ode-initval2 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	5.105 sec
Failed	None

None

Test case:	[libs/GSL] run permutation under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.121 sec
Failed	None

None

Test case:	[libs/GSL] run poly under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.105 sec
Failed	None

None

Test case:	[libs/GSL] run qrng under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.108 sec
Failed	None

None

Test case:	[libs/GSL] run randist under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	1.354 sec
Failed	None

None

Test case:	[libs/GSL] run rng under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	3.003 sec
Failed	None

None

Test case:	[libs/GSL] run roots under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.122 sec
Failed	None

None

Test case:	[libs/GSL] run rstat under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	1.459 sec
Failed	None

None

Test case:	[libs/GSL] run siman under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	1.272 sec
Failed	None

None

Test case:	[libs/GSL] run sort under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.207 sec
Failed	None

None

Test case:	[libs/GSL] run spblas under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.304 sec
Failed	None

None

Test case:	[libs/GSL] run specfunc under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	2.231 sec
Failed	None

None

Test case:	[libs/GSL] run splinalg under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.322 sec
Failed	None

None

Test case:	[libs/GSL] run spmatrix under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.129 sec
Failed	None

None

Test case:	[libs/GSL] run statistics under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.107 sec
Failed	None

None

Test case:	[libs/GSL] run sum under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.104 sec
Failed	None

None

Test case:	[libs/GSL] run sys under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.112 sec
Failed	None

None

Test case:	[libs/GSL] run vector under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	8.097 sec
Failed	None

None

Test case:	[libs/GSL] run wavelet under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.778 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.286 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/GSL] Verify GSL module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.163 sec
Failed	None

None

Test case:	[libs/GSL] Verify module GSL_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/GSL] Verify module GSL_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/GSL] Verify dynamic library available in GSL_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/GSL] Verify static library is not present in GSL_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/GSL] Verify module GSL_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/GSL] Verify header file is present in GSL_INC (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	25.632 sec
Tests	12
Failures	0

Tests

rm_execution

Test case:	[libs/HYPRE] 2 PE structured test binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.259 sec
Failed	None

None

Test case:	[libs/HYPRE] 2 PE PCG with SMG preconditioner binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.204 sec
Failed	None

None

Test case:	[libs/HYPRE] 2 PE Semi-Structured PCG binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.317 sec
Failed	None

None

Test case:	[libs/HYPRE] 2 PE Three-part stencil binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.319 sec
Failed	None

None

Test case:	[libs/HYPRE] 2 PE FORTRAN PCG with PFMG preconditioner binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.257 sec
Failed	None

None

Test case:	[libs/HYPRE] 2 PE -Delta u = 1 binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.317 sec
Failed	None

None

Test case:	[libs/HYPRE] 2 PE convection-reaction-diffusion binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	4.378 sec
Failed	None

None

Test case:	[libs/HYPRE] 2 PE FORTRAN 2-D Laplacian binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.263 sec
Failed	None

None

Test case:	[libs/HYPRE] 2 PE Semi-Structured convection binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skipped

Test case:	[libs/HYPRE] 2 PE biharmonic binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skipped

Test case:	[libs/HYPRE] 2 PE C++ Finite Element Interface binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

C++ example depends on non-installed header

Test case:	[libs/HYPRE] 2 PE 2-D Laplacian eigenvalue binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.318 sec
Failed	None

None

Test Suite: test_module

Results

Duration	2.593 sec
Tests	9
Failures	0

Tests

test_module

Test case:	[libs/HYPRE] Verify HYPRE module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.198 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify module HYPRE_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify module HYPRE_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify module HYPRE_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify dynamic library available in HYPRE_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify static library is not present in HYPRE_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify module HYPRE_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify header file is present in HYPRE_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/HYPRE] Sample job (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.253 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	47.097 sec
Tests	12
Failures	9

Tests

rm_execution

Test case:	[libs/HYPRE] 2 PE structured test binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	7.59 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 23)
  `run_mpi_binary ./ex1 $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.9377
Batch job 167 submitted

Job 167 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex1 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346829.855099] [c2:5925 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c7c430 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05925] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346829.910381] [c2:5926 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x24fd550 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05926] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346830.110715] [c1:6742 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xd66010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06742] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346830.119552] [c1:6741 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x242c010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06741] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:6742 :0:6742] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc8dd40)
==== backtrace (tid:   6742) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7ff5203e92bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7ff5203e95ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7ff5203e9854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7ff52108e910]
 4  [0xc8dd40]
=================================
[c1:6741 :0:6741] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2371af0)
==== backtrace (tid:   6741) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f91668ec2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f91668ec5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f91668ec854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f9167591910]
 4  [0x2371af0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 6741 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/HYPRE] 2 PE PCG with SMG preconditioner binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.465 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 32)
  `run_mpi_binary ./ex2 $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.3496
Batch job 168 submitted

Job 168 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex2 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346837.814221] [c2:5971 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f20430 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05971] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346837.861050] [c2:5972 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1cf2550 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:05972] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346838.133795] [c1:6779 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb89010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06779] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346838.144246] [c1:6780 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2340010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06780] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:6780 :0:6780] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2267d40)
==== backtrace (tid:   6780) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc293c9c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc293c9c5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc293c9c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc294941910]
 4  [0x2267d40]
=================================
[c1:6779 :0:6779] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xaceaf0)
==== backtrace (tid:   6779) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fbded2202bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fbded2205ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fbded220854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fbdedec1910]
 4  [0xaceaf0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 6780 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/HYPRE] 2 PE Semi-Structured PCG binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.403 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 41)
  `run_mpi_binary ./ex6 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.7108
Batch job 169 submitted

Job 169 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex6  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346842.849547] [c2:6009 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x13d1430 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06009] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346842.858172] [c2:6010 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x17d8450 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06010] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346843.105907] [c1:6818 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e63010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06818] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346843.124573] [c1:6817 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1cb7010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06817] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:6817 :0:6817] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1bfcaf0)
[c1:6818 :0:6818] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1d8ad40)
==== backtrace (tid:   6817) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2700f812bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2700f815ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2700f81854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2701c26910]
 4  [0x1bfcaf0]
=================================
==== backtrace (tid:   6818) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fcece0862bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fcece0865ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fcece086854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fceced29910]
 4  [0x1d8ad40]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 6818 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/HYPRE] 2 PE Three-part stencil binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.478 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 50)
  `run_mpi_binary ./ex8 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.7663
Batch job 170 submitted

Job 170 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex8  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346847.886705] [c2:6048 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1901550 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06048] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346847.900801] [c2:6047 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x75b430 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06047] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346848.121474] [c1:6856 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x9e5010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06856] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346848.140513] [c1:6855 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc42010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06855] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:6856 :0:6856] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x90cd40)
[c1:6855 :0:6855] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xba5810)
==== backtrace (tid:   6856) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7ff0e1a5a2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7ff0e1a5a5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7ff0e1a5a854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7ff0e26ff910]
 4  [0x90cd40]
=================================
==== backtrace (tid:   6855) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f3d591392bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f3d591395ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f3d59139854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f3d59dde910]
 4  [0xba5810]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 6856 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/HYPRE] 2 PE FORTRAN PCG with PFMG preconditioner binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	3.345 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 59)
  `run_mpi_binary ./ex12f "" 1 2' failed
job script = /tmp/job.ohpc-test.14327
Batch job 171 submitted

Job 171 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex12f  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346852.939433] [c1:6888 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xed3010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06888] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346852.983169] [c1:6887 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xefd050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06887] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7ff8ac2bd862 in ???
#1  0x7ff8ac2bc8f5 in ???
#2  0x7ff8abab9dbf in ???
#3  0xc95f60 in ???
#0  0x7f02cf4fd862 in ???
#1  0x7f02cf4fc8f5 in ???
#2  0x7f02cecf9dbf in ???
#3  0xdf6980 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 6887 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/HYPRE] 2 PE -Delta u = 1 binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.467 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 68)
  `run_mpi_binary ./ex3 "-n 33 -solver 0 -v 1 1" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.17777
Batch job 172 submitted

Job 172 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex3 -n 33 -solver 0 -v 1 1 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346856.856839] [c2:6086 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x13654d0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06086] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346856.869714] [c2:6087 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb61450 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06087] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346857.097199] [c1:6926 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xa2e010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06926] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346857.140279] [c1:6925 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xce71a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06925] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:6925 :0:6925] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xa81bd0)
==== backtrace (tid:   6925) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7efe81e1c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7efe81e1c5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7efe81e1c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7efe82ac1910]
 4  [0xa81bd0]
=================================
[c1:6926 :0:6926] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x991100)
==== backtrace (tid:   6926) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5330f392bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5330f395ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5330f39854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5331bde910]
 4  [0x991100]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 6925 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/HYPRE] 2 PE convection-reaction-diffusion binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.408 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 77)
  `run_mpi_binary ./ex4 "-n 33 -solver 10  -K 3 -B 0 -C 1 -U0 2 -F 4" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.25688
Batch job 173 submitted

Job 173 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex4 -n 33 -solver 10 -K 3 -B 0 -C 1 -U0 2 -F 4 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346861.848794] [c2:6125 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1eb34a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06125] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346861.885964] [c2:6124 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x184a520 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06124] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346862.108883] [c1:6973 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1396050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06973] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346862.126487] [c1:6972 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x19b0050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06972] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:6972 :0:6972] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1754f30)
==== backtrace (tid:   6972) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f1da97a62bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f1da97a65ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f1da97a6854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f1daa449910]
 4  [0x1754f30]
=================================
[c1:6973 :0:6973] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x107b1e0)
==== backtrace (tid:   6973) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f771bcbe2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f771bcbe5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f771bcbe854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f771c961910]
 4  [0x107b1e0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 6972 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/HYPRE] 2 PE FORTRAN 2-D Laplacian binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.467 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 86)
  `run_mpi_binary ./ex5f "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.13273
Batch job 174 submitted

Job 174 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex5f  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346866.862239] [c2:6165 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x251c550 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06165] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346866.882305] [c2:6166 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c02470 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06166] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346867.096073] [c1:7010 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x148f050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07010] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346867.103202] [c1:7011 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x12cecb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07011] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f6424965862 in ???
#1  0x7f64249648f5 in ???
#2  0x7f642415cdbf in ???
#3  0xf8fad0 in ???
#0  0x7f113ca85862 in ???
#1  0x7f113ca848f5 in ???
#2  0x7f113c27edbf in ???
#3  0x122bd40 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 7011 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/HYPRE] 2 PE Semi-Structured convection binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skipped

Test case:	[libs/HYPRE] 2 PE biharmonic binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skipped

Test case:	[libs/HYPRE] 2 PE C++ Finite Element Interface binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

C++ example depends on non-installed header

Test case:	[libs/HYPRE] 2 PE 2-D Laplacian eigenvalue binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.474 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 125)
  `run_mpi_binary ./ex11 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.23937
Batch job 175 submitted

Job 175 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex11  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346872.861505] [c2:6204 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x81b550 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06204] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346872.884768] [c2:6203 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e4e430 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06203] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346873.094826] [c1:7050 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26b5010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07050] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346873.146215] [c1:7051 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f0f010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07051] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:7050 :0:7050] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x25faaf0)
[c1:7051 :0:7051] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1e36d40)
==== backtrace (tid:   7051) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f816358e2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f816358e5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f816358e854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f8164231910]
 4  [0x1e36d40]
=================================
==== backtrace (tid:   7050) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f6c311ba2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f6c311ba5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f6c311ba854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f6c31e5f910]
 4  [0x25faaf0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 7051 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	3.687 sec
Tests	9
Failures	1

Tests

test_module

Test case:	[libs/HYPRE] Verify HYPRE module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.2 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify module HYPRE_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify module HYPRE_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify module HYPRE_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify dynamic library available in HYPRE_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify static library is not present in HYPRE_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify module HYPRE_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/HYPRE] Verify header file is present in HYPRE_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/HYPRE] Sample job (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	3.343 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file test_module, line 131)
  `run_mpi_binary ./ex1 "atest" 1 1' failed
job script = /tmp/job.ohpc-test.19537
Batch job 166 submitted

Job 166 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex1 atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729346826.035021] [c1:6707 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xd30050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:06707] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:6707 :0:6707] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x8d5250)
==== backtrace (tid:   6707) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f78275292bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f78275295ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f7827529854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f78281ce910]
 4  [0x8d5250]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 6707 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_metis

Results

Duration	3.445 sec
Tests	4
Failures	0

Tests

test_metis

Test case:	[libs/Metis] Graph partition (gnu14)
Outcome:	Passed
Duration:	0.338 sec
Failed	None

None

Test case:	[libs/Metis] Fill-reducing ordering (gnu14)
Outcome:	Passed
Duration:	2.685 sec
Failed	None

None

Test case:	[libs/Metis] Mesh to graph conversion (gnu14)
Outcome:	Passed
Duration:	0.06 sec
Failed	None

None

Test case:	[libs/Metis] C API mesh partitioning (slurm/gnu14)
Outcome:	Passed
Duration:	0.362 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.371 sec
Tests	9
Failures	0

Tests

test_module

Test case:	[libs/Metis] Verify METIS module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.192 sec
Failed	None

None

Test case:	[libs/Metis] Verify module METIS_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/Metis] Verify module METIS_BIN is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/Metis] Verify availability of m2gmetis binary (gnu14)
Outcome:	Passed
Duration:	0.041 sec
Failed	None

None

Test case:	[libs/Metis] Verify module METIS_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/Metis] Verify dynamic library available in METIS_LIB (gnu14)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/Metis] Verify static library is not present in METIS_LIB (gnu14)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/Metis] Verify module METIS_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/Metis] Verify header file is present in METIS_INC (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	11.251 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[libs/MFEM] p_laplace MPI binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.278 sec
Failed	None

None

Test case:	[libs/MFEM] p_laplace_perf MPI binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.344 sec
Failed	None

None

Test case:	[libs/MFEM] p_cantilever MPI binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.281 sec
Failed	None

None

Test case:	[libs/MFEM] p_diffusion MPI binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.348 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	20.964 sec
Tests	4
Failures	4

Tests

rm_execution

Test case:	[libs/MFEM] p_laplace MPI binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.562 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 24)
  `run_mpi_binary ./${binary} "-no-vis" $NODES $TASKS ' failed
job script = /tmp/job.ohpc-test.24210
Batch job 180 submitted

Job 180 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./p_laplace -no-vis (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347050.323374] [c2:6382 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1340d00 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06382] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347050.341073] [c2:6381 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xaaede0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729347050.341074] [c2:6380 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ff4de0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347050.349314] [c2:6383 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc03e10 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06383] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:06381] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:06380] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347050.752309] [c1:7321 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc3f1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07321] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347050.769277] [c1:7319 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xad71a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07319] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347050.779710] [c1:7322 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xaa31a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07322] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347050.796361] [c1:7320 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ba91a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07320] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:7320 :0:7320] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1941c90)
[c1:7322 :0:7322] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x778a20)
[c1:7321 :0:7321] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x9d7de0)
[c1:7319 :0:7319] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7ac940)
==== backtrace (tid:   7320) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f1b9d2972bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f1b9d2975ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f1b9d297854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f1b9f6dd910]
 4  [0x1941c90]
=================================
==== backtrace (tid:   7322) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f427ae782bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f427ae785ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f427ae78854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f427d2be910]
 4  [0x778a20]
=================================
==== backtrace (tid:   7321) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fa5d1d672bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fa5d1d675ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fa5d1d67854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fa5d41ad910]
 4  [0x9d7de0]
=================================
==== backtrace (tid:   7319) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f32b4ab32bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f32b4ab35ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f32b4ab3854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f32b6ef9910]
 4  [0x7ac940]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 7320 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/MFEM] p_laplace_perf MPI binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.447 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 34)
  `run_mpi_binary ./${binary} "-no-vis -rs 2" $NODES $TASKS ' failed
job script = /tmp/job.ohpc-test.12426
Batch job 181 submitted

Job 181 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./p_laplace_perf -no-vis -rs 2 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347056.854534] [c2:6445 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x9d2d90 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06445] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347056.876498] [c2:6447 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x215ed60 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06447] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347056.879370] [c2:6446 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x21f3d10 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06446] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347056.906327] [c2:6444 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x9b4cc0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06444] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347057.160202] [c1:7390 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25da010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07390] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347057.164644] [c1:7389 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xa70050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07389] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347057.192160] [c1:7388 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c6d050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07388] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347057.200377] [c1:7387 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ca61a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07387] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:7387 :0:7387] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1a3eec0)
==== backtrace (tid:   7387) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f3c7c3222bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f3c7c3225ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f3c7c322854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f3c7e768910]
 4  [0x1a3eec0]
=================================
[c1:7388 :0:7388] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1966470)
==== backtrace (tid:   7388) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f302823b2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f302823b5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f302823b854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f302a681910]
 4  [0x1966470]
=================================
[c1:7390 :0:7390] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2375490)
[c1:7389 :0:7389] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x80dde0)
==== backtrace (tid:   7390) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fef1a7ab2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fef1a7ab5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fef1a7ab854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fef1cbf1910]
 4  [0x2375490]
=================================
==== backtrace (tid:   7389) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f13131552bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f13131555ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f1313155854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f131559b910]
 4  [0x80dde0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 7387 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/MFEM] p_cantilever MPI binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.509 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 44)
  `run_mpi_binary ./${binary} "-no-vis" $NODES $TASKS ' failed
job script = /tmp/job.ohpc-test.10195
Batch job 182 submitted

Job 182 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./p_cantilever -no-vis (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347061.869428] [c2:6490 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x245bde0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06490] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347061.898835] [c2:6491 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1a38e00 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729347061.899192] [c2:6488 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2250cf0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347061.899534] [c2:6489 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xfe7e10 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06491] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:06488] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:06489] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347062.140837] [c1:7436 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c631a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347062.141652] [c1:7435 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1eb21a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07436] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:07435] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347062.150796] [c1:7433 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x113c1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07433] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347062.201094] [c1:7434 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x11e8cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07434] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:7434 :0:7434] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xea9c10)
==== backtrace (tid:   7434) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc7d03712bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc7d03715ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc7d0371854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc7d27b7910]
 4  [0xea9c10]
=================================
[c1:7435 :0:7435] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1a45940)
[c1:7433 :0:7433] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xe11940)
[c1:7436 :0:7436] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1938a20)
==== backtrace (tid:   7435) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fe456c2d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fe456c2d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fe456c2d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fe459073910]
 4  [0x1a45940]
=================================
==== backtrace (tid:   7436) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fbb364802bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fbb364805ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fbb36480854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fbb388c6910]
 4  [0x1938a20]
=================================
==== backtrace (tid:   7433) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4f79f972bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4f79f975ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4f79f97854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f4f7c3dd910]
 4  [0xe11940]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 7434 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/MFEM] p_diffusion MPI binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.446 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 54)
  `run_mpi_binary ./${binary} "-no-vis" $NODES $TASKS ' failed
job script = /tmp/job.ohpc-test.2467
Batch job 183 submitted

Job 183 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./p_diffusion -no-vis (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347066.865769] [c2:6533 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x9a7de0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06533] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347066.877673] [c2:6532 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x216ae10 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06532] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347066.894295] [c2:6535 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xdaed00 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06535] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347066.905764] [c2:6534 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x208ce00 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06534] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347067.136477] [c1:7481 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x278a050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07481] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347067.145166] [c1:7482 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xf59010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07482] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347067.205618] [c1:7480 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x28f1cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07480] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347067.236899] [c1:7479 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fb41a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07479] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:7479 :0:7479] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1c89a20)
==== backtrace (tid:   7479) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fcc9de032bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fcc9de035ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fcc9de03854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fcca0249910]
 4  [0x1c89a20]
=================================
[c1:7482 :0:7482] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xcf4490)
[c1:7480 :0:7480] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x25b2c10)
[c1:7481 :0:7481] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2527fb0)
==== backtrace (tid:   7482) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc55e19a2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc55e19a5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc55e19a854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc5605e0910]
==== backtrace (tid:   7480) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f70bf9e62bc]
 4  [0xcf4490]
=================================
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f70bf9e65ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f70bf9e6854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f70c1e2c910]
 4  [0x25b2c10]
=================================
==== backtrace (tid:   7481) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f749e6892bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f749e6895ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f749e689854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f74a0acf910]
 4  [0x2527fb0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 7479 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.353 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/MFEM] Verify MFEM module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.223 sec
Failed	None

None

Test case:	[libs/MFEM] Verify module MFEM_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/MFEM] Verify module MFEM_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/MFEM] Verify module MFEM_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/MFEM] Verify (dynamic) library available in MFEM_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/MFEM] Verify module MFEM_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/MFEM] Verify header file is present in MFEM_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.359 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/MFEM] Verify MFEM module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.228 sec
Failed	None

None

Test case:	[libs/MFEM] Verify module MFEM_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/MFEM] Verify module MFEM_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/MFEM] Verify module MFEM_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/MFEM] Verify (dynamic) library available in MFEM_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/MFEM] Verify module MFEM_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/MFEM] Verify header file is present in MFEM_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	27.349 sec
Tests	5
Failures	5

Tests

rm_execution

Test case:	[libs/Mumps] C (double precision) runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.471 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 26)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.12494
Batch job 191 submitted

Job 191 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./C_double null (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347123.957202] [c2:6702 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x21d9440 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06702] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347123.970466] [c2:6703 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ed8460 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06703] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347124.212581] [c1:7771 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x250d010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07771] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347124.244760] [c1:7772 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1189010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07772] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:7771 :0:7771] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x246fe80)
[c1:7772 :0:7772] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x10a2510)
==== backtrace (tid:   7771) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2d097ad2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2d097ad5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2d097ad854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2d0a43d910]
 4  [0x246fe80]
=================================
==== backtrace (tid:   7772) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7ff72a59d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7ff72a59d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7ff72a59d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7ff72b227910]
 4  [0x10a2510]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 7771 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/Mumps] Fortran (single precision) runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.411 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 36)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.3180
Batch job 192 submitted

Job 192 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./F_single null (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347128.845304] [c2:6741 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x12f4460 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06741] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347128.853993] [c2:6740 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x210d440 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06740] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347129.139034] [c1:7809 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x140f010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07809] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347129.163376] [c1:7810 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xd90010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07810] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fb3baf74862 in ???
#1  0x7fb3baf738f5 in ???
#2  0x7fb3bac00dbf in ???
#3  0xca9510 in ???
#0  0x7fbacaba4862 in ???
#1  0x7fbacaba38f5 in ???
#2  0x7fbaca82edbf in ???
#3  0x1371c10 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 7810 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/Mumps] Fortran (double precision) runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.524 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 46)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.196
Batch job 193 submitted

Job 193 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./F_double null (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347133.837564] [c2:6778 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26ca440 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06778] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347133.886233] [c2:6779 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1523460 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06779] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347134.109917] [c1:7847 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xf77010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07847] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347134.112896] [c1:7848 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2671010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07848] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7efe97f04862 in ???
#1  0x7efe97f038f5 in ???
#2  0x7efe97b90dbf in ???
#3  0xed9c10 in ???
#0  0x7f4e6a494862 in ???
#1  0x7f4e6a4938f5 in ???
#2  0x7f4e6a120dbf in ???
#3  0x258a510 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 7847 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/Mumps] Fortran (complex) runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.473 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 56)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.7355
Batch job 194 submitted

Job 194 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./F_complex null (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347140.838879] [c2:6834 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb90460 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06834] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347140.858997] [c2:6833 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2389440 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06833] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347141.118914] [c1:7886 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x7f0010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07886] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347141.132684] [c1:7885 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xe07010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07885] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f0fe6b2e862 in ???
#1  0x7f0fe6b2d8f5 in ???
#2  0x7f0fe67badbf in ???
#3  0xd69c10 in ???
#0  0x7f902c75e862 in ???
#1  0x7f902c75d8f5 in ???
#2  0x7f902c3eadbf in ???
#3  0x709510 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 7885 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/Mumps] Fortran (double complex) runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.47 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 66)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.14285
Batch job 195 submitted

Job 195 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./F_doublecomplex null (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347145.866341] [c2:6872 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1500460 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06872] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347145.886154] [c2:6873 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1271450 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:06873] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347146.142097] [c1:7924 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x24eb010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07924] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347146.170422] [c1:7923 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1109010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:07923] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f1ff3679862 in ???
#1  0x7f1ff36788f5 in ???
#2  0x7f1ff3303dbf in ???
#3  0x106be80 in ???
#0  0x7fd26cf41862 in ???
#1  0x7fd26cf408f5 in ???
#2  0x7fd26cbcddbf in ???
#3  0x2404510 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 7924 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.222 sec
Tests	2
Failures	0

Tests

test_module

Test case:	[libs/MUMPS] Verify MUMPS module is loaded and matches rpm version (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.201 sec
Failed	None

None

Test case:	[libs/MUMPS] Verify MUMPS_DIR is defined and directory exists (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	14.492 sec
Tests	5
Failures	0

Tests

rm_execution

Test case:	[libs/Mumps] C (double precision) runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.262 sec
Failed	None

None

Test case:	[libs/Mumps] Fortran (single precision) runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.26 sec
Failed	None

None

Test case:	[libs/Mumps] Fortran (double precision) runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.267 sec
Failed	None

None

Test case:	[libs/Mumps] Fortran (complex) runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	4.381 sec
Failed	None

None

Test case:	[libs/Mumps] Fortran (double complex) runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.322 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.213 sec
Tests	2
Failures	0

Tests

test_module

Test case:	[libs/MUMPS] Verify MUMPS module is loaded and matches rpm version (gnu14-mpich)
Outcome:	Passed
Duration:	0.193 sec
Failed	None

None

Test case:	[libs/MUMPS] Verify MUMPS_DIR is defined and directory exists (gnu14-mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: test_C_module

Results

Duration	0.391 sec
Tests	9
Failures	0

Tests

test_C_module

Test case:	[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.194 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify availability of nc-config binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.049 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: test_CXX_module

Results

Duration	0.401 sec
Tests	9
Failures	0

Tests

test_CXX_module

Test case:	[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.202 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.049 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: test_Fortran_module

Results

Duration	0.399 sec
Tests	9
Failures	0

Tests

test_Fortran_module

Test case:	[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.199 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.049 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: test_netcdf

Results

Duration	3.92 sec
Tests	7
Failures	0

Tests

test_netcdf

Test case:	[libs/NetCDF] ncdump availability (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.238 sec
Failed	None

None

Test case:	[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.065 sec
Failed	None

None

Test case:	[libs/NetCDF] verify nc4 available for Fortran interface (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.041 sec
Failed	None

None

Test case:	[libs/NetCDF] verify nc4 available for C++ interface (gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

option no longer supported

Test case:	[libs/NetCDF] C write/read (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	1.362 sec
Failed	None

None

Test case:	[libs/NetCDF] Fortran write/read (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	1.128 sec
Failed	None

None

Test case:	[libs/NetCDF] C++ write/read (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	1.086 sec
Failed	None

None

Test Suite: test_C_module

Results

Duration	0.38 sec
Tests	9
Failures	0

Tests

test_C_module

Test case:	[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.188 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify availability of nc-config binary (gnu14)
Outcome:	Passed
Duration:	0.048 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: test_CXX_module

Results

Duration	0.398 sec
Tests	9
Failures	0

Tests

test_CXX_module

Test case:	[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.207 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu14)
Outcome:	Passed
Duration:	0.047 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: test_Fortran_module

Results

Duration	0.376 sec
Tests	9
Failures	0

Tests

test_Fortran_module

Test case:	[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.184 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu14)
Outcome:	Passed
Duration:	0.048 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: test_netcdf

Results

Duration	1.385 sec
Tests	7
Failures	0

Tests

test_netcdf

Test case:	[libs/NetCDF] ncdump availability (gnu14)
Outcome:	Passed
Duration:	0.063 sec
Failed	None

None

Test case:	[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu14)
Outcome:	Passed
Duration:	0.062 sec
Failed	None

None

Test case:	[libs/NetCDF] verify nc4 available for Fortran interface (gnu14)
Outcome:	Passed
Duration:	0.038 sec
Failed	None

None

Test case:	[libs/NetCDF] verify nc4 available for C++ interface (gnu14)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

option no longer supported

Test case:	[libs/NetCDF] C write/read (slurm/gnu14)
Outcome:	Passed
Duration:	0.471 sec
Failed	None

None

Test case:	[libs/NetCDF] Fortran write/read (slurm/gnu14)
Outcome:	Passed
Duration:	0.378 sec
Failed	None

None

Test case:	[libs/NetCDF] C++ write/read (slurm/gnu14)
Outcome:	Passed
Duration:	0.373 sec
Failed	None

None

Test Suite: test_C_module

Results

Duration	0.383 sec
Tests	9
Failures	0

Tests

test_C_module

Test case:	[libs/NetCDF] Verify NETCDF module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.186 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify availability of nc-config binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.049 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify dynamic library available in NETCDF_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify static library is not present in NETCDF_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify module NETCDF_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF] Verify header file is present in NETCDF_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: test_CXX_module

Results

Duration	0.395 sec
Tests	9
Failures	0

Tests

test_CXX_module

Test case:	[libs/NetCDF-CXX] Verify NETCDF_CXX module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.198 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify availability of ncxx4-config binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.048 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify dynamic library available in NETCDF_CXX_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify static library is not present in NETCDF_CXX_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify module NETCDF_CXX_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-CXX] Verify header file is present in NETCDF_CXX_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: test_Fortran_module

Results

Duration	0.394 sec
Tests	9
Failures	0

Tests

test_Fortran_module

Test case:	[libs/NetCDF-Fortran] Verify NETCDF_FORTRAN module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.199 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify availability of nf-config binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.047 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify dynamic library available in NETCDF_FORTRAN_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify static library is not present in NETCDF_FORTRAN_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify module NETCDF_FORTRAN_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/NetCDF-Fortran] Verify header file is present in NETCDF_FORTRAN_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: test_netcdf

Results

Duration	3.098 sec
Tests	7
Failures	0

Tests

test_netcdf

Test case:	[libs/NetCDF] ncdump availability (gnu14/mpich)
Outcome:	Passed
Duration:	0.178 sec
Failed	None

None

Test case:	[libs/NetCDF] verify nc4/hdf5 available for C interface (gnu14/mpich)
Outcome:	Passed
Duration:	0.061 sec
Failed	None

None

Test case:	[libs/NetCDF] verify nc4 available for Fortran interface (gnu14/mpich)
Outcome:	Passed
Duration:	0.041 sec
Failed	None

None

Test case:	[libs/NetCDF] verify nc4 available for C++ interface (gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

option no longer supported

Test case:	[libs/NetCDF] C write/read (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.089 sec
Failed	None

None

Test case:	[libs/NetCDF] Fortran write/read (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.884 sec
Failed	None

None

Test case:	[libs/NetCDF] C++ write/read (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.845 sec
Failed	None

None

Test Suite: test_pnetcdf

Results

Duration	5.23 sec
Tests	2
Failures	0

Tests

test_pnetcdf

Test case:	[libs/NetCDF] C parallel I/O (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.624 sec
Failed	None

None

Test case:	[libs/NetCDF] Fortran parallel I/O (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.606 sec
Failed	None

None

Test Suite: test_pnetcdf

Results

Duration	10.933 sec
Tests	2
Failures	2

Tests

test_pnetcdf

Test case:	[libs/NetCDF] C parallel I/O (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.468 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file test_pnetcdf, line 28)
  `run_mpi_binary ./C_parallel "atest" 2 4' failed
job script = /tmp/job.ohpc-test.24760
Batch job 218 submitted

Job 218 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./C_parallel atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347237.886759] [c2:7154 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xf91760 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07154] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347237.921143] [c2:7153 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x143c740 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07153] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347238.206726] [c1:8285 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xed1050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08285] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347238.228900] [c1:8286 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2642010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08286] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:8286 :0:8286] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x25a5890)
==== backtrace (tid:   8286) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5ed8b852bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5ed8b855ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5ed8b85854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5ed96ff910]
 4  [0x25a5890]
=================================
[c1:8285 :0:8285] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc6dd40)
==== backtrace (tid:   8285) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f61156112bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f61156115ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f6115611854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f611618b910]
 4  [0xc6dd40]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 8286 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/NetCDF] Fortran parallel I/O (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.465 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file test_pnetcdf, line 64)
  `run_mpi_binary -t "00:02:00" ./F90_parallel "atest" 2 4' failed
job script = /tmp/job.ohpc-test.17260
Batch job 219 submitted

Job 219 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./F90_parallel atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347243.862954] [c2:7191 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xa70dd0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07191] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347243.883543] [c2:7192 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e15e40 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07192] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347244.117533] [c1:8324 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1181050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08324] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347244.146474] [c1:8325 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2699cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08325] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f3ac38e6862 in ???
#1  0x7f3ac38e58f5 in ???
#2  0x7f3ac2a05dbf in ???
#3  0x2356b40 in ???
#0  0x7f09b0ff1862 in ???
#1  0x7f09b0ff08f5 in ???
#2  0x7f09b0110dbf in ???
#3  0xe7a470 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 8325 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: rm_execution

Results

Duration	3.087 sec
Tests	11
Failures	0

Tests

rm_execution

Test case:	[libs/openblas/dblat1] dblat1 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.14 sec
Failed	None

None

Test case:	[libs/openblas/xccblat1] xccblat1 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.109 sec
Failed	None

None

Test case:	[libs/openblas/xzcblat1] xzcblat1 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.133 sec
Failed	None

None

Test case:	[libs/openblas/xscblat2] xscblat2 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.306 sec
Failed	None

None

Test case:	[libs/openblas/xdcblat2] xdcblat2 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.295 sec
Failed	None

None

Test case:	[libs/openblas/xccblat2] xccblat2 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.543 sec
Failed	None

None

Test case:	[libs/openblas/xzcblat2] xzcblat2 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.43 sec
Failed	None

None

Test case:	[libs/openblas/xscblat3] xscblat3 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.16 sec
Failed	None

None

Test case:	[libs/openblas/xdcblat3] xdcblat3 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.204 sec
Failed	None

None

Test case:	[libs/openblas/xccblat3] xccblat3 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.184 sec
Failed	None

None

Test case:	[libs/openblas/xzcblat3] xzcblat3 under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.583 sec
Failed	None

None

Test Suite: test_lapack

Results

Duration	0.354 sec
Tests	1
Failures	0

Tests

test_lapack

Test case:	[libs/OpenBLAS/eigen] run lapack eigen-value solver (gnu14)
Outcome:	Passed
Duration:	0.354 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.251 sec
Tests	5
Failures	0

Tests

test_module

Test case:	[libs/OpenBLAS] Verify OPENBLAS module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.171 sec
Failed	None

None

Test case:	[libs/OpenBLAS] Verify module OPENBLAS_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/OpenBLAS] Verify module OPENBLAS_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/OpenBLAS] Verify dynamic library available in OPENBLAS_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/OpenBLAS] Verify static library is not present in OPENBLAS_LIB (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	2.188 sec
Tests	2
Failures	0

Tests

rm_execution

Test case:	[libs/OpenCoarrays] hello_multiverse binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.188 sec
Failed	None

None

Test case:	[libs/OpenCoarrays] hello_multiverse binary runs under resource manager using cafrun script (slurm/gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

cafrun not supported for all MPI variants

Test Suite: sms_execution

Results

Duration	0.369 sec
Tests	1
Failures	0

Tests

sms_execution

Test case:	[libs/OpenCoarrays] hello_multiverse binary runs on head node (gnu14/mpich)
Outcome:	Passed
Duration:	0.369 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.34 sec
Tests	9
Failures	0

Tests

test_module

Test case:	[libs/opencoarrays] Verify opencoarrays module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.184 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify module OPENCOARRAYS_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify module OPENCOARRAYS_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify dynamic library available in OPENCOARRAYS_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify static library is not present in OPENCOARRAYS_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify module OPENCOARRAYS_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify header file is present in OPENCOARRAYS_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify F90 module is present in OPENCOARRAYS_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify module OPENCOARRAYS_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	5.787 sec
Tests	2
Failures	1

Tests

rm_execution

Test case:	[libs/OpenCoarrays] hello_multiverse binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.787 sec
Failed	None

(from function `assert_success' in file ./common/test_helper_functions.bash, line 58,
 in test file rm_execution, line 39)
  `assert_success' failed

-- command failed --
status : 1
output (93 lines):
  job script = /tmp/job.ohpc-test.9323
  Batch job 238 submitted

  Job 238 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = mpirun -- ./hello  (family=openmpi5)
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [c1:8660 :0:8660] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x244cbb0)
  ==== backtrace (tid:   8660) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc7b5af62bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc7b5af65ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc7b5af6854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7fc7b61a9910]
   4  [0x244cbb0]
  =================================
  --------------------------------------------------------------------------
  prterun noticed that process rank 0 with PID 8660 on node c1 exited on
  signal 11 (Segmentation fault).
  --------------------------------------------------------------------------
--

Test case:	[libs/OpenCoarrays] hello_multiverse binary runs under resource manager using cafrun script (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

cafrun not supported for all MPI variants

Test Suite: sms_execution

Results

Duration	1.481 sec
Tests	1
Failures	0

Tests

sms_execution

Test case:	[libs/OpenCoarrays] hello_multiverse binary runs on head node (gnu14/openmpi5)
Outcome:	Passed
Duration:	1.481 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.346 sec
Tests	9
Failures	0

Tests

test_module

Test case:	[libs/opencoarrays] Verify opencoarrays module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.19 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify module OPENCOARRAYS_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify module OPENCOARRAYS_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify dynamic library available in OPENCOARRAYS_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify static library is not present in OPENCOARRAYS_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify module OPENCOARRAYS_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify header file is present in OPENCOARRAYS_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify F90 module is present in OPENCOARRAYS_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/opencoarrays] Verify module OPENCOARRAYS_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	7.85 sec
Tests	3
Failures	0

Tests

rm_execution

Test case:	[libs/PETSc] MPI C binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.262 sec
Failed	None

None

Test case:	[libs/PETSc] MPI F77 binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.266 sec
Failed	None

None

Test case:	[libs/PETSc] MPI F90 binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.322 sec
Failed	None

None

Test Suite: test_module

Results

Duration	1.555 sec
Tests	9
Failures	0

Tests

test_module

Test case:	[libs/PETSc] Verify PETSC module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.213 sec
Failed	None

None

Test case:	[libs/PETSc] Verify module PETSC_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/PETSc] Verify module PETSC_BIN is defined and exists
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PETSc] Verify module PETSC_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PETSc] Verify dynamic library available in PETSC_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/PETSc] Verify static library is not present in PETSC_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/PETSc] Verify module PETSC_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/PETSc] Verify header file is present in PETSC_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/PETSc] Sample job (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.204 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	15.362 sec
Tests	3
Failures	3

Tests

rm_execution

Test case:	[libs/PETSc] MPI C binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.477 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 25)
  `run_mpi_binary ./C_mpi_test $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.15050
Batch job 244 submitted

Job 244 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./C_mpi_test 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347363.874872] [c2:7466 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc697b0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729347363.875029] [c2:7465 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ebd760 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07465] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347363.881270] [c2:7463 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x124f750 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07463] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347363.884958] [c2:7464 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26d5760 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07464] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:07466] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347364.117633] [c1:8928 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b4c1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08928] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347364.123428] [c1:8929 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2a13cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08929] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347364.169170] [c1:8927 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x24cb1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08927] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347364.172143] [c1:8930 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2636cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08930] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:8927 :0:8927] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x205dee0)
==== backtrace (tid:   8927) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5a109b82bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5a109b85ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5a109b8854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5a1422c910]
 4  [0x205dee0]
=================================
[c1:8930 :0:8930] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x22f3f50)
==== backtrace (tid:   8930) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2cdf6062bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2cdf6065ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2cdf606854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2ce2e7b910]
 4  [0x22f3f50]
=================================
[c1:8929 :0:8929] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x26d0f50)
[c1:8928 :0:8928] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18218e0)
==== backtrace (tid:   8928) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fa31d5da2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fa31d5da5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fa31d5da854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fa320e4f910]
 4  [0x18218e0]
=================================
==== backtrace (tid:   8929) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2f39dfa2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2f39dfa5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2f39dfa854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2f3d66e910]
 4  [0x26d0f50]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 8927 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/PETSc] MPI F77 binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.414 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 34)
  `run_mpi_binary ./F_test $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.6988
Batch job 245 submitted

Job 245 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./F_test 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347368.854268] [c2:7507 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x19f8750 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07507] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347368.887918] [c2:7508 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xa5b760 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729347368.888129] [c2:7510 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f5d7b0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347368.894572] [c2:7509 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1977750 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07509] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:07508] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:07510] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347369.123030] [c1:8976 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26e8cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08976] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347369.167523] [c1:8974 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x14fa1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08974] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347369.174803] [c1:8975 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c5dcb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347369.175725] [c1:8973 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x9f61a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08975] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:08973] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fa1f628b862 in ???
#1  0x7fa1f628a8f5 in ???
#2  0x7fa1f5a80dbf in ???
#3  0x191af50 in ???
#0  0x7f09acbd4862 in ???
#1  0x7f09acbd38f5 in ???
#2  0x7f09ac3c9dbf in ???
#3  0x11cf8e0 in ???
#0  0x7fd72cd2a862 in ???
#1  0x7fd72cd298f5 in ???
#2  0x7fd72c51fdbf in ???
#3  0x23a5f50 in ???
#0  0x7f9115d34862 in ???
#1  0x7f9115d338f5 in ???
#2  0x7f9115529dbf in ???
#3  0x588ee0 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 2 with PID 8975 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/PETSc] MPI F90 binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.471 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 43)
  `run_mpi_binary ./F90_test "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.11923
Batch job 246 submitted

Job 246 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./F90_test  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347373.857832] [c2:7555 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1984760 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07555] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347373.868739] [c2:7554 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x10a47b0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07554] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347373.871803] [c2:7552 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x223c750 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07552] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347373.888952] [c2:7553 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fe7760 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07553] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347374.092303] [c1:9021 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x121acb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09021] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347374.144864] [c1:9020 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb1c1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09020] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347374.170058] [c1:9019 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x29751a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09019] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347374.178190] [c1:9022 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1499cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09022] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f9efae83862 in ???
#1  0x7f9efae828f5 in ???
#2  0x7f9efa678dbf in ???
#3  0x1156f50 in ???
#0  0x7f1a329a7862 in ???
#1  0x7f1a329a68f5 in ???
#2  0x7f1a3219cdbf in ???
#3  0xed7f50 in ???
#0  0x7f5ffef6d862 in ???
#1  0x7f5ffef6c8f5 in ???
#2  0x7f5ffe762dbf in ???
#3  0x270e7c0 in ???
#0  0x7f39b19ab862 in ???
#1  0x7f39b19aa8f5 in ???
#2  0x7f39b11a0dbf in ???
#3  0x7f18e0 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 3 with PID 9022 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	3.704 sec
Tests	9
Failures	1

Tests

test_module

Test case:	[libs/PETSc] Verify PETSC module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.217 sec
Failed	None

None

Test case:	[libs/PETSc] Verify module PETSC_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/PETSc] Verify module PETSC_BIN is defined and exists
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/PETSc] Verify module PETSC_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/PETSc] Verify dynamic library available in PETSC_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PETSc] Verify static library is not present in PETSC_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PETSc] Verify module PETSC_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/PETSc] Verify header file is present in PETSC_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/PETSc] Sample job (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	3.345 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file test_module, line 130)
  `run_mpi_binary ./C_test "atest" 1 1' failed
job script = /tmp/job.ohpc-test.19791
Batch job 243 submitted

Job 243 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./C_test atest (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347359.978118] [c1:8892 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xd40050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:08892] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:8892 :0:8892] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xa39c40)
==== backtrace (tid:   8892) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fa3987c02bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fa3987c05ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fa3987c0854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fa39c034910]
 4  [0xa39c40]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 8892 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.317 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[HDF5] Verify HDF5 module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.194 sec
Failed	None

None

Test case:	[HDF5] Verify module HDF5_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[HDF5] Verify module HDF5_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[HDF5] Verify dynamic library available in HDF5_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[HDF5] Verify static library is not present in HDF5_LIB (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[HDF5] Verify module HDF5_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[HDF5] Verify header file is present in HDF5_INC (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	4.516 sec
Tests	2
Failures	0

Tests

rm_execution

Test case:	[libs/PHDF5] MPI C binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.258 sec
Failed	None

None

Test case:	[libs/PHDF5] Parallel Fortran binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.258 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	9.865 sec
Tests	2
Failures	2

Tests

rm_execution

Test case:	[libs/PHDF5] MPI C binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.407 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 24)
  `run_mpi_binary -t $CMD_TIMEOUT ./C_mpi_test $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.14753
Batch job 249 submitted

Job 249 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./C_mpi_test /mnt/lustre/ohpc-test/phdf_test.out (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347401.862341] [c2:7656 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x21b9cf0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07656] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347401.876779] [c2:7654 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1809c50 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07654] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347401.882502] [c2:7655 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xe6be60 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07655] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347401.900793] [c2:7653 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x19ccbd0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07653] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347402.140078] [c1:9171 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x10bdcb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09171] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347402.177240] [c1:9170 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xbea1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09170] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347402.208109] [c1:9172 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fda050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09172] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347402.211899] [c1:9169 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1d22cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09169] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:9169 :0:9169] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x19e3760)
==== backtrace (tid:   9169) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f1be5f942bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f1be5f945ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f1be5f94854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f1be67b3910]
 4  [0x19e3760]
=================================
[c1:9172 :0:9172] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1d70d10)
==== backtrace (tid:   9172) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f745a93a2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f745a93a5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f745a93a854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f745b159910]
 4  [0x1d70d10]
=================================
[c1:9171 :0:9171] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc52ee0)
[c1:9170 :0:9170] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x8bfa20)
==== backtrace (tid:   9171) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fcc3d9af2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fcc3d9af5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fcc3d9af854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fcc3e1ce910]
 4  [0xc52ee0]
=================================
==== backtrace (tid:   9170) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fd690f902bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fd690f905ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fd690f90854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fd6917af910]
 4  [0x8bfa20]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 9169 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/PHDF5] Parallel Fortran binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.458 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 33)
  `run_mpi_binary -t $CMD_TIMEOUT ./F_mpi_test $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.6520
Batch job 250 submitted

Job 250 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./F_mpi_test /mnt/lustre/ohpc-test/phdf_test.out (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347406.831108] [c2:7700 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x7fe2b0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07700] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347406.844916] [c2:7697 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b54220 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07697] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347406.864818] [c2:7698 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c3f470 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07698] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347406.873485] [c2:7699 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1367340 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07699] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347407.076740] [c1:9217 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x221e010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09217] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347407.135187] [c1:9215 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x13d3010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09215] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347407.169529] [c1:9216 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x227c1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09216] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347407.190101] [c1:9218 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c391a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09218] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fd3705c4862 in ???
#1  0x7fd3705c38f5 in ???
#2  0x7fd37024edbf in ???
#3  0x190ea20 in ???
#0  0x7fe5cd30b862 in ???
#1  0x7fe5cd30a8f5 in ???
#2  0x7fe5ccf95dbf in ???
#3  0x116db30 in ???
#0  0x7fdda6908862 in ???
#1  0x7fdda69078f5 in ???
#2  0x7fdda6592dbf in ???
#3  0x1f51a20 in ???
#0  0x7f677f08e862 in ???
#1  0x7f677f08d8f5 in ???
#2  0x7f677ed18dbf in ???
#3  0x1ffffe0 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 3 with PID 9218 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: rm_execution

Results

Duration	1.503 sec
Tests	3
Failures	0

Tests

rm_execution

Test case:	[libs/PLASMA/C_test] C_test under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.356 sec
Failed	None

None

Test case:	[libs/PLASMA/F_test] F_test under resource manager (slurm/gnu14)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skipped

Test case:	[libs/PLASMA/F90_test] F90_test under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	1.147 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.297 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/PLASMA] Verify plasma module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.173 sec
Failed	None

None

Test case:	[libs/PLASMA] Verify module PLASMA_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/PLASMA] Verify module PLASMA_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PLASMA] Verify dynamic library available in PLASMA_LIB (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PLASMA] Verify static library is not present in PLASMA_LIB (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/PLASMA] Verify module PLASMA_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/PLASMA] Verify header file is present in PLASMA_INC (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.275 sec
Tests	5
Failures	0

Tests

test_module

Test case:	[PNETCDF] Verify PNETCDF module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.193 sec
Failed	None

None

Test case:	[PNETCDF] Verify module PNETCDF_DIR is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[PNETCDF] Verify module PNETCDF_LIB is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[PNETCDF] Verify module PNETCDF_INC is defined and exists (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[PNETCDF] Verify header file is present in PNETCDF_INC (gnu14)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	10.068 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[libs/PNETCDF] Parallel Fortran binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.257 sec
Failed	None

None

Test case:	[libs/PNETCDF] Parallel Fortran binary 2 runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.249 sec
Failed	None

None

Test case:	[libs/PNETCDF] Parallel Fortran binary 3 runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.253 sec
Failed	None

None

Test case:	[libs/PNETCDF] Parallel Fortran binary 4 runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.309 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	15.498 sec
Tests	4
Failures	4

Tests

rm_execution

Test case:	[libs/PNETCDF] Parallel Fortran binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.406 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 22)
  `run_mpi_binary ./f90tst_parallel f90tst_parallel.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.29066
Batch job 257 submitted

Job 257 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./f90tst_parallel f90tst_parallel.nc (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347551.975969] [c1:9479 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1819050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09479] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347551.983181] [c1:9480 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1cee1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09480] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f0983af4862 in ???
#1  0x7f0983af38f5 in ???
#2  0x7f0983780dbf in ???
#3  0x19c3a20 in ???
#0  0x7f3501a3d862 in ???
#1  0x7f3501a3c8f5 in ???
#2  0x7f35016c9dbf in ???
#3  0x150ef40 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 9480 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/PNETCDF] Parallel Fortran binary 2 runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	3.341 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 31)
  `run_mpi_binary ./f90tst_parallel2 f90tst_parallel2.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.19650
Batch job 258 submitted

Job 258 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./f90tst_parallel2 f90tst_parallel2.nc (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347555.922322] [c1:9511 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e6e010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09511] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347555.962645] [c1:9512 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x107d010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09512] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f29e7829862 in ???
#1  0x7f29e78288f5 in ???
#2  0x7f29e74b5dbf in ???
#3  0x1d85f30 in ???
#0  0x7f3da1ecd862 in ???
#1  0x7f3da1ecc8f5 in ???
#2  0x7f3da1b59dbf in ???
#3  0xf94f30 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 9511 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/PNETCDF] Parallel Fortran binary 3 runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.405 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 40)
  `run_mpi_binary ./f90tst_parallel3 f90tst_parallel3.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.3441
Batch job 259 submitted

Job 259 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./f90tst_parallel3 f90tst_parallel3.nc (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347559.950398] [c1:9544 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16071a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09544] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347559.965403] [c1:9543 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x32c1cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09543] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f3f22227862 in ???
#1  0x7f3f222268f5 in ???
#2  0x7f3f21eb3dbf in ???
#3  0x12dca20 in ???
#0  0x7f5a5ac94862 in ???
#1  0x7f5a5ac938f5 in ???
#2  0x7f5a5a920dbf in ???
#3  0x2e56ee0 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 9544 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/PNETCDF] Parallel Fortran binary 4 runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	3.346 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 49)
  `run_mpi_binary ./f90tst_parallel4 f90tst_parallel4.nc $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.2045
Batch job 260 submitted

Job 260 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./f90tst_parallel4 f90tst_parallel4.nc (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347563.935556] [c1:9577 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x15fa1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09577] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347564.001606] [c1:9576 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x30f7cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09576] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f0c0722d862 in ???
#1  0x7f0c0722c8f5 in ???
#2  0x7f0c06eb9dbf in ???
#3  0x2c8cee0 in ???
#0  0x7f6d08649862 in ???
#1  0x7f6d086488f5 in ???
#2  0x7f6d082d5dbf in ???
#3  0x12cfa20 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 9576 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: rm_execution

Results

Duration	7.834 sec
Tests	3
Failures	0

Tests

rm_execution

Test case:	[libs/PTScotch] dgraph_redist binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.258 sec
Failed	None

None

Test case:	[libs/PTScotch] strat_par binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.258 sec
Failed	None

None

Test case:	[libs/PTScotch] dgord binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.318 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.36 sec
Tests	9
Failures	0

Tests

test_module

Test case:	[libs/PTScotch] Verify PTSCOTCH module is loaded and matches rpm version (gnu14-mpich)
Outcome:	Passed
Duration:	0.189 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify PTSCOTCH_DIR is defined and directory exists (gnu14-mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify module PTSCOTCH_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify dynamic library available in PTSCOTCH_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify static library is not present in PTSCOTCH_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify module PTSCOTCH_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify header file is present in PTSCOTCH_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify module PTSCOTCH_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify availability of dgscat binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	15.306 sec
Tests	3
Failures	2

Tests

rm_execution

Test case:	[libs/PTScotch] dgraph_redist binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.522 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 22)
  `run_mpi_binary ./dgraph_redist "bump.grf" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.19295
Batch job 264 submitted

Job 264 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./dgraph_redist bump.grf (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347588.872852] [c2:7816 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x12930e0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07816] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347588.892644] [c2:7815 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x14d1120 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07815] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347589.098052] [c1:9750 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1958cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09750] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347589.164815] [c1:9751 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xd40cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09751] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:9751 :0:9751] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x9fdb40)
==== backtrace (tid:   9751) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f3c340292bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f3c340295ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f3c34029854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f3c346d8910]
 4  [0x9fdb40]
=================================
[c1:9750 :0:9750] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x138d3b0)
==== backtrace (tid:   9750) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f43cee1b2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f43cee1b5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f43cee1b854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f43cf4ce910]
 4  [0x138d3b0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 9751 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/PTScotch] strat_par binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	3.323 sec
Failed	None

None

Test case:	[libs/PTScotch] dgord binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.461 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 40)
  `run_mpi_binary dgord "bump.grf /dev/null -vt" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.7799
Batch job 266 submitted

Job 266 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- dgord bump.grf /dev/null -vt (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347598.841348] [c2:7896 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1267d30 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07896] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347598.852540] [c2:7895 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1244cf0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:07895] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347599.100249] [c1:9820 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b801a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09820] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347599.125991] [c1:9821 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b61050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:09821] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:9821 :0:9821] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1858f60)
==== backtrace (tid:   9821) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fd5c59f02bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fd5c59f05ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fd5c59f0854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fd5c6805910]
 4  [0x1858f60]
=================================
[c1:9820 :0:9820] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x19187c0)
==== backtrace (tid:   9820) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fe957dae2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fe957dae5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fe957dae854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fe958bbf910]
 4  [0x19187c0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 9821 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.368 sec
Tests	9
Failures	0

Tests

test_module

Test case:	[libs/PTScotch] Verify PTSCOTCH module is loaded and matches rpm version (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.193 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify PTSCOTCH_DIR is defined and directory exists (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify module PTSCOTCH_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify dynamic library available in PTSCOTCH_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify static library is not present in PTSCOTCH_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify module PTSCOTCH_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify header file is present in PTSCOTCH_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify module PTSCOTCH_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/PTScotch] Verify availability of dgscat binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	9.032 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.258 sec
Failed	None

None

Test case:	[libs/ScaLAPACK/PDSCAEX] DPCGESV under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.319 sec
Failed	None

None

Test case:	[libs/ScaLAPACK/PSSCAEX] SPCGESV under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.196 sec
Failed	None

None

Test case:	[libs/ScaLAPACK/PZSCAEX] ZPCGESV under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.259 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.256 sec
Tests	5
Failures	0

Tests

test_module

Test case:	[libs/ScaLAPACK] Verify SCALAPACK module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.181 sec
Failed	None

None

Test case:	[libs/ScaLAPACK] Verify module SCALAPACK_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/ScaLAPACK] Verify module SCALAPACK_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/ScaLAPACK] Verify dynamic library available in SCALAPACK_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.017 sec
Failed	None

None

Test case:	[libs/ScaLAPACK] Verify static library is not present in SCALAPACK_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	21.855 sec
Tests	4
Failures	4

Tests

rm_execution

Test case:	[libs/ScaLAPACK/PCSCAEX] CPCGESV under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	7.586 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 30)
  `run_mpi_binary ./pcscaex $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.915
Batch job 272 submitted

Job 272 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pcscaex 0 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347662.870467] [c2:8074 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e53390 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08074] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347662.917718] [c2:8073 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1263330 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729347662.917881] [c2:8072 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x15f8380 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[1729347662.917881] [c2:8071 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xa082f0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08073] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:08072] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:08071] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347663.151416] [c1:10117:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1438cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347663.154067] [c1:10118:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e31050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10117] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:10118] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347663.172212] [c1:10116:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x273c050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10116] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347663.214570] [c1:10119:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x14bc1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10119] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f7dc35d1862 in ???
#1  0x7f7dc35d08f5 in ???
#2  0x7f7dc2dcddbf in ???
#3  0x22d4f60 in ???
#0  0x7fda15c11862 in ???
#1  0x7fda15c108f5 in ???
#2  0x7fda15406dbf in ???
#3  0x1b11ba0 in ???
#0  0x7ffad0241862 in ???
#1  0x7ffad02408f5 in ???
#2  0x7ffacfa3ddbf in ???
#3  0xe6d880 in ???
#0  0x7fcad6599862 in ???
#1  0x7fcad65988f5 in ???
#2  0x7fcad5d95dbf in ???
#3  0x1254c90 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 10117 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/ScaLAPACK/PDSCAEX] DPCGESV under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.401 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 39)
  `run_mpi_binary ./pdscaex $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.18144
Batch job 273 submitted

Job 273 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pdscaex 0 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347669.865861] [c2:8127 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1caa390 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08127] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347669.874636] [c2:8124 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x22272f0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347669.884917] [c2:8125 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xec8360 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347669.885343] [c2:8126 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x192f330 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08125] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:08126] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:08124] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347670.106283] [c1:10162:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x168c050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[c1:10162] pml_ucx.c:313  Error: Failed to create UCP worker
[1729347670.108470] [c1:10164:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x193e050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10164] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347670.137426] [c1:10165:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x12051a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10165] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347670.160212] [c1:10163:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1a67cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10163] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fa5e4281862 in ???
#1  0x7fa5e42808f5 in ???
#2  0x7fa5e3a78dbf in ???
#3  0x149c880 in ???
#0  0x7f54f7111862 in ???
#1  0x7f54f71108f5 in ???
#2  0x7f54f690ddbf in ???
#3  0x1224f60 in ???
#0  0x7fdec5359862 in ???
#1  0x7fdec53588f5 in ???
#2  0x7fdec4b55dbf in ???
#3  0x161eba0 in ???
#0  0x7f66d3a79862 in ???
#1  0x7f66d3a788f5 in ???
#2  0x7f66d3275dbf in ???
#3  0xf9dc90 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 10163 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/ScaLAPACK/PSSCAEX] SPCGESV under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.463 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 48)
  `run_mpi_binary ./psscaex $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.26757
Batch job 274 submitted

Job 274 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./psscaex 0 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347674.821071] [c2:8169 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ec2330 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08169] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347674.833524] [c2:8170 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2863360 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08170] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347674.858437] [c2:8168 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x21a6370 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08168] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347674.870292] [c2:8171 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1712370 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08171] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347675.087560] [c1:10212:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x13a61a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10212] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347675.147890] [c1:10210:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1809cb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10210] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347675.170173] [c1:10209:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1242050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10209] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347675.188939] [c1:10211:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x289c050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10211] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f63d3e31862 in ???
#1  0x7f63d3e308f5 in ???
#2  0x7f63d362ddbf in ???
#3  0x257cba0 in ???
#0  0x7fa53c1e9862 in ???
#1  0x7fa53c1e88f5 in ???
#2  0x7fa53b9e2dbf in ???
#3  0xddaf60 in ???
#0  0x7f4aada11862 in ???
#1  0x7f4aada108f5 in ???
#2  0x7f4aad208dbf in ???
#0  0x7f0da9b91862 in ???
#1  0x7f0da9b908f5 in ???
#3  0x113ec90 in ???
#2  0x7f0da9386dbf in ???
#3  0x123e880 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 2 with PID 10211 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/ScaLAPACK/PZSCAEX] ZPCGESV under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.405 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 57)
  `run_mpi_binary ./pzscaex $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.16668
Batch job 275 submitted

Job 275 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pzscaex 0 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347679.866099] [c2:8213 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1bba360 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08213] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347679.874751] [c2:8212 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26432e0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347679.886074] [c2:8214 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1581340 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08214] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:08212] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347679.901456] [c2:8215 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb6a390 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08215] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347680.091397] [c1:10256:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x17ddcb0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347680.101658] [c1:10257:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2119050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10257] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:10256] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347680.172486] [c1:10255:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2ae2050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10255] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347680.217348] [c1:10258:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x176f1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10258] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f3bae7b9862 in ???
#1  0x7f3bae7b88f5 in ???
#2  0x7f3badfaedbf in ???
#3  0x1444a20 in ???
#0  0x7fce10a59862 in ???
#1  0x7fce10a588f5 in ???
#2  0x7fce10255dbf in ???
#3  0x267af60 in ???
#0  0x7f742f4c9862 in ???
#1  0x7f742f4c88f5 in ???
#2  0x7f742ecbedbf in ???
#0  0x7f40a5651862 in ???
#1  0x7f40a56508f5 in ???
#2  0x7f40a4e48dbf in ???
#3  0x1212880 in ???
#3  0x1df9ba0 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 3 with PID 10258 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.268 sec
Tests	5
Failures	0

Tests

test_module

Test case:	[libs/ScaLAPACK] Verify SCALAPACK module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.188 sec
Failed	None

None

Test case:	[libs/ScaLAPACK] Verify module SCALAPACK_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/ScaLAPACK] Verify module SCALAPACK_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/ScaLAPACK] Verify dynamic library available in SCALAPACK_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/ScaLAPACK] Verify static library is not present in SCALAPACK_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.384 sec
Tests	3
Failures	0

Tests

rm_execution

Test case:	[libs/scotch] graph_map binary runs under resource manager (slurm/gnu14/)
Outcome:	Passed
Duration:	0.148 sec
Failed	None

None

Test case:	[libs/scotch] strat_seq binary runs under resource manager (slurm/gnu14/)
Outcome:	Passed
Duration:	0.107 sec
Failed	None

None

Test case:	[libs/scotch] gout binary runs under resource manager (slurm/gnu14/)
Outcome:	Passed
Duration:	0.129 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.335 sec
Tests	9
Failures	0

Tests

test_module

Test case:	[libs/Scotch] Verify SCOTCH module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.177 sec
Failed	None

None

Test case:	[libs/Scotch] Verify SCOTCH_DIR is defined and directory exists (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/Scotch] Verify module SCOTCH_LIB is defined and exists (gnu14/)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/Scotch] Verify dynamic library available in SCOTCH_LIB (gnu14/)
Outcome:	Passed
Duration:	0.017 sec
Failed	None

None

Test case:	[libs/Scotch] Verify static library is not present in SCOTCH_LIB (gnu14/)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/Scotch] Verify module SCOTCH_INC is defined and exists (gnu14/)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/Scotch] Verify header file is present in SCOTCH_INC (gnu14/)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/Scotch] Verify module SCOTCH_BIN is defined and exists
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/Scotch] Verify availability of gout binary
Outcome:	Passed
Duration:	0.028 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	10.121 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[libs/SLEPc] F90 SVD test binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.268 sec
Failed	None

None

Test case:	[libs/SLEPc] C SVD of the Lauchli matrix binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.325 sec
Failed	None

None

Test case:	[libs/SLEPc] F90 quadratic eigensystem with PEP object binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.204 sec
Failed	None

None

Test case:	[libs/SLEPc] C nonsymmetric eignenproblem binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.324 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.322 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/slepc] Verify slepc module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.202 sec
Failed	None

None

Test case:	[libs/slepc] Verify module SLEPC_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/slepc] Verify module SLEPC_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/slepc] Verify dynamic library available in SLEPC_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/slepc] Verify static library is not present in SLEPC_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/slepc] Verify module SLEPC_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[libs/slepc] Verify header file is present in SLEPC_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	19.776 sec
Tests	4
Failures	4

Tests

rm_execution

Test case:	[libs/SLEPc] F90 SVD test binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.477 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 23)
  `run_mpi_binary ./test4f "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.14418
Batch job 283 submitted

Job 283 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./test4f  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347727.978867] [c2:8326 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb20960 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08326] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347727.983734] [c2:8327 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ea69b0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08327] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347728.211918] [c1:10522:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1690010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10522] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347728.239439] [c1:10523:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b7d050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10523] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f14513cf862 in ???
#1  0x7f14513ce8f5 in ???
#2  0x7f1451059dbf in ???
#3  0x1874e10 in ???
#0  0x7f72af423862 in ???
#1  0x7f72af4228f5 in ???
#2  0x7f72af0addbf in ???
#3  0x15f3760 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 10523 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/SLEPc] C SVD of the Lauchli matrix binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.474 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 32)
  `run_mpi_binary ./ex15 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.28677
Batch job 284 submitted

Job 284 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex15  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347733.840755] [c2:8365 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x162d990 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08365] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347733.866447] [c2:8364 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xe74960 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08364] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347734.095469] [c1:10561:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x219b010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10561] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347734.153729] [c1:10562:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xa55050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10562] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:10562:0:10562] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7f1d40)
==== backtrace (tid:  10562) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5b826302bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5b826305ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5b82630854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5b85ff0910]
 4  [0x7f1d40]
=================================
[c1:10561:0:10561] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20fe760)
==== backtrace (tid:  10561) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7ff2d669e2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7ff2d669e5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7ff2d669e854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7ff2da05f910]
 4  [0x20fe760]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 10562 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/SLEPc] F90 quadratic eigensystem with PEP object binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	3.349 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 41)
  `run_mpi_binary ./ex16f90 " -pep_nev 4 -terse " 1 1' failed
job script = /tmp/job.ohpc-test.6399
Batch job 285 submitted

Job 285 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex16f90 -pep_nev 4 -terse (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347738.974718] [c1:10602:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f001a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10602] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fd6b3860862 in ???
#1  0x7fd6b385f8f5 in ???
#2  0x7fd6b34eadbf in ???
#3  0x1bf1a80 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 10602 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/SLEPc] C nonsymmetric eignenproblem binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.476 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 50)
  `run_mpi_binary ./ex29 "" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.25882
Batch job 286 submitted

Job 286 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./ex29  (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347742.860944] [c2:8403 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x97e9b0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347742.864492] [c2:8402 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x12ca960 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08402] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:08403] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347743.084629] [c1:10636:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x10dc010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10636] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347743.170890] [c1:10637:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x268a050 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:10637] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:10637:0:10637] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2426d40)
==== backtrace (tid:  10637) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f78b64642bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f78b64645ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f78b6464854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f78b9e24910]
 4  [0x2426d40]
=================================
[c1:10636:0:10636] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x103f760)
==== backtrace (tid:  10636) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fd38b2402bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fd38b2405ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fd38b240854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fd38ec00910]
 4  [0x103f760]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 10637 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.336 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/slepc] Verify slepc module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.213 sec
Failed	None

None

Test case:	[libs/slepc] Verify module SLEPC_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/slepc] Verify module SLEPC_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/slepc] Verify dynamic library available in SLEPC_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/slepc] Verify static library is not present in SLEPC_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/slepc] Verify module SLEPC_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/slepc] Verify header file is present in SLEPC_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	0.275 sec
Tests	2
Failures	0

Tests

rm_execution

Test case:	[libs/SuperLU] C test runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.142 sec
Failed	None

None

Test case:	[libs/SuperLU] F77 test runs under resource manager (slurm/gnu14)
Outcome:	Passed
Duration:	0.133 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.198 sec
Tests	2
Failures	0

Tests

test_module

Test case:	[libs/SUPERLU] Verify SUPERLU module is loaded and matches rpm version (gnu14)
Outcome:	Passed
Duration:	0.179 sec
Failed	None

None

Test case:	[libs/SUPERLU] Verify SUPERLU_DIR is defined and directory exists (gnu14)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	24.621 sec
Tests	9
Failures	0

Tests

rm_execution

Test case:	[libs/superLU_dist] PDGSSVX with full (default) options (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.206 sec
Failed	None

None

Test case:	[libs/superLU_dist] pdgssvx_ABglobal with full (default) options (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.264 sec
Failed	None

None

Test case:	[libs/superLU_dist] vary RHS (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.323 sec
Failed	None

None

Test case:	[libs/superLU_dist] vary RHS ABglobal (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.267 sec
Failed	None

None

Test case:	[libs/superLU_dist] reuse permutation vector (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.326 sec
Failed	None

None

Test case:	[libs/superLU_dist] reuse permutation vector ABglobal (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.327 sec
Failed	None

None

Test case:	[libs/superLU_dist] reuse symbolic factorization (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.267 sec
Failed	None

None

Test case:	[libs/superLU_dist] reuse symbolic factorization ABglobal (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.32 sec
Failed	None

None

Test case:	[libs/superLU_dist] multi-grid (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.321 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.322 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/superlu_dist] Verify SUPERLU_DIST module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.202 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify module SUPERLU_DIST_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify module SUPERLU_DIST_LIB is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify dynamic library available in SUPERLU_DIST_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify static library is not present in SUPERLU_DIST_LIB (gnu14/mpich)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify module SUPERLU_DIST_INC is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify header file is present in SUPERLU_DIST_INC (gnu14/mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	50.34 sec
Tests	9
Failures	9

Tests

rm_execution

Test case:	[libs/superLU_dist] PDGSSVX with full (default) options (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.475 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 34)
  `run_mpi_binary ./pddrive "-r 2 -c 2 g20.rua" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.21063
Batch job 298 submitted

Job 298 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pddrive -r 2 -c 2 g20.rua (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347806.889964] [c2:8667 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x23ca510 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08667] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347806.898441] [c2:8668 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1d5b560 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08668] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347807.189463] [c1:11136:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x9e4010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11136] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347807.213037] [c1:11135:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x22f9010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11135] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11135:0:11135] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20f3e70)
==== backtrace (tid:  11135) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f56cb9f72bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f56cb9f75ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f56cb9f7854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f56cc9e6910]
 4  [0x20f3e70]
=================================
[c1:11136:0:11136] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x929b50)
==== backtrace (tid:  11136) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f12b2dc82bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f12b2dc85ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f12b2dc8854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f12b3db0910]
 4  [0x929b50]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 11135 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/superLU_dist] pdgssvx_ABglobal with full (default) options (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.472 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 43)
  `run_mpi_binary ./pddrive_ABglobal "-r 1 -c 1 g20.rua" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.8691
Batch job 299 submitted

Job 299 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pddrive_ABglobal -r 1 -c 1 g20.rua (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347812.858242] [c2:8705 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xe80530 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08705] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347812.889515] [c2:8706 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x248d580 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08706] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347813.161390] [c1:11173:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e68010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11173] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347813.172928] [c1:11174:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x20c1010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11174] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11174:0:11174] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1e7eb90)
==== backtrace (tid:  11174) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc04d7d62bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc04d7d65ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc04d7d6854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc04e7c0910]
 4  [0x1e7eb90]
=================================
[c1:11173:0:11173] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1c00c10)
==== backtrace (tid:  11173) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f8e24aae2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f8e24aae5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f8e24aae854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f8e25a98910]
 4  [0x1c00c10]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 11174 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/superLU_dist] vary RHS (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.475 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 52)
  `run_mpi_binary ./pddrive1 "-r 2 -c 2 g20.rua" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.24625
Batch job 300 submitted

Job 300 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pddrive1 -r 2 -c 2 g20.rua (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347817.854708] [c2:8743 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2136510 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08743] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347817.869141] [c2:8744 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1dd6560 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08744] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347818.115537] [c1:11211:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c31010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11211] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347818.162459] [c1:11212:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1d68010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11212] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11212:0:11212] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1cadb50)
==== backtrace (tid:  11212) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fbea0fc82bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fbea0fc85ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fbea0fc8854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fbea1fb0910]
 4  [0x1cadb50]
=================================
[c1:11211:0:11211] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1922f20)
==== backtrace (tid:  11211) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f21213bd2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f21213bd5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f21213bd854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f21223a8910]
 4  [0x1922f20]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 11212 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/superLU_dist] vary RHS ABglobal (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.475 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 61)
  `run_mpi_binary ./pddrive1_ABglobal "-r 1 -c 1 g20.rua" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.29775
Batch job 301 submitted

Job 301 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pddrive1_ABglobal -r 1 -c 1 g20.rua (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347823.805099] [c2:8782 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x20a4530 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08782] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347823.862051] [c2:8783 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1242580 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08783] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347824.128197] [c1:11249:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c7c010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11249] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347824.165336] [c1:11250:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25b7010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11250] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11250:0:11250] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2374b90)
[c1:11249:0:11249] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1a14c30)
==== backtrace (tid:  11250) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f96a4f172bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f96a4f175ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f96a4f17854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f96a5f00910]
 4  [0x2374b90]
=================================
==== backtrace (tid:  11249) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fdaa08182bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fdaa08185ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fdaa0818854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fdaa1808910]
 4  [0x1a14c30]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 11250 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/superLU_dist] reuse permutation vector (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.414 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 70)
  `run_mpi_binary ./pddrive2 "-r 2 -c 2 g20.rua" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.19126
Batch job 302 submitted

Job 302 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pddrive2 -r 2 -c 2 g20.rua (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347828.845353] [c2:8820 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xdc4510 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08820] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347828.873219] [c2:8821 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2180560 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08821] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347829.117431] [c1:11287:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x262a010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11287] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347829.169379] [c1:11288:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1d5d1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11288] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11288:0:11288] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1a32c50)
==== backtrace (tid:  11288) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f31bd8c52bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f31bd8c55ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f31bd8c5854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f31be8b6910]
 4  [0x1a32c50]
=================================
[c1:11287:0:11287] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x23c2c10)
==== backtrace (tid:  11287) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f62622502bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f62622505ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f6262250854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f6263238910]
 4  [0x23c2c10]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 11288 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/superLU_dist] reuse permutation vector ABglobal (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.478 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 79)
  `run_mpi_binary ./pddrive2_ABglobal "-r 1 -c 1 g20.rua" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.19060
Batch job 303 submitted

Job 303 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pddrive2_ABglobal -r 1 -c 1 g20.rua (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347833.862547] [c2:8859 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x13c5580 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347833.877484] [c2:8858 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x95f530 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08859] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:08858] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347834.161181] [c1:11325:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1450010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11325] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347834.238713] [c1:11326:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2050010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11326] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11326:0:11326] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1e0db90)
==== backtrace (tid:  11326) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f501002d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f501002d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f501002d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5011020910]
 4  [0x1e0db90]
=================================
[c1:11325:0:11325] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x11e8c30)
==== backtrace (tid:  11325) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc7489422bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc7489425ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc748942854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc749930910]
 4  [0x11e8c30]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 11326 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/superLU_dist] reuse symbolic factorization (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.536 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 88)
  `run_mpi_binary ./pddrive3 "-r 2 -c 2 g20.rua" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.19355
Batch job 304 submitted

Job 304 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pddrive3 -r 2 -c 2 g20.rua (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347838.850380] [c2:8897 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x22e3560 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08897] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347838.889746] [c2:8896 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c95510 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08896] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347839.127259] [c1:11363:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2390010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347839.127680] [c1:11364:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x9b3010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11364] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11363] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11364:0:11364] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x8f8b50)
==== backtrace (tid:  11364) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fbad88ba2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fbad88ba5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fbad88ba854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fbad98a8910]
 4  [0x8f8b50]
=================================
[c1:11363:0:11363] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x218ae70)
==== backtrace (tid:  11363) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2c4fc982bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2c4fc985ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2c4fc98854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f2c50c80910]
 4  [0x218ae70]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 11364 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/superLU_dist] reuse symbolic factorization ABglobal (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.539 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 97)
  `run_mpi_binary ./pddrive3_ABglobal "-r 1 -c 1 g20.rua" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.6451
Batch job 305 submitted

Job 305 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pddrive3_ABglobal -r 1 -c 1 g20.rua (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347845.849963] [c2:8943 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x135e580 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08943] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347845.858634] [c2:8942 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1714530 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08942] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347846.156649] [c1:11401:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x142a010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347846.157502] [c1:11402:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x262c1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11401] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11402] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11401:0:11401] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x11c2c30)
==== backtrace (tid:  11401) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f7f91ca02bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f7f91ca05ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f7f91ca0854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f7f92c88910]
 4  [0x11c2c30]
=================================
[c1:11402:0:11402] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2301b30)
==== backtrace (tid:  11402) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f6ed2b632bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f6ed2b635ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f6ed2b63854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f6ed3b90910]
 4  [0x2301b30]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 11401 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/superLU_dist] multi-grid (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.476 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 110)
  `run_mpi_binary ./pddrive4 "g20.rua" $NODES 10' failed
job script = /tmp/job.ohpc-test.26818
Batch job 306 submitted

Job 306 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./pddrive4 g20.rua (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347852.846000] [c2:8990 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f18440 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347852.861489] [c2:8989 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1122500 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08990] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:08989] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347852.881654] [c2:8992 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c76450 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08992] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347852.890423] [c2:8991 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2330450 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08991] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347852.910369] [c2:8993 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x89f450 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:08993] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347853.121865] [c1:11439:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25ba010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11439] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347853.161569] [c1:11442:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2646010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347853.169879] [c1:11440:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x27881a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11440] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347853.177919] [c1:11443:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1570010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11442] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11443] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347853.179945] [c1:11441:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2174010 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11441] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11441:0:11441] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20b9b50)
[c1:11443:0:11443] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x14b5b50)
==== backtrace (tid:  11441) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f6728d382bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f6728d385ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f6728d38854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f6729d20910]
==== backtrace (tid:  11443) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fd9e0fd22bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fd9e0fd25ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fd9e0fd2854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fd9e1fb8910]
 4  [0x14b5b50]
=================================
 4  [0x20b9b50]
=================================
[c1:11440:0:11440] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2522e80)
[c1:11442:0:11442] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x258bb50)
[c1:11439:0:11439] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2377b90)
==== backtrace (tid:  11440) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4615c7f2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4615c7f5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4615c7f854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f4616c68910]
 4  [0x2522e80]
=================================
==== backtrace (tid:  11442) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f28cd4752bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f28cd4755ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f28cd475854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f28ce460910]
 4  [0x258bb50]
=================================
==== backtrace (tid:  11439) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f29e510c2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f29e510c5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f29e510c854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f29e60f6910]
 4  [0x2377b90]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 2 with PID 11441 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.334 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[libs/superlu_dist] Verify SUPERLU_DIST module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.209 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify module SUPERLU_DIST_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify module SUPERLU_DIST_LIB is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify dynamic library available in SUPERLU_DIST_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify static library is not present in SUPERLU_DIST_LIB (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify module SUPERLU_DIST_INC is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[libs/superlu_dist] Verify header file is present in SUPERLU_DIST_INC (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test Suite: build

Results

Duration	4.815 sec
Tests	2
Failures	0

Tests

build

Test case:	[Trilinos] verify availability of Makefile.export.Trilinos (gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

deprecated with newer Trilinos

Test case:	[Trilinos] build Trilinos executables (gnu14/mpich)
Outcome:	Passed
Duration:	4.815 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	17.943 sec
Tests	9
Failures	0

Tests

rm_execution

Test case:	[libs/Trilinos] Kokkos-MemorySpace runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.199 sec
Failed	None

None

Test case:	[libs/Trilinos] Tpetra-InitMPI runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.254 sec
Failed	None

None

Test case:	[libs/Trilinos] Tpetra-DataRedistribution runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.324 sec
Failed	None

None

Test case:	[libs/Trilinos] Epetra-DataRedistribution runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	4.377 sec
Failed	None

None

Test case:	[libs/Trilinos] Epetra-Galeri runs under resource manager (slurm/gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skip Epetra-Galeri in 2.x

Test case:	[libs/Trilinos] Epetra-Ifpack runs under resource manager (slurm/gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skip Epetra-Ifpack in 2.x

Test case:	[libs/Trilinos] Teuchos-ParameterList runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.267 sec
Failed	None

None

Test case:	[libs/Trilinos] Teuchos-LAPACK runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.2 sec
Failed	None

None

Test case:	[libs/Trilinos] Teuchos-BLAS runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.322 sec
Failed	None

None

Test Suite: build

Results

Duration	3.297 sec
Tests	2
Failures	0

Tests

build

Test case:	[Trilinos] verify availability of Makefile.export.Trilinos (gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

deprecated with newer Trilinos

Test case:	[Trilinos] build Trilinos executables (gnu14/openmpi5)
Outcome:	Passed
Duration:	3.297 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	28.665 sec
Tests	9
Failures	3

Tests

rm_execution

Test case:	[libs/Trilinos] Kokkos-MemorySpace runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	3.329 sec
Failed	None

None

Test case:	[libs/Trilinos] Tpetra-InitMPI runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.469 sec
Failed	None

(from function `run_mpi_binary' in file ../../../common/functions, line 414,
 in test file rm_execution, line 36)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.28441
Batch job 315 submitted

Job 315 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./lesson_tpetra_init.exe null (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347900.881671] [c2:9228 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x206d340 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347900.893494] [c2:9227 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16fa380 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09227] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:09228] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347901.115210] [c1:11837:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xa231a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11837] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347901.142703] [c1:11838:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ecf1a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11838] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11838:0:11838] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1c67c90)
==== backtrace (tid:  11838) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fe04d5602bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fe04d5605ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fe04d560854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fe0514a7910]
 4  [0x1c67c90]
=================================
[c1:11837:0:11837] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7bbc90)
==== backtrace (tid:  11837) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7ff4776b82bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7ff4776b85ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7ff4776b8854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7ff47b600910]
 4  [0x7bbc90]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 11838 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/Trilinos] Tpetra-DataRedistribution runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.477 sec
Failed	None

(from function `run_mpi_binary' in file ../../../common/functions, line 414,
 in test file rm_execution, line 46)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.30409
Batch job 316 submitted

Job 316 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./lesson_tpetra_dataredist.exe null (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347906.877873] [c2:9280 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ff6340 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09280] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347906.906356] [c2:9279 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x255c380 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09279] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347907.164459] [c1:11890:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x10501a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11890] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347907.220539] [c1:11889:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xf041a0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c1:11889] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11889:0:11889] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xa978a0)
==== backtrace (tid:  11889) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fe3b24dc2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fe3b24dc5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fe3b24dc854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fe3b6423910]
 4  [0xa978a0]
=================================
[c1:11890:0:11890] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xde8c90)
==== backtrace (tid:  11890) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc884c122bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc884c125ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc884c12854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc888b5a910]
 4  [0xde8c90]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 11889 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/Trilinos] Epetra-DataRedistribution runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.471 sec
Failed	None

(from function `run_mpi_binary' in file ../../../common/functions, line 414,
 in test file rm_execution, line 56)
  `run_mpi_binary $EXE $ARGS $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.8045
Batch job 317 submitted

Job 317 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./lesson_epetra_dataredist.exe null (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347911.858688] [c2:9319 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xdb4380 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09319] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729347911.878802] [c2:9320 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xef8240 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09320] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729347912.131377] [c1:11927:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2713050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:11927] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729347912.153405] [c1:11928:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26e2050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:11928] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:11928:0:11928] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x25c7f30)
==== backtrace (tid:  11928) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f07c2db02bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f07c2db05ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f07c2db0854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f07c6cf8910]
 4  [0x25c7f30]
=================================
[c1:11927:0:11927] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x26a6770)
==== backtrace (tid:  11927) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fbc88f102bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fbc88f105ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fbc88f10854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fbc8ce57910]
 4  [0x26a6770]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 11928 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[libs/Trilinos] Epetra-Galeri runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skip Epetra-Galeri in 2.x

Test case:	[libs/Trilinos] Epetra-Ifpack runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skip Epetra-Ifpack in 2.x

Test case:	[libs/Trilinos] Teuchos-ParameterList runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	3.329 sec
Failed	None

None

Test case:	[libs/Trilinos] Teuchos-LAPACK runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	2.265 sec
Failed	None

None

Test case:	[libs/Trilinos] Teuchos-BLAS runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	3.325 sec
Failed	None

None

Test Suite: lmod_installed

Results

Duration	0.039 sec
Tests	1
Failures	0

Tests

lmod_installed

Test case:	[modules] Check if lmod RPM installed
Outcome:	Passed
Duration:	0.039 sec
Failed	None

None

Test Suite: interactive_commands

Results

Duration	5.393 sec
Tests	8
Failures	0

Tests

interactive_commands

Test case:	[modules] module purge
Outcome:	Passed
Duration:	0.157 sec
Failed	None

None

Test case:	[modules] module list
Outcome:	Passed
Duration:	0.261 sec
Failed	None

None

Test case:	[modules] module help
Outcome:	Passed
Duration:	0.285 sec
Failed	None

None

Test case:	[modules] module load/unload
Outcome:	Passed
Duration:	2.478 sec
Failed	None

None

Test case:	[modules] module whatis
Outcome:	Passed
Duration:	0.285 sec
Failed	None

None

Test case:	[modules] module swap
Outcome:	Passed
Duration:	0.578 sec
Failed	None

None

Test case:	[modules] path updated
Outcome:	Passed
Duration:	0.774 sec
Failed	None

None

Test case:	[modules] module depends-on
Outcome:	Passed
Duration:	0.575 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	1.803 sec
Tests	4
Failures	0

Tests

rm_execution

Test case:	[modules] env variable passes through ()
Outcome:	Passed
Duration:	0.409 sec
Failed	None

None

Test case:	[modules] loaded module passes through ()
Outcome:	Passed
Duration:	0.489 sec
Failed	None

None

Test case:	[modules] module commands available in RMS job ()
Outcome:	Passed
Duration:	0.464 sec
Failed	None

None

Test case:	[modules] module load propagates thru RMS ()
Outcome:	Passed
Duration:	0.441 sec
Failed	None

None

Test Suite: build

Results

Duration	1.703 sec
Tests	3
Failures	0

Tests

build

Test case:	[MPI] build/execute C binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.533 sec
Failed	None

None

Test case:	[MPI] build/execute C++ binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.642 sec
Failed	None

None

Test case:	[MPI] build/execute F90 binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.528 sec
Failed	None

None

Test Suite: man_page_check

Results

Duration	0.12 sec
Tests	1
Failures	0

Tests

man_page_check

Test case:	[MPI] mpicc man page availible (gnu14/mpich)
Outcome:	Passed
Duration:	0.12 sec
Failed	None

None

Test Suite: rm_execution_multi_host

Results

Duration	8.629 sec
Tests	3
Failures	0

Tests

rm_execution_multi_host

Test case:	[MPI] C binary runs on two nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.184 sec
Failed	None

None

Test case:	[MPI] C++ binary runs on two nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	4.263 sec
Failed	None

None

Test case:	[MPI] F90 binary runs on two nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.182 sec
Failed	None

None

Test Suite: rm_execution_single_host

Results

Duration	6.527 sec
Tests	3
Failures	0

Tests

rm_execution_single_host

Test case:	[MPI] C binary runs on single node under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.139 sec
Failed	None

None

Test case:	[MPI] C++ binary runs on single node under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.176 sec
Failed	None

None

Test case:	[MPI] F90 binary runs on single node under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.212 sec
Failed	None

None

Test Suite: version_match

Results

Duration	0.565 sec
Tests	3
Failures	0

Tests

version_match

Test case:	[MPI] MPI module loaded (gnu14/mpich)
Outcome:	Passed
Duration:	0.136 sec
Failed	None

None

Test case:	[MPI] MPI module version available (gnu14/mpich)
Outcome:	Passed
Duration:	0.139 sec
Failed	None

None

Test case:	[MPI] mpicc, mpicxx, and mpif90 versions match module (gnu14/mpich)
Outcome:	Passed
Duration:	0.29 sec
Failed	None

None

Test Suite: build

Results

Duration	3.329 sec
Tests	3
Failures	0

Tests

build

Test case:	[MPI] build/execute C binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	1.624 sec
Failed	None

None

Test case:	[MPI] build/execute C++ binary (gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

C++ not available with openmpi5

Test case:	[MPI] build/execute F90 binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	1.705 sec
Failed	None

None

Test Suite: man_page_check

Results

Duration	0.113 sec
Tests	1
Failures	0

Tests

man_page_check

Test case:	[MPI] mpicc man page availible (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.113 sec
Failed	None

None

Test Suite: rm_execution_multi_host

Results

Duration	11.848 sec
Tests	3
Failures	2

Tests

rm_execution_multi_host

Test case:	[MPI] C binary runs on two nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.401 sec
Failed	None

(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file rm_execution_multi_host, line 24)
  `assert_success' failed

-- command failed --
status : 1
output (277 lines):
  job script = /tmp/job.ohpc-test.13948
  Batch job 329 submitted

  Job 329 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = mpirun -- ./C_test 8 (family=openmpi5)
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [1729347970.839849] [c2:9565 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x7b5c80 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [1729347970.845520] [c2:9564 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x12d7c50 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
  [1729347970.845522] [c2:9563 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2229c30 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
  [c2:09564] pml_ucx.c:313  Error: Failed to create UCP worker
  [c2:09563] pml_ucx.c:313  Error: Failed to create UCP worker
  [c2:09565] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [1729347970.880331] [c2:9562 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b78c10 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
  [c2:09562] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347971.110624] [c1:12415:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c1f050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12415] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347971.149901] [c1:12416:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x230d020 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12416] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347971.204930] [c1:12413:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x15d3050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12413] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347971.231885] [c1:12414:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc3b460 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12414] pml_ucx.c:313  Error: Failed to create UCP worker
  [c1:12414:0:12414] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xab0d80)
  ==== backtrace (tid:  12414) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2b8355e2bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2b8355e5ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2b8355e854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7f2b83c11910]
   4  [0xab0d80]
  =================================
  [c1:12413:0:12413] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x142e880)
  ==== backtrace (tid:  12413) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f55061632bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f55061635ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5506163854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7f5506816910]
   4  [0x142e880]
  =================================
  [c1:12416:0:12416] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x20e8940)
  [c1:12415:0:12415] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x19d1f20)
  ==== backtrace (tid:  12415) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fd82e48e2bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fd82e48e5ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fd82e48e854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7fd82eb41910]
   4  [0x19d1f20]
  =================================
  ==== backtrace (tid:  12416) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f3ed49f12bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f3ed49f15ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f3ed49f1854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7f3ed50a4910]
   4  [0x20e8940]
  =================================
  --------------------------------------------------------------------------
  prterun noticed that process rank 1 with PID 12414 on node c1 exited on
  signal 11 (Segmentation fault).
  --------------------------------------------------------------------------
--

Test case:	[MPI] C++ binary runs on two nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

C++ not available with openmpi5

Test case:	[MPI] F90 binary runs on two nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.447 sec
Failed	None

(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file rm_execution_multi_host, line 45)
  `assert_success' failed

-- command failed --
status : 1
output (277 lines):
  job script = /tmp/job.ohpc-test.7420
  Batch job 330 submitted

  Job 330 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = mpirun -- ./F90_test 8 (family=openmpi5)
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [1729347976.849037] [c2:9608 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fb22d0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
  [c2:09608] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [1729347976.881036] [c2:9606 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ffe270 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [1729347976.882540] [c2:9609 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1529260 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
  [c2:09609] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
  [1729347976.896751] [c2:9607 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x22d0250 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
  [c2:09606] pml_ucx.c:313  Error: Failed to create UCP worker
  [c2:09607] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347977.118524] [c1:12460:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1eaa010 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347977.129830] [c1:12462:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xa76020 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12460] pml_ucx.c:313  Error: Failed to create UCP worker
  [c1:12462] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347977.150030] [c1:12459:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xa90050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12459] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347977.191159] [c1:12461:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x27a5050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12461] pml_ucx.c:313  Error: Failed to create UCP worker

  Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

  Backtrace for this error:

  Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

  Backtrace for this error:

  Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

  Backtrace for this error:

  Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

  Backtrace for this error:
  #0  0x7fc1fa20f862 in ???
  #1  0x7fc1fa20e8f5 in ???
  #2  0x7fc1f9e9bdbf in ???
  #3  0x24d4c60 in ???
  #0  0x7f2ca21f4862 in ???
  #1  0x7f2ca21f38f5 in ???
  #2  0x7f2ca1e80dbf in ???
  #3  0x7b7c10 in ???
  #0  0x7feb2c1ca862 in ???
  #1  0x7feb2c1c98f5 in ???
  #2  0x7feb2be56dbf in ???
  #3  0xa248e0 in ???
  #0  0x7fa28150a862 in ???
  #1  0x7fa2815098f5 in ???
  #2  0x7fa281196dbf in ???
  #3  0x1be04d0 in ???
  --------------------------------------------------------------------------
  prterun noticed that process rank 2 with PID 12461 on node c1 exited on
  signal 11 (Segmentation fault).
  --------------------------------------------------------------------------
--

Test Suite: rm_execution_single_host

Results

Duration	8.181 sec
Tests	3
Failures	2

Tests

rm_execution_single_host

Test case:	[MPI] C binary runs on single node under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.608 sec
Failed	None

(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file rm_execution_single_host, line 24)
  `assert_success' failed

-- command failed --
status : 1
output (67 lines):
  job script = /tmp/job.ohpc-test.15907
  Batch job 327 submitted

  Job 327 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = mpirun -- ./C_test 2 (family=openmpi5)
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347962.906209] [c1:12343:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f8e460 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12343] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347962.995750] [c1:12344:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x25a2460 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12344] pml_ucx.c:313  Error: Failed to create UCP worker
  [c1:12344:0:12344] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2541f60)
  ==== backtrace (tid:  12344) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f9de5a722bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f9de5a725ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f9de5a72854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7f9de6125910]
   4  [0x2541f60]
  =================================
  [c1:12343:0:12343] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1e87e50)
  ==== backtrace (tid:  12343) ====
   0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f6e515082bc]
   1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f6e515085ec]
   2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f6e51508854]
   3  /lib64/libpthread.so.0(+0x16910) [0x7f6e51bbb910]
   4  [0x1e87e50]
  =================================
  --------------------------------------------------------------------------
  prterun noticed that process rank 1 with PID 12344 on node c1 exited on
  signal 11 (Segmentation fault).
  --------------------------------------------------------------------------
--

Test case:	[MPI] C++ binary runs on single node under resource manager (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

C++ not available with openmpi5

Test case:	[MPI] F90 binary runs on single node under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	3.573 sec
Failed	None

(from function `assert_success' in file ../../common/test_helper_functions.bash, line 58,
 in test file rm_execution_single_host, line 45)
  `assert_success' failed

-- command failed --
status : 1
output (67 lines):
  job script = /tmp/job.ohpc-test.31548
  Batch job 328 submitted

  Job 328 failed...
  Reason=NonZeroExitCode

  [prun] Master compute host = c1
  [prun] Resource manager = slurm
  [prun] Launch cmd = mpirun -- ./F90_test 2 (family=openmpi5)
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347966.950313] [c1:12375:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xb60050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12375] pml_ucx.c:313  Error: Failed to create UCP worker
  [create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
  [1729347967.022894] [c1:12376:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fb2060 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
  [c1:12376] pml_ucx.c:313  Error: Failed to create UCP worker

  Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

  Backtrace for this error:

  Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

  Backtrace for this error:
  #0  0x7f25f9085862 in ???
  #1  0x7f25f90848f5 in ???
  #2  0x7f25f8d11dbf in ???
  #3  0x1d787c0 in ???
  #0  0x7f711cfa6862 in ???
  #1  0x7f711cfa58f5 in ???
  #2  0x7f711cc32dbf in ???
  #3  0x89f5e0 in ???
  --------------------------------------------------------------------------
  prterun noticed that process rank 1 with PID 12376 on node c1 exited on
  signal 11 (Segmentation fault).
  --------------------------------------------------------------------------
--

Test Suite: version_match

Results

Duration	0.618 sec
Tests	3
Failures	0

Tests

version_match

Test case:	[MPI] MPI module loaded (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.141 sec
Failed	None

None

Test case:	[MPI] MPI module version available (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.138 sec
Failed	None

None

Test case:	[MPI] mpicc, mpicxx, and mpif90 versions match module (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.339 sec
Failed	None

None

Test Suite: conman

Results

Duration	0.15 sec
Tests	3
Failures	0

Tests

conman

Test case:	[ConMan] Verify conman binary available
Outcome:	Passed
Duration:	0.038 sec
Failed	None

None

Test case:	[ConMan] Verify rpm version matches binary
Outcome:	Passed
Duration:	0.049 sec
Failed	None

None

Test case:	[ConMan] Verify man page availability
Outcome:	Passed
Duration:	0.063 sec
Failed	None

None

Test Suite: warewulf-ipmi

Results

Duration	0.21 sec
Tests	1
Failures	0

Tests

warewulf-ipmi

Test case:	[warewulf-ipmi] ipmitool lanplus protocol
Outcome:	Passed
Duration:	0.21 sec
Failed	None

None

Test Suite: test_module

Results

Duration	1.984 sec
Tests	6
Failures	0

Tests

test_module

Test case:	[Dimemas] Verify dimemas module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.21 sec
Failed	None

None

Test case:	[Dimemas] Verify module DIMEMAS_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.03 sec
Failed	None

None

Test case:	[Dimemas] Verify module DIMEMAS_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.037 sec
Failed	None

None

Test case:	[Dimemas] Verify availability of prv2dim binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.043 sec
Failed	None

None

Test case:	[Dimemas] Verify availability of Dimemas binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.043 sec
Failed	None

None

Test case:	[Dimemas] Run Dimemas simulation (gnu14/mpich)
Outcome:	Passed
Duration:	1.621 sec
Failed	None

None

Test Suite: test_module

Results

Duration	1.998 sec
Tests	6
Failures	0

Tests

test_module

Test case:	[Dimemas] Verify dimemas module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.228 sec
Failed	None

None

Test case:	[Dimemas] Verify module DIMEMAS_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.035 sec
Failed	None

None

Test case:	[Dimemas] Verify module DIMEMAS_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.033 sec
Failed	None

None

Test case:	[Dimemas] Verify availability of prv2dim binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.042 sec
Failed	None

None

Test case:	[Dimemas] Verify availability of Dimemas binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.044 sec
Failed	None

None

Test case:	[Dimemas] Run Dimemas simulation (gnu14/openmpi5)
Outcome:	Passed
Duration:	1.616 sec
Failed	None

None

Test Suite: test_imb_mpi1

Results

Duration	3.314 sec
Tests	1
Failures	0

Tests

test_imb_mpi1

Test case:	[Libs/IMB] run IMB-MPI1 on 2 nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.314 sec
Failed	None

None

Test Suite: test_imb_mpi2

Results

Duration	17.21 sec
Tests	2
Failures	0

Tests

test_imb_mpi2

Test case:	[Libs/IMB] run IMB-EXT on 2 nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.32 sec
Failed	None

None

Test case:	[Libs/IMB] run IMB-IO on 2 nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	13.89 sec
Failed	None

None

Test Suite: test_imb_mpi3

Results

Duration	2.256 sec
Tests	2
Failures	0

Tests

test_imb_mpi3

Test case:	[Libs/IMB] run IMB-NBC on 2 nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skipped

Test case:	[Libs/IMB] run IMB-RMA on 2 nodes under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.256 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.231 sec
Tests	3
Failures	0

Tests

test_module

Test case:	[IMB] Verify IMB module is loaded and matches rpm version (gnu14-mpich)
Outcome:	Passed
Duration:	0.182 sec
Failed	None

None

Test case:	[IMB] Verify IMB_DIR is defined and directory exists (gnu14-mpich)
Outcome:	Passed
Duration:	0.02 sec
Failed	None

None

Test case:	[IMB] Verify executables are present in IMB_DIR/bin (gnu14-mpich)
Outcome:	Passed
Duration:	0.029 sec
Failed	None

None

Test Suite: test_imb_mpi1

Results

Duration	4.398 sec
Tests	1
Failures	1

Tests

test_imb_mpi1

Test case:	[Libs/IMB] run IMB-MPI1 on 2 nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.398 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file test_imb_mpi1, line 38)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.6746
Batch job 335 submitted

Job 335 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- /opt/ohpc/pub/libs/gnu14/openmpi5/imb/2021.3/bin/IMB-MPI1 -off_cache -1 -time 1.5 -npmin 100 -msglog 1:4 PingPong Sendrecv Bcast Allreduce (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729348018.843113] [c2:9760 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2467950 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09760] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729348018.866800] [c2:9757 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1129750 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09757] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729348018.878249] [c2:9759 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x101b4c0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09759] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729348018.905936] [c2:9758 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1712910 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09758] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729348018.926593] [c2:9756 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ae0870 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09756] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729348018.933152] [c2:9761 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2042b10 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09761] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729348018.941348] [c2:9762 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1508b70 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09762] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 22
[1729348018.965149] [c2:9763 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x14d1fc0 failed to create RC QP TX wr:256 sge:5 inl:64 resp:64 RX wr:0 sge:1 resp:64 failed: Invalid argument
[c2:09763] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348019.119356] [c1:12706:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1687470 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:12706] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348019.172033] [c1:12708:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x21b0050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348019.172997] [c1:12710:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x161b470 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348019.173999] [c1:12704:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2217050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:12704] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:12708] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348019.192103] [c1:12705:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2269050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:12705] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:12710] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348019.212388] [c1:12707:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fc0460 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:12707] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348019.271900] [c1:12711:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f67020 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:12711] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348019.289536] [c1:12709:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2293050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:12709] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:12709:0:12709] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2178f30)
==== backtrace (tid:  12709) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f30aba452bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f30aba455ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f30aba45854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f30ac0f6910]
 4  [0x2178f30]
=================================
[c1:12704:0:12704] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x21b8d10)
[c1:12708:0:12708] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2095f30)
[c1:12705:0:12705] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x214ef30)
[c1:12711:0:12711] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1ca4f60)
[c1:12707:0:12707] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1cff8f0)
==== backtrace (tid:  12704) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f9a323612bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f9a323615ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f9a32361854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f9a32a12910]
 4  [0x21b8d10]
=================================
[c1:12706:0:12706] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x138ff30)
==== backtrace (tid:  12708) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f3735b152bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f3735b155ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f3735b15854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f37361c6910]
 4  [0x2095f30]
=================================
[c1:12710:0:12710] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1323f30)
==== backtrace (tid:  12705) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fa40c3f82bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fa40c3f85ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fa40c3f8854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fa40caa9910]
 4  [0x214ef30]
=================================
==== backtrace (tid:  12711) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5222a622bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5222a625ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5222a62854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5223113910]
 4  [0x1ca4f60]
=================================
==== backtrace (tid:  12707) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f83c5fa12bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f83c5fa15ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f83c5fa1854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f83c6652910]
 4  [0x1cff8f0]
=================================
==== backtrace (tid:  12706) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fefc6a5d2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fefc6a5d5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fefc6a5d854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fefc710e910]
 4  [0x138ff30]
=================================
==== backtrace (tid:  12710) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f6b4fac32bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f6b4fac35ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f6b4fac3854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f6b50174910]
 4  [0x1323f30]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 5 with PID 12709 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_imb_mpi2

Results

Duration	9.853 sec
Tests	2
Failures	2

Tests

test_imb_mpi2

Test case:	[Libs/IMB] run IMB-EXT on 2 nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.395 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file test_imb_mpi2, line 41)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.32321
Batch job 336 submitted

Job 336 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- /opt/ohpc/pub/libs/gnu14/openmpi5/imb/2021.3/bin/IMB-EXT -npmin 100 -msglog 1:4 Window (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[c1:12766:0:12766] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x22d5c90)
[c1:12767:0:12767] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1855ca0)
==== backtrace (tid:  12766) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f26cc0272bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f26cc0275ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f26cc027854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f26cc6d8910]
 4  [0x22d5c90]
=================================
==== backtrace (tid:  12767) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f457e43e2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f457e43e5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f457e43e854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f457eaef910]
 4  [0x1855ca0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 12766 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[Libs/IMB] run IMB-IO on 2 nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.458 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file test_imb_mpi2, line 53)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.7193
Batch job 337 submitted

Job 337 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- /opt/ohpc/pub/libs/gnu14/openmpi5/imb/2021.3/bin/IMB-IO -npmin 100 Open_Close (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348028.848752] [c2:9848 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1876640 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:09848] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348028.906027] [c2:9849 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1cbc6e0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:09849] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348029.102360] [c1:12805:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x147d050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:12805] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348029.126421] [c1:12804:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26c7010 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:12804] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:12804:0:12804] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x23ad3b0)
==== backtrace (tid:  12804) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7ff769f6f2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7ff769f6f5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7ff769f6f854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7ff76a620910]
 4  [0x23ad3b0]
=================================
[c1:12805:0:12805] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1412ed0)
==== backtrace (tid:  12805) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fc08e1d32bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fc08e1d35ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fc08e1d3854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fc08e884910]
 4  [0x1412ed0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 12804 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_imb_mpi3

Results

Duration	6.514 sec
Tests	2
Failures	1

Tests

test_imb_mpi3

Test case:	[Libs/IMB] run IMB-NBC on 2 nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skipped

Test case:	[Libs/IMB] run IMB-RMA on 2 nodes under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.514 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file test_imb_mpi3, line 56)
  `run_mpi_binary -t $CMD_TIMEOUT $EXE "$ARGS" $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.31673
Batch job 338 submitted

Job 338 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- /opt/ohpc/pub/libs/gnu14/openmpi5/imb/2021.3/bin/IMB-RMA -npmin 100 -msglog 1:4 Unidir_put Unidir_get (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[c1:12842:0:12842] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1cf0e40)
==== backtrace (tid:  12842) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f163c3212bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f163c3215ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f163c321854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f163c9d2910]
 4  [0x1cf0e40]
=================================
[c1:12843:0:12843] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1983a60)
[c1:12845:0:12845] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1521a40)
[c1:12844:0:12844] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xfccca0)
==== backtrace (tid:  12843) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f2278c362bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f2278c365ec]
==== backtrace (tid:  12845) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5992a382bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5992a385ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5992a38854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f59930e9910]
 4  [0x1521a40]
=================================
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f2278c36854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f22792e7910]
 4  [0x1983a60]
=================================
==== backtrace (tid:  12844) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fd385f902bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fd385f905ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fd385f90854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fd386641910]
 4  [0xfccca0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 12842 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.232 sec
Tests	3
Failures	0

Tests

test_module

Test case:	[IMB] Verify IMB module is loaded and matches rpm version (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.187 sec
Failed	None

None

Test case:	[IMB] Verify IMB_DIR is defined and directory exists (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[IMB] Verify executables are present in IMB_DIR/bin (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.026 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	11.925 sec
Tests	2
Failures	0

Tests

rm_execution

Test case:	[OMB] run osu_bw on 2 nodes under resource manager (/gnu14/mpich)
Outcome:	Passed
Duration:	2.256 sec
Failed	None

None

Test case:	[OMB] run osu_latency on 2 nodes under resource manager (/gnu14/mpich)
Outcome:	Passed
Duration:	9.669 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.347 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[OMB] Verify OMB module is loaded and matches rpm version (gnu14-mpich)
Outcome:	Passed
Duration:	0.182 sec
Failed	None

None

Test case:	[OMB] Verify OMB_DIR is defined and directory exists (gnu14-mpich)
Outcome:	Passed
Duration:	0.018 sec
Failed	None

None

Test case:	[OMB] Verify osu_bw binary is available (gnu14-mpich)
Outcome:	Passed
Duration:	0.029 sec
Failed	None

None

Test case:	[OMB] Verify osu_latency binary is available (gnu14-mpich)
Outcome:	Passed
Duration:	0.029 sec
Failed	None

None

Test case:	[OMB] Verify osu_allgather binary is available (gnu14-mpich)
Outcome:	Passed
Duration:	0.03 sec
Failed	None

None

Test case:	[OMB] Verify osu_get_bw binary is available (gnu14-mpich)
Outcome:	Passed
Duration:	0.03 sec
Failed	None

None

Test case:	[OMB] Verify osu_put_latency binary is available (gnu14-mpich)
Outcome:	Passed
Duration:	0.029 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	11.99 sec
Tests	2
Failures	2

Tests

rm_execution

Test case:	[OMB] run osu_bw on 2 nodes under resource manager (/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.398 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 22)
  `run_mpi_binary -t "$CMD_TIMEOUT" "$EXE" "-m 512" 2 2' failed
job script = /tmp/job.ohpc-test.17753
Batch job 341 submitted

Job 341 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- osu_bw -m 512 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348061.866118] [c2:9970 :0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x19f8f30 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:09970] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348062.121518] [c1:12970:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1899050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:12970] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:12970:0:12970] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1787ec0)
==== backtrace (tid:  12970) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f8af37dc2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f8af37dc5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f8af37dc854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f8af3e8d910]
 4  [0x1787ec0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 12970 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[OMB] run osu_latency on 2 nodes under resource manager (/gnu14/openmpi5)
Outcome:	Failed
Duration:	7.592 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 35)
  `run_mpi_binary -t "$CMD_TIMEOUT" "$EXE" "-m $MESSAGE_SIZE" 2 2' failed
job script = /tmp/job.ohpc-test.19231
Batch job 342 submitted

Job 342 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- osu_latency -m 0 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348065.531377] [c2:10006:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1c6bbe0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10006] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348065.784643] [c1:13004:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xaf8020 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13004] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:13004:0:13004] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x839920)
==== backtrace (tid:  13004) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f78d53f62bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f78d53f65ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f78d53f6854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f78d5aa7910]
 4  [0x839920]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 13004 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.36 sec
Tests	7
Failures	0

Tests

test_module

Test case:	[OMB] Verify OMB module is loaded and matches rpm version (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.189 sec
Failed	None

None

Test case:	[OMB] Verify OMB_DIR is defined and directory exists (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.019 sec
Failed	None

None

Test case:	[OMB] Verify osu_bw binary is available (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[OMB] Verify osu_latency binary is available (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.029 sec
Failed	None

None

Test case:	[OMB] Verify osu_allgather binary is available (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test case:	[OMB] Verify osu_get_bw binary is available (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.03 sec
Failed	None

None

Test case:	[OMB] Verify osu_put_latency binary is available (gnu14-openmpi5)
Outcome:	Passed
Duration:	0.03 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	15.009 sec
Tests	3
Failures	0

Tests

rm_execution

Test case:	[perf-tools/Scalasca] MPI C binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	9.091 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] MPI C++ binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.776 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Serial C OpenMP binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.142 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.506 sec
Tests	11
Failures	0

Tests

test_module

Test case:	[perf-tools/Scalasca] Verify scalasca module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.204 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify module SCALASCA_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify module SCALASCA_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.023 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scalasca binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.034 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scan binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scout.mpi binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scout.omp binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scout.hyb binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scout.ser binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of square binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of user guide for scalasca (gnu14/mpich)
Outcome:	Passed
Duration:	0.033 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	13.112 sec
Tests	3
Failures	2

Tests

rm_execution

Test case:	[perf-tools/Scalasca] MPI C binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.545 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 24)
  `run_mpi_binary -s 1 ./mpi/C_mpi_test $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.10179
Batch job 365 submitted

Job 365 failed...
Reason=NonZeroExitCode

S=C=A=N: Scalasca 2.6.1 trace collection and analysis
S=C=A=N: ./scorep_C_mpi_test_8x2_trace experiment archive
S=C=A=N: Sat Oct 19 10:30:37 2024: Collect start
/opt/ohpc/pub/utils/prun/2.2/prun ./mpi/C_mpi_test 8
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./mpi/C_mpi_test 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348241.560824] [c2:10656:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2a9f430 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10656] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348241.570757] [c2:10655:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2a17fe0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10655] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348241.597762] [c2:10658:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x24eb350 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10658] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348241.640675] [c2:10657:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x18e0fe0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10657] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348241.794402] [c1:14302:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xeb4420 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:14302] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348241.827485] [c1:14303:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e6c010 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:14303] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348241.869112] [c1:14301:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2a99050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348241.870023] [c1:14304:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xf7b010 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:14301] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:14304] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:14304:0:14304] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xc7da30)
[c1:14301:0:14301] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2910230)
==== backtrace (tid:  14304) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f68bc8e42bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f68bc8e45ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f68bc8e4854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f68bd886910]
 4  [0xc7da30]
=================================
==== backtrace (tid:  14301) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f509bfb52bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f509bfb55ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f509bfb5854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f509d68e910]
 4  [0x2910230]
=================================
[c1:14302:0:14302] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xaaf080)
[c1:14303:0:14303] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1c3df30)
==== backtrace (tid:  14302) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f47282222bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f47282225ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4728222854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f47291d2910]
 4  [0xaaf080]
=================================
==== backtrace (tid:  14303) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f529dfab2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f529dfab5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f529dfab854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f529f689910]
 4  [0x1c3df30]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 3 with PID 14304 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------
S=C=A=N: Sat Oct 19 10:30:42 2024: Collect done (status=0) 5s
Abort: missing experiment archive ./scorep_C_mpi_test_8x2_trace

Test case:	[perf-tools/Scalasca] MPI C++ binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.427 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 38)
  `run_mpi_binary -s 1 ./mpi/CXX_mpi_test $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.25631
Batch job 366 submitted

Job 366 failed...
Reason=NonZeroExitCode

S=C=A=N: Scalasca 2.6.1 trace collection and analysis
S=C=A=N: ./scorep_CXX_mpi_test_8x2_trace experiment archive
S=C=A=N: Sat Oct 19 10:30:44 2024: Collect start
/opt/ohpc/pub/utils/prun/2.2/prun ./mpi/CXX_mpi_test 8
[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./mpi/CXX_mpi_test 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348246.528198] [c2:10699:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x29d5500 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10699] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348246.531382] [c2:10700:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xf43960 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10700] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348246.555214] [c2:10702:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1d99960 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[1729348246.555216] [c2:10701:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b8c500 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10701] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:10702] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348246.769090] [c1:14452:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b60050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:14452] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348246.771546] [c1:14453:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x294c050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:14453] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348246.778960] [c1:14454:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x21e9050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:14454] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348246.783605] [c1:14455:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2173050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:14455] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:14454:0:14454] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2180f20)
[c1:14452:0:14452] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18ef6b0)
[c1:14455:0:14455] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1f47c20)
[c1:14453:0:14453] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x28edb40)
==== backtrace (tid:  14454) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f96fa2222bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f96fa2225ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f96fa222854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f96fb1a5910]
 4  [0x2180f20]
=================================
==== backtrace (tid:  14455) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f7c5f3692bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f7c5f3695ec]
==== backtrace (tid:  14452) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f91547b52bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f91547b55ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f91547b5854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f9155ec2910]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f7c5f369854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f7c602d8910]
 4  [0x1f47c20]
=================================
 4  [0x18ef6b0]
=================================
==== backtrace (tid:  14453) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fd9600222bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fd9600225ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fd960022854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fd960fa5910]
 4  [0x28edb40]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 2 with PID 14454 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------
S=C=A=N: Sat Oct 19 10:30:47 2024: Collect done (status=0) 3s
Abort: missing experiment archive ./scorep_CXX_mpi_test_8x2_trace

Test case:	[perf-tools/Scalasca] Serial C OpenMP binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	2.14 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.517 sec
Tests	11
Failures	0

Tests

test_module

Test case:	[perf-tools/Scalasca] Verify scalasca module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.211 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify module SCALASCA_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify module SCALASCA_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scalasca binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scan binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scout.mpi binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.033 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scout.omp binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.034 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scout.hyb binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of scout.ser binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of square binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.033 sec
Failed	None

None

Test case:	[perf-tools/Scalasca] Verify availability of user guide for scalasca (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.036 sec
Failed	None

None

Test Suite: instrumenter_test

Results

Duration	1.987 sec
Tests	9
Failures	0

Tests

instrumenter_test

Test case:	[perf-tools/Score-P] MPI C binary includes expected Score-P instrumenter symbols (/gnu14/mpich)
Outcome:	Passed
Duration:	0.158 sec
Failed	None

None

Test case:	[perf-tools/Score-P] MPI C++ binary includes expected Score-P instrumenter symbols (/gnu14/mpich)
Outcome:	Passed
Duration:	0.158 sec
Failed	None

None

Test case:	[perf-tools/Score-P] MPI Fortran binary includes expected Score-P instrumenter symbols (/gnu14/mpich)
Outcome:	Passed
Duration:	0.162 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial C OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/mpich)
Outcome:	Passed
Duration:	0.179 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial C++ OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/mpich)
Outcome:	Passed
Duration:	0.178 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial Fortran OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/mpich)
Outcome:	Passed
Duration:	0.176 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary includes expected Score-P instrumenter symbols (/gnu14/mpich)
Outcome:	Passed
Duration:	0.325 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary includes expected Score-P instrumenter symbols (/gnu14/mpich)
Outcome:	Passed
Duration:	0.326 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary includes expected Score-P instrumenter symbols (/gnu14/mpich)
Outcome:	Passed
Duration:	0.325 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	24.636 sec
Tests	9
Failures	0

Tests

rm_execution

Test case:	[perf-tools/Score-P] MPI C binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.474 sec
Failed	None

None

Test case:	[perf-tools/Score-P] MPI C++ binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.477 sec
Failed	None

None

Test case:	[perf-tools/Score-P] MPI Fortran binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	3.87 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial C OpenMP binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.389 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial C++ OpenMP binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.359 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial Fortran OpenMP binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	0.337 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.512 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.513 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary runs under resource manager (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	9.705 sec
Failed	None

None

Test Suite: test_module

Results

Duration	0.693 sec
Tests	14
Failures	0

Tests

test_module

Test case:	[perf-tools/Score-P] Verify scorep module is loaded and matches rpm version (gnu14/mpich)
Outcome:	Passed
Duration:	0.196 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify module SCOREP_DIR is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify module SCOREP_BIN is defined and exists (gnu14/mpich)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-backend-info binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-config binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-info binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.033 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-preload-init binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-score binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-wrapper binary (gnu14/mpich)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep wrapper binaries (gnu14/mpich)
Outcome:	Passed
Duration:	0.128 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of user guide for scorep (gnu14/mpich)
Outcome:	Passed
Duration:	0.034 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of OPEN_ISSUES for scorep (gnu14/mpich)
Outcome:	Passed
Duration:	0.034 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of COPYING for scorep (gnu14/mpich)
Outcome:	Passed
Duration:	0.035 sec
Failed	None

None

Test Suite: instrumenter_test

Results

Duration	2.055 sec
Tests	9
Failures	0

Tests

instrumenter_test

Test case:	[perf-tools/Score-P] MPI C binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.166 sec
Failed	None

None

Test case:	[perf-tools/Score-P] MPI C++ binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.167 sec
Failed	None

None

Test case:	[perf-tools/Score-P] MPI Fortran binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.166 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial C OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.179 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial C++ OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.183 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial Fortran OpenMP binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.179 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.339 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.34 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary includes expected Score-P instrumenter symbols (/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.336 sec
Failed	None

None

Test Suite: rm_execution

Results

Duration	32.987 sec
Tests	9
Failures	6

Tests

rm_execution

Test case:	[perf-tools/Score-P] MPI C binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	6.549 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 45)
  `run_mpi_binary ./mpi/main_mpi_c $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.3606
Batch job 352 submitted

Job 352 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./mpi/main_mpi_c 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348157.128853] [c2:10223:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x11cc550 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10223] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348157.149185] [c2:10221:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x20c8240 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[1729348157.149187] [c2:10222:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x21bc320 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348157.149362] [c2:10220:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1f495e0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10222] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:10221] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:10220] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348157.430495] [c1:13416:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x26f7460 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13416] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348157.433199] [c1:13414:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1852020 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348157.437895] [c1:13415:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1a2b050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13415] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348157.440424] [c1:13417:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16f0470 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13417] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:13414] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:13417:0:13417] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1112f30)
[c1:13416:0:13416] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2712cf0)
[c1:13415:0:13415] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x18151b0)
[c1:13414:0:13414] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1480210)
==== backtrace (tid:  13417) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f397198a2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f397198a5ec]
==== backtrace (tid:  13416) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fa4f59942bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fa4f59945ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fa4f5994854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fa4f7078910]
 4  [0x2712cf0]
=================================
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f397198a854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f3973064910]
 4  [0x1112f30]
=================================
==== backtrace (tid:  13415) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f81d51b52bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f81d51b55ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f81d51b5854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f81d68c0910]
 4  [0x18151b0]
=================================
==== backtrace (tid:  13414) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fbc48fab2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fbc48fab5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fbc48fab854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fbc4a687910]
 4  [0x1480210]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 13415 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[perf-tools/Score-P] MPI C++ binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.489 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 62)
  `run_mpi_binary ./mpi/main_mpi_cxx $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.2289
Batch job 353 submitted

Job 353 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./mpi/main_mpi_cxx 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348163.897757] [c2:10274:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc075d0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10274] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348163.951890] [c2:10276:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xc485f0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[1729348163.952059] [c2:10275:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16484e0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10276] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348163.964246] [c2:10277:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x126be70 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10275] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:10277] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348164.144356] [c1:13461:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2a0d060 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348164.145771] [c1:13464:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x22f3050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13461] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:13464] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348164.193268] [c1:13462:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1ae7420 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13462] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348164.196672] [c1:13463:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16e9050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13463] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:13462:0:13462] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x181c790)
[c1:13463:0:13463] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1426d50)
==== backtrace (tid:  13462) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f5cecc222bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f5cecc225ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f5cecc22854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f5cedbb7910]
 4  [0x181c790]
=================================
==== backtrace (tid:  13463) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fd75ca222bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fd75ca225ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fd75ca22854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fd75d9b5910]
 4  [0x1426d50]
=================================
[c1:13461:0:13461] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x27d2e10)
[c1:13464:0:13464] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x21daee0)
==== backtrace (tid:  13461) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f4796cee2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f4796cee5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f4796cee854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f479865c910]
 4  [0x27d2e10]
=================================
==== backtrace (tid:  13464) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f9c8f3b52bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f9c8f3b55ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f9c8f3b5854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f9c90338910]
 4  [0x21daee0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 2 with PID 13463 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[perf-tools/Score-P] MPI Fortran binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.494 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 79)
  `run_mpi_binary ./mpi/main_mpi_fort $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.6323
Batch job 354 submitted

Job 354 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./mpi/main_mpi_fort 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348168.870750] [c2:10318:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1610b90 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10318] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348168.929619] [c2:10321:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x21f3b90 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348168.937481] [c2:10320:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x215fb90 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10320] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348168.945290] [c2:10319:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2814920 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10321] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:10319] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348169.178517] [c1:13508:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2780050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13508] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348169.192765] [c1:13510:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x227c020 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348169.193963] [c1:13507:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2401050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13510] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:13507] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348169.201234] [c1:13509:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2aa5050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13509] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fefe511d862 in ???
#1  0x7fefe511c8f5 in ???
#2  0x7fefe4a67dbf in ???
#3  0x287bf20 in ???
#0  0x7f968351d862 in ???
#1  0x7f968351c8f5 in ???
#2  0x7f9682cd9dbf in ???
#3  0x219dd30 in ???
#0  0x7f58b0b1d862 in ???
#1  0x7f58b0b1c8f5 in ???
#2  0x7f58b02d9dbf in ???
#3  0x2717f00 in ???
#0  0x7f41b851d862 in ???
#1  0x7f41b851c8f5 in ???
#2  0x7f41b7d90dbf in ???
#3  0x1e5c890 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 13507 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[perf-tools/Score-P] Serial C OpenMP binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.406 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial C++ OpenMP binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.348 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Serial Fortran OpenMP binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	0.348 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) C binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.43 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 147)
  `run_mpi_binary ./mpi/main_hyb_c $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.10287
Batch job 358 submitted

Job 358 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./mpi/main_hyb_c 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348175.913581] [c2:10365:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fb2790 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10365] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348175.944417] [c2:10363:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1df06d0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10363] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348175.947136] [c2:10362:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x270d760 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10362] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348175.964928] [c2:10364:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xe39810 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10364] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348176.185059] [c1:13600:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1b25020 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13600] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348176.249236] [c1:13601:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1e8a060 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13601] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348176.271228] [c1:13599:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2a61020 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13599] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348176.284171] [c1:13598:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fee060 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13598] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:13599:0:13599] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x27a2c70)
[c1:13598:0:13598] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1cf0210)
[c1:13601:0:13601] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1b8c210)
==== backtrace (tid:  13599) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f23be4ac2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f23be4ac5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f23be4ac854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f23bfa3d910]
 4  [0x27a2c70]
=================================
[c1:13600:0:13600] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x190b890)
==== backtrace (tid:  13601) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f8315cc32bc]
==== backtrace (tid:  13598) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fb85eaee2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fb85eaee5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fb85eaee854]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f8315cc35ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f8315cc3854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f8316c63910]
 4  [0x1b8c210]
=================================
 3  /lib64/libpthread.so.0(+0x16910) [0x7fb860252910]
 4  [0x1cf0210]
=================================
==== backtrace (tid:  13600) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f9cda4ee2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f9cda4ee5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f9cda4ee854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f9cdb486910]
 4  [0x190b890]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 13599 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) C++ binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	5.495 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 164)
  `run_mpi_binary ./mpi/main_hyb_cxx $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.975
Batch job 359 submitted

Job 359 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./mpi/main_hyb_cxx 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348180.901661] [c2:10409:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x242c010 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10409] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348180.915109] [c2:10407:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2009030 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348180.915589] [c2:10408:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2570090 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10407] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:10408] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348180.934462] [c2:10406:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xce0080 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10406] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348181.156330] [c1:13646:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1fdf470 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13646] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348181.170695] [c1:13647:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2ae7460 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13647] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348181.234576] [c1:13644:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x220b430 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13644] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348181.283341] [c1:13645:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2f5b0b0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13645] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:13645:0:13645] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2a5e6a0)
==== backtrace (tid:  13645) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fdbdd2ee2bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fdbdd2ee5ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fdbdd2ee854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fdbdec5b910]
 4  [0x2a5e6a0]
=================================
[c1:13646:0:13646] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1da4ec0)
[c1:13647:0:13647] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2a89750)
[c1:13644:0:13644] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x1f4cbe0)
==== backtrace (tid:  13646) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f0e5cfb52bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f0e5cfb55ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f0e5cfb5854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f0e5df38910]
 4  [0x1da4ec0]
=================================
==== backtrace (tid:  13647) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f028f9942bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f028f9945ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f028f994854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f02912c6910]
 4  [0x2a89750]
=================================
==== backtrace (tid:  13644) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7fe9941692bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7fe9941695ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7fe994169854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7fe995a86910]
 4  [0x1f4cbe0]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 13645 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[perf-tools/Score-P] Hybrid (MPI+OpenMP) Fortran binary runs under resource manager (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.428 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file rm_execution, line 181)
  `run_mpi_binary ./mpi/main_hyb_fort $ARGS $NODES "$TASKS"' failed
job script = /tmp/job.ohpc-test.6562
Batch job 360 submitted

Job 360 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./mpi/main_hyb_fort 8 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348185.900909] [c2:10450:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x10f80a0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10450] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348185.946893] [c2:10453:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x16ce070 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348185.948065] [c2:10452:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x220e200 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348185.950236] [c2:10451:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2ac3050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10451] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:10453] pml_ucx.c:313  Error: Failed to create UCP worker
[c2:10452] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348186.183268] [c1:13693:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1663460 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13693] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348186.187231] [c1:13692:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x2409020 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13692] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348186.225463] [c1:13691:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1033060 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13691] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348186.292871] [c1:13690:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x1a87050 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:13690] pml_ucx.c:313  Error: Failed to create UCP worker

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f432cf1d862 in ???
#1  0x7f432cf1c8f5 in ???
#2  0x7f432c7b4dbf in ???
#3  0x185df20 in ???
#0  0x7fc23eb1d862 in ???
#1  0x7fc23eb1c8f5 in ???
#2  0x7fc23e459dbf in ???
#3  0x15f9aa0 in ???
#0  0x7f5052b1d862 in ???
#1  0x7f5052b1c8f5 in ???
#2  0x7f50523e2dbf in ???
#3  0x21f1f20 in ???
#0  0x7f18dff1d862 in ???
#1  0x7f18dff1c8f5 in ???
#2  0x7f18df71bdbf in ???
#3  0xc14d40 in ???
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 13690 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test Suite: test_module

Results

Duration	0.741 sec
Tests	14
Failures	0

Tests

test_module

Test case:	[perf-tools/Score-P] Verify scorep module is loaded and matches rpm version (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.203 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify module SCOREP_DIR is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify module SCOREP_BIN is defined and exists (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.021 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.033 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-backend-info binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-config binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-info binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-preload-init binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-score binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.032 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep-wrapper binary (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.035 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of scorep wrapper binaries (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.168 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of user guide for scorep (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.033 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of OPEN_ISSUES for scorep (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.034 sec
Failed	None

None

Test case:	[perf-tools/Score-P] Verify availability of COPYING for scorep (gnu14/openmpi5)
Outcome:	Passed
Duration:	0.033 sec
Failed	None

None

Test Suite: test_harness

Results

Duration	3.406 sec
Tests	3
Failures	0

Tests

test_harness

Test case:	[RMS/harness] Verify zero exit code from MPI job runs OK (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	2.254 sec
Failed	None

None

Test case:	[RMS/harness] Verify non-zero exit code from MPI job detected as failure (slurm/gnu14/mpich)
Outcome:	Passed
Duration:	1.152 sec
Failed	None

None

Test case:	[RMS/harness] Verify long-running MPI job terminates with timeout parameter (slurm/gnu14/mpich)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

disabled timeout test

Test Suite: test_harness

Results

Duration	8.674 sec
Tests	3
Failures	1

Tests

test_harness

Test case:	[RMS/harness] Verify zero exit code from MPI job runs OK (slurm/gnu14/openmpi5)
Outcome:	Failed
Duration:	4.397 sec
Failed	None

(from function `run_mpi_binary' in file ./common/functions, line 414,
 in test file test_harness, line 23)
  `run_mpi_binary ./mpi_exit 0 $NODES $TASKS' failed
job script = /tmp/job.ohpc-test.24406
Batch job 371 submitted

Job 371 failed...
Reason=NonZeroExitCode

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun -- ./mpi_exit 0 (family=openmpi5)
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 12
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348265.897678] [c2:10806:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0xf89c30 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c2:10806] pml_ucx.c:313  Error: Failed to create UCP worker
[create_qp:2753]create qp: failed on ibv_cmd_create_qp with 95
[1729348266.117694] [c1:14618:0]        ib_iface.c:1117 UCX  ERROR qedr0: iface 0x169d640 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
[c1:14618] pml_ucx.c:313  Error: Failed to create UCP worker
[c1:14618:0:14618] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x15cff00)
==== backtrace (tid:  14618) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(ucs_handle_error+0x12c) [0x7f32e7a012bc]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x355ec) [0x7f32e7a015ec]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.17.0/lib/libucs.so.0(+0x35854) [0x7f32e7a01854]
 3  /lib64/libpthread.so.0(+0x16910) [0x7f32e80b4910]
 4  [0x15cff00]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 14618 on node c1 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Test case:	[RMS/harness] Verify non-zero exit code from MPI job detected as failure (slurm/gnu14/openmpi5)
Outcome:	Passed
Duration:	4.277 sec
Failed	None

None

Test case:	[RMS/harness] Verify long-running MPI job terminates with timeout parameter (slurm/gnu14/openmpi5)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

disabled timeout test

Test Suite: run

Results

Duration	6.675 sec
Tests	5
Failures	0

Tests

run

Test case:	[charliecloud] check for RPM
Outcome:	Passed
Duration:	0.304 sec
Failed	None

None

Test case:	[charliecloud] build image
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

skip experimental charliecloud image builder

Test case:	[charliecloud] build alpine image from Docker (using singularity)
Outcome:	Passed
Duration:	5.594 sec
Failed	None

None

Test case:	[charliecloud] exec image locally
Outcome:	Passed
Duration:	0.314 sec
Failed	None

None

Test case:	[charliecloud] exec image via slurm
Outcome:	Passed
Duration:	0.463 sec
Failed	None

None

Test Suite: run

Results

Duration	12.103 sec
Tests	4
Failures	0

Tests

run

Test case:	[singularity] check for RPM
Outcome:	Passed
Duration:	0.028 sec
Failed	None

None

Test case:	[singularity] pull down ubuntu docker image
Outcome:	Passed
Duration:	11.308 sec
Failed	None

None

Test case:	[singularity] exec image
Outcome:	Passed
Duration:	0.321 sec
Failed	None

None

Test case:	[singularity] exec image via slurm
Outcome:	Passed
Duration:	0.446 sec
Failed	None

None

Test Suite: completion

Results

Duration	0.103 sec
Tests	1
Failures	0

Tests

completion

Test case:	[singularity] check for bash completion
Outcome:	Passed
Duration:	0.103 sec
Failed	None

None

Test Suite: ntp

Results

Duration	0.355 sec
Tests	3
Failures	0

Tests

ntp

Test case:	[ntp] check for chronyc binary
Outcome:	Passed
Duration:	0.03 sec
Failed	None

None

Test case:	[ntp] verify local time in sync on SMS
Outcome:	Passed
Duration:	0.137 sec
Failed	None

None

Test case:	[ntp] verify local time in sync on compute
Outcome:	Passed
Duration:	0.188 sec
Failed	None

None

Test Suite: mem_limits

Results

Duration	0.404 sec
Tests	2
Failures	1

Tests

mem_limits

Test case:	[memlock] check increased soft limit
Outcome:	Passed
Duration:	0.166 sec
Failed	None

None

Test case:	[memlock] check increased hard limit
Outcome:	Failed
Duration:	0.238 sec
Failed	None

(in test file mem_limits, line 38)
  `grep -v HARD <"${OUTPUT}" | while IFS= read -r limit; do' failed
srun: job 8 queued and waiting for resources

Test Suite: pdsh

Results

Duration	0.536 sec
Tests	4
Failures	0

Tests

pdsh

Test case:	[pdsh] check for RPM
Outcome:	Passed
Duration:	0.047 sec
Failed	None

None

Test case:	[pdsh] run a shell command on c[1-4]
Outcome:	Passed
Duration:	0.22 sec
Failed	None

None

Test case:	[pdsh] check for pdsh-mod-slurm RPM
Outcome:	Passed
Duration:	0.049 sec
Failed	None

None

Test case:	[pdsh] run a shell command on -P normal
Outcome:	Passed
Duration:	0.22 sec
Failed	None

None

Test Suite: ompi_info

Results

Duration	0.68 sec
Tests	1
Failures	0

Tests

ompi_info

Test case:	[openmpi] check for no output to stderr with ompi_info
Outcome:	Passed
Duration:	0.68 sec
Failed	None

None

Test Suite: magpie

Results

Duration	0.922 sec
Tests	3
Failures	0

Tests

magpie

Test case:	[magpie] check for RPM
Outcome:	Passed
Duration:	0.047 sec
Failed	None

None

Test case:	[magpie] Verify MAGPIE module is loaded and matches rpm version
Outcome:	Passed
Duration:	0.52 sec
Failed	None

None

Test case:	[magpie] Verify module MAGPIE_DIR is defined and exists
Outcome:	Passed
Duration:	0.355 sec
Failed	None

None

Test Suite: munge

Results

Duration	1.13 sec
Tests	4
Failures	0

Tests

munge

Test case:	[munge] check for OS provdied RPM
Outcome:	Passed
Duration:	0.044 sec
Failed	None

None

Test case:	[munge] Generate a credential
Outcome:	Passed
Duration:	0.031 sec
Failed	None

None

Test case:	[munge] Decode credential locally
Outcome:	Passed
Duration:	0.022 sec
Failed	None

None

Test case:	[munge] Run benchmark
Outcome:	Passed
Duration:	1.033 sec
Failed	None

None

Test Suite: sacct

Results

Duration	0.0 sec
Tests	1
Failures	0

Tests

sacct

Test case:	[slurm] check for working sacct
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

Temporarily skip sacct check

Test Suite: sinfo

Results

Duration	0.078 sec
Tests	1
Failures	0

Tests

sinfo

Test case:	[slurm] Verify SLURM RPM version matches sinfo binary
Outcome:	Passed
Duration:	0.078 sec
Failed	None

None

Test Suite: oom

Results

Duration	0.0 sec
Tests	1
Failures	0

Tests

oom

Test case:	[oom] Test job OOM condition (gnu14/slurm)
Outcome:	Skipped
Duration:	0.0 sec
Failed	None
Skipped	None

None

disable to invistigate disk fillup

Test Report : results

Test Suite: InstallTests.computes_installed-20241019093110

Results

Tests

InstallTests.computes_installed

Test Suite: sms_installed.bats

Results

Tests

sms_installed.bats

Test Suite: clustershell

Results

Tests

clustershell

Test Suite: conman

Results

Tests

conman

Test Suite: genders

Results

Tests

genders

Test Suite: nhc

Results

Tests

nhc

Test Suite: slurm-plugins

Results

Tests

slurm-plugins

Test Suite: lmod

Results

Tests

lmod

Test Suite: spack

Results

Tests

spack

Test Suite: build

Results

Tests

build

Test Suite: rm_execution_multi_host

Results

Tests

rm_execution_multi_host

Test Suite: rm_execution_single_host

Results

Tests

rm_execution_single_host

Test Suite: build

Results

Tests

build

Test Suite: rm_execution_multi_host

Results

Tests

rm_execution_multi_host

Test Suite: rm_execution_single_host

Results

Tests

rm_execution_single_host

Test Suite: build

Results

Tests

build

Test Suite: rm_execution_multi_host

Results

Tests

rm_execution_multi_host

Test Suite: rm_execution_single_host

Results

Tests

rm_execution_single_host

Test Suite: build

Results

Tests

build

Test Suite: rm_execution_multi_host

Results

Tests