Support peer access DPC++ extension #2077

ndgrigorian · 2025-05-05T21:32:50Z

This PR adds full support for the DPC++ sycl_ext_oneapi_peer_access extension

This adds a Python interface to calls that check if a dpctl.SyclDevice can enable peer access to another dpctl.SyclDevice and, if so, to enable or disable it.

Have you provided a meaningful PR description?
Have you added a test, reproducer or referred to an issue with a reproducer?
Have you tested your changes locally for CPU and GPU devices?
Have you made sure that new changes do not introduce compiler warnings?
Have you checked performance impact of proposed changes?
Have you added documentation for your changes, if necessary?
Have you added your changes to the changelog?
If this PR is a work in progress, are you opening the PR as a draft?

github-actions · 2025-05-05T22:09:11Z

View rendered docs @ https://intelpython.github.io/dpctl/pulls/2077/index.html

github-actions · 2025-05-05T22:16:36Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_194 ran successfully.
Passed: 1109
Failed: 3
Skipped: 119

coveralls · 2025-05-05T22:18:18Z

coverage: 84.974% (-1.4%) from 86.372%
when pulling 031cff1 on feature/support-peer-access-extension
into 3aff6ca on master.

github-actions · 2025-05-06T00:06:08Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_194 ran successfully.
Passed: 1109
Failed: 3
Skipped: 119

dpctl/_sycl_device.pyx

libsyclinterface/include/syclinterface/dpctl_sycl_device_interface.h

libsyclinterface/source/dpctl_sycl_device_interface.cpp

dpctl/_sycl_device.pyx

only implemented for backends HIP, CUDA, Level Zero. Validation prevents crashes

Peer device docstrings link to related methods and docs on peer access extension Slips in fixes to other docstrings for SyclDevice methods

More user-readable

github-actions · 2025-05-07T00:31:53Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_199 ran successfully.
Passed: 1107
Failed: 5
Skipped: 119

github-actions · 2025-05-07T00:36:52Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_200 ran successfully.
Passed: 1109
Failed: 3
Skipped: 119

github-actions · 2025-05-07T05:38:55Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_201 ran successfully.
Passed: 1108
Failed: 4
Skipped: 119

dpctl/_sycl_device.pyx

dpctl/tests/test_sycl_device.py

libsyclinterface/source/dpctl_sycl_device_interface.cpp

ndgrigorian · 2025-05-09T22:15:54Z

Noting here that for future reference, we will probably need to document how this extension interacts with sub-devices and composite devices.

For example, per composite device documentation

Interaction with the default context

If the implementation supports the sycl_ext_oneapi_default_context extension, then each platform has the notion of a default context.
This default context contains only the platform’s root devices, which are those returned by platform::get_devices.
Any composite devices for the platform are not considered root devices, and thus the default context does not include these composite devices.
If an application wants a context that contains the composite devices, then it must explicitly create such a context.

this means that while the peer access methods will work between composite device and its component devices, and vice versa, to actually leverage it, the user needs to explicitly construct a context containing component and composite devices.

also make peer access invalid when the device and peer device are the same

github-actions · 2025-05-12T05:03:55Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_203 ran successfully.
Passed: 1105
Failed: 7
Skipped: 119

github-actions · 2025-05-12T22:56:21Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_205 ran successfully.
Passed: 1113
Failed: 6
Skipped: 119

github-actions · 2025-05-12T22:57:49Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_205 ran successfully.
Passed: 1113
Failed: 6
Skipped: 119

github-actions · 2025-05-12T23:00:08Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_206 ran successfully.
Passed: 1113
Failed: 6
Skipped: 119

github-actions · 2025-05-13T00:00:32Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_206 ran successfully.
Passed: 1114
Failed: 5
Skipped: 119

github-actions · 2025-05-13T00:57:21Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_206 ran successfully.
Passed: 1113
Failed: 6
Skipped: 119

github-actions · 2025-05-13T01:43:22Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_206 ran successfully.
Passed: 1113
Failed: 6
Skipped: 119

dpctl/_sycl_device.pyx

libsyclinterface/source/dpctl_sycl_device_interface.cpp

libsyclinterface/tests/test_sycl_peer_access.cpp

Discard unused header for sstream

github-actions · 2025-05-14T09:46:24Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_211 ran successfully.
Passed: 1111
Failed: 8
Skipped: 119

github-actions · 2025-05-14T10:07:22Z

Array API standard conformance tests for dpctl=0.20.0dev0=py310h93fe807_213 ran successfully.
Passed: 1112
Failed: 7
Skipped: 119

antonwolfy · 2025-05-14T09:55:58Z

dpctl/tests/test_sycl_device.py

+    with pytest.raises(ValueError):
+        dev.enable_peer_access(dev)
+    with pytest.raises(ValueError):
+        dev.enable_peer_access(dev)


Suggested change

dev.enable_peer_access(dev)

dev.disable_peer_access(dev)

antonwolfy · 2025-05-14T09:56:53Z

libsyclinterface/source/dpctl_sycl_device_interface.cpp

+    if (D && PD) {
+        if (_CallPeerAccess(*D, *PD)) {
+            try {
+                throw std::invalid_argument("test");


seems debug leftover

Suggested change

throw std::invalid_argument("test");

antonwolfy · 2025-05-14T10:01:53Z

libsyclinterface/tests/test_sycl_peer_access.cpp

+    auto D0 = DPCTLDeviceVector_GetAt(DV, 0);
+    ASSERT_TRUE(D1 != nullptr);
+    bool canEnable = false;
+    EXPECT_NO_FATAL_FAILURE(canEnable = DPCTLDevice_CanAccessPeer(


Do we need to add ASSERT_FALSE(canEnable) after that?

and probably initial value has to be true then:

bool canEnable = true;

antonwolfy · 2025-05-14T10:03:21Z

libsyclinterface/tests/test_sycl_peer_access.cpp

+    ASSERT_TRUE(D1 != nullptr);
+    bool canEnable = false;
+    EXPECT_NO_FATAL_FAILURE(canEnable = DPCTLDevice_CanAccessPeer(
+                                D0, D0, DPCTLPeerAccessType::access_supported));


Do we need to repeat the call and check with DPCTLPeerAccessType::atomics_supported?

In general it seems access type can be also parametrized to reduce duplication, but not sure if it is easy to implement with gtest f/w.

antonwolfy · 2025-05-14T10:07:06Z

libsyclinterface/tests/test_sycl_peer_access.cpp

+TEST_P(TestDPCTLPeerAccess, ChkPeerAccessToSelf)
+{
+    auto D0 = DPCTLDeviceVector_GetAt(DV, 0);
+    ASSERT_TRUE(D1 != nullptr);


Suggested change

ASSERT_TRUE(D1 != nullptr);

ASSERT_TRUE(D0 != nullptr);

ndgrigorian force-pushed the feature/support-peer-access-extension branch from da9f596 to 20e80f1 Compare May 5, 2025 21:33

ndgrigorian mentioned this pull request May 5, 2025

[Feature] Requesting peer2peer access check for PVC for Cupy equivalents #2073

Open

antonwolfy reviewed May 6, 2025

View reviewed changes

ndgrigorian added 9 commits May 6, 2025 15:33

Implement dpctl.SyclDevice peer access

59d6646

Add backend validation for peer access

7d2d20f

only implemented for backends HIP, CUDA, Level Zero. Validation prevents crashes

Directly use get_backend() from sycl_device

fc50e3b

Tweak docstrings

e08c2c9

Peer device docstrings link to related methods and docs on peer access extension Slips in fixes to other docstrings for SyclDevice methods

Simplify logic checking backends for peer access

3af91a8

link method for disabling peer access to enabling method

dfca713

Fix copy paste errors in peer device interface header

bdfee1b

Clarify which device has invalid backend in peer_access methods

4b3e9f1

Use ostringstream to output backend names instead of integers

a21d585

More user-readable

ndgrigorian force-pushed the feature/support-peer-access-extension branch from 20e80f1 to a21d585 Compare May 6, 2025 23:47

Use ostringstream in kernel bundle interface

b8c899e

Adds Python tests for peer_access methods

bb1f1d9

antonwolfy reviewed May 7, 2025

View reviewed changes

parametrize test_peer_device_arg_validation

7106845

Factor out peer access validation

8563cb0

also make peer access invalid when the device and peer device are the same

Factor common code out of C-API peer access functions

9e7031e

ndgrigorian force-pushed the feature/support-peer-access-extension branch from 04c424b to 5a92522 Compare May 12, 2025 22:12

ndgrigorian requested a review from antonwolfy May 12, 2025 22:13

ndgrigorian marked this pull request as ready for review May 12, 2025 22:13

ndgrigorian requested review from vlad-perevezentsev and vtavana as code owners May 12, 2025 22:13

ndgrigorian force-pushed the feature/support-peer-access-extension branch from c8a6345 to b3ead6f Compare May 12, 2025 23:17

ndgrigorian force-pushed the feature/support-peer-access-extension branch from b3ead6f to 3e81777 Compare May 13, 2025 00:11

ndgrigorian added 2 commits May 12, 2025 17:59

Add C-API tests for peer access functions

da45cdb

Add check that backend types are the same in _CallPeerAccess

ba7222f

ndgrigorian force-pushed the feature/support-peer-access-extension branch from 3e81777 to ba7222f Compare May 13, 2025 00:59

antonwolfy reviewed May 13, 2025

View reviewed changes

ndgrigorian added 7 commits May 13, 2025 14:34

Inline _CallPeerAccess helper function

417889d

In;ine _check_peer_access helper function

95e7478

Clean up peer access functions in libsyclinterface

26f27d1

Discard unused header for sstream

Add helper for raising more specific errors in peer access methods

a3df074

Add tests for dpctl peer access enum helper utilities

27162a8

Add an additional libsyclinterface test for peer access to self

3814d56

Adds a Python test for peer access to self raising ValueError

031cff1

ndgrigorian requested a review from antonwolfy May 14, 2025 09:28

antonwolfy reviewed May 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support peer access DPC++ extension #2077

Support peer access DPC++ extension #2077

ndgrigorian commented May 5, 2025

github-actions bot commented May 5, 2025

github-actions bot commented May 5, 2025

coveralls commented May 5, 2025 •

edited

Loading

github-actions bot commented May 6, 2025

github-actions bot commented May 7, 2025

github-actions bot commented May 7, 2025

github-actions bot commented May 7, 2025

ndgrigorian commented May 9, 2025 •

edited

Loading

github-actions bot commented May 12, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 13, 2025

github-actions bot commented May 13, 2025

github-actions bot commented May 13, 2025

github-actions bot commented May 14, 2025

github-actions bot commented May 14, 2025

antonwolfy May 14, 2025

antonwolfy May 14, 2025

antonwolfy May 14, 2025

antonwolfy May 14, 2025

antonwolfy May 14, 2025

antonwolfy May 14, 2025

antonwolfy May 14, 2025

Support peer access DPC++ extension #2077

Are you sure you want to change the base?

Support peer access DPC++ extension #2077

Conversation

ndgrigorian commented May 5, 2025

github-actions bot commented May 5, 2025

github-actions bot commented May 5, 2025

coveralls commented May 5, 2025 • edited Loading

github-actions bot commented May 6, 2025

github-actions bot commented May 7, 2025

github-actions bot commented May 7, 2025

github-actions bot commented May 7, 2025

ndgrigorian commented May 9, 2025 • edited Loading

github-actions bot commented May 12, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 12, 2025

github-actions bot commented May 13, 2025

github-actions bot commented May 13, 2025

github-actions bot commented May 13, 2025

github-actions bot commented May 14, 2025

github-actions bot commented May 14, 2025

antonwolfy May 14, 2025

Choose a reason for hiding this comment

antonwolfy May 14, 2025

Choose a reason for hiding this comment

antonwolfy May 14, 2025

Choose a reason for hiding this comment

antonwolfy May 14, 2025

Choose a reason for hiding this comment

antonwolfy May 14, 2025

Choose a reason for hiding this comment

antonwolfy May 14, 2025

Choose a reason for hiding this comment

antonwolfy May 14, 2025

Choose a reason for hiding this comment

coveralls commented May 5, 2025 •

edited

Loading

ndgrigorian commented May 9, 2025 •

edited

Loading