-
Notifications
You must be signed in to change notification settings - Fork 194
Open
Labels
Description
Description
This issue is a call for a design of zero-copy intra-process communications available to all ROS2 language clients.
The current implementation of this feature exists in rclcpp
and therefore is not usable for Python (or less-supported languages C, Java, Rust, etc.)
Acceptance Criteria
To close this issue, we want a design document that proposes the architecture for
- Intra-process optimized communications
- zero copy
- Available to all ROS2 language clients (therefore exists at or below the level of
rcl
)
As a follow up, will attempt to collect existing thoughts from #239 and add as comments below.
Note
I do not consider myself personally an expert on this, however I'm very interested in collaborating towards a top down view of what this part of the ROS2 core should look like, and figuring out how the community can pull together towards a solution
eborghi10, atyshka, DunaiFuentes, rty813, connoranderson and 3 more
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Select code repository
Activity
ivanpauno commentedon Aug 26, 2019
I think that we can take some ideas from Connext "zero copy transfer over shared memory":
That's actually interprocess communication over shared memory, but something similar can be replicated using a buffer instead of a piece of shared memory.
The basic idea is that you have to ask to the publisher for a new message, instead of allocating an
unique_ptr
:Currently, message lifetime can be extended to be longer than the scope of the callback (in cpp). That would not be possible if we go ahead with something like this (or at least, it will be really hard to implement that feature).
The implementation could live in
rcl
orrmw
, I'm not sure what would be better.allenh1 commentedon Aug 26, 2019
@ivanpauno I don't think
publisher->get_new_message()
ever returnnullptr
. I'd prefer a more asynchronous way to fetch a message, or potentially blocking on that call instead. I'm not very fond of the blocking call idea, but maybe an asynchronous trigger could be set up?Maybe it could be set up so that we can
std::invoke
a callback in thepublish()
function? This isn't great though, since this would need to be done inrcl
, which means it would be wasting cycles checking if there arestd::bind
ed callbacks on non-shared memory platforms.I'm not seeing a way to make this happen in anything above
rmw
, except of course when there are multiple nodes inside the same process.Sorry for the rambles, very interested in this idea.
fujitatomoya commentedon Aug 27, 2019
just sharing my thought,
i believe that it is better to be implemented in rmw, not rcl.
emersonknapp commentedon Aug 27, 2019
Collecting some relevant parts of the previous discussion here for easier review, and to feed the design:
Re: location of implementation @gbiggs wrote
Re: location of implementation @raghaprasad wrote
Re: smart-ptr messages @gbiggs wrote
Re: implementation @ivanpauno wrote
emersonknapp commentedon Aug 27, 2019
A question: do we want to have intra-process communication always optimized in ROS2, regardless of choice of RMW?
If yes we want it always available, what about this idea?
rmw_intraprocess
rmw_intraprocess
for use by nodes within the same processrcl
orrmw
layer route API calls to the appropriate of the two co-existing RMWs based on whether the communication is within the processOr, this is a possible outcome, should we just expect that intraprocess communications should be the job of the choice of RMW implementation, and just push development to add this to our RMW impl of choice, e.g. FastRTPS or CycloneDDS or wherever?
dirk-thomas commentedon Aug 27, 2019
The overhead described here is addressed by the proposal in #250 and isn't related to intra process communication. Even with intra process communication every node / participant has to perform discovery and comes with that overhead.
ivanpauno commentedon Aug 28, 2019
I guess that it's possible to not return ever
nullptr
(probably with locking behavior), I just added it because I'm not super sure about how the implementation would be.I agree, specially with the first and last points.
Each time I think about the intraprocess communication problem, I'm more convinced that it's a problem that should be addressed by the underlying middleware (FastRTPS, Connext, OpenSplice, etc), and we should only wrap their zero copy transfer API. Of course, that's probably out of our scope and we have to provide a solution on top of the middleware. But that have the cost of re-implementing a lot of things (supporting a lot of different QoS features, etc).
👍
qootec commentedon Sep 17, 2019
I initially posted this as an topic on answers.ros.org (see https://answers.ros.org/question/333180/ros2-micro-ros-intra-process/) but was advised by the moderator to move it to discourse... I think the core of my concern touches your discussion.
(My context: ROS2 inside a machine controller)
Looking at your proposals for intra-process communication, I fail to see whether you also take into account the multi-priority requirements such (often embedded) environments typically have.
I currently see fragmented solution elements or approaches:
From Micro-ROS: Multiple executors could be hosted in the same process/node, each having their own queue for messages (or in fact their handlers) of the corresponding priority (based on their handlers' callbackgroup priority).
From ROS2: ROS2 does not create its own queuing mechanism, but instead relies on the queues already available in the DDS middleware.
From ROS2 (close to this topic): use_intra_process_comms() … if true, messages will go through a special intra-process communication code path. So potentially excluding DDS. Then how will they get queued / priority managed?
(RTI) DDS has a Transport_Priority_QoS defined per DataWriter, which is then to be kept in sync with the cbGroup priority?
Is there any documented vision on how your intra-process-communication would co-exist with multi-priority queuing/handling?
Johan
gavanderhoorn commentedon Sep 17, 2019
I did, but this is not the embedded category on ROS Discourse.
atyshka commentedon Aug 12, 2020
Any updates on this roughly a year later?
ivanpauno commentedon Aug 13, 2020
Not that I know of.
The problem isn't trivial, and AFAIK there is no people assigned to work on it.
twaddellberkeley commentedon Sep 11, 2021
Hi @ivanpauno, is there any work on this problem, if not do you need help? Would love to dive into it.
Cheers
ivanpauno commentedon Sep 13, 2021
AFAIK, nobody is working on this right now.
I'm not sure if there's a plan to work on the topic soon.
emersonknapp commentedon Sep 13, 2021
I'm not sure, but does the Cyclone+iceoryx combo do this automatically for C++ nodes in the same process?
ivanpauno commentedon Sep 13, 2021
Not zero copy, zero-copy requires a different API.
ros-discourse commentedon Jun 13, 2025
This issue has been mentioned on ROS Discourse. There might be relevant details there:
https://discourse.ros.org/t/intra-process-communications-for-all-clients/10416/1