Add OS scheduling parameters control

[smorita-esol]

Referring to the draft paper of the executor design below, I found that we face the same challenge below (particularly b.).
https://docs.google.com/document/d/1O53xOVlk4zwsfNukLaDbSWfT64wnoD9xh2EjWwX43qo/edit

 1. What kind of cooperation do we want with the OS?
 a. E.g. CPU pinning, memory locking, using the OS scheduling policies
 b. How could OS scheduling parameters be assigned to ROS 2 Executor and
 enforced by the OS (see note in rclc-executor)?

My team is currently investigating how to pass the scheduling parameters to the ROS 2 Executor. And we have already implemented a prototype to receive and apply a set of scheduling parameters for the threads controlled by the ROS 2 real-time executor. We'd like to share the code with the community as some pull requests, but it might be better to share it here and get some feedback from the members who face the same problem here beforehand.

Our basic idea is as below.

1. Implement a new class rclcpp::thread and modify rclcpp to use it.
   This class has the same function set as the std::thread but also additional features to control its thread attributions.
2. Modify the rcl layer to receive a set of scheduling parameters.
   The parameters are described in YAML format and passed via command line parameters, environment variables, or files.
3. the rclcpp reads the parameters from rcl and applies them to each thread in the thread pool. *1
   *1. the current implementation uses them only for the Multithread Executor.

I've forked the rcl and rclcpp repositories to our team's account and created pull requests for them so that we will be able to have a discussion about it.
rcl: esol-community/rcl#1
rclcpp: esol-community/rclcpp#1

And, considering the discussion below, my team will add a rclcpp::thread implementation delegating all the functions to the std::thread ASAP so as to be able to be used in the environment where the specific rclcpp::thread is not implemented yet.
ros2/rcutils#406

[razr]

Nice, could you join our regular call on Mon next week and tell us a little bit more about your work?

[WideAwakeTN]

Hi everyone, I wanted to provide some feedback on this proposal as a follow-up to our RT working group meeting on last monday (27th Jan 23). First, I like the idea of having a replacement for std::thread in an interface compatible way. This is somewhat similar with what I tried to achieve with replacing std::mutex (see ros2/rcpputils#174).

I took a look at the pull requests referenced above and if I understood them correctly the thread configuration is implemented in a global way, i.e. there is no distinction possible for different executor instances. I think that this distinction is necessary and I adressed this functionality in a proposal that I published recently, see https://discourse.ros.org/t/adding-thread-configuration-functionality-for-rclcpp-executors/29753
I suggested to introduce named executors that read their thread configuration from a ROS parameters file. I am not sure, though, if it is better to have the thread configuration in a separate file or integrated into a ROS parameters file. What do you think?

I agree with Carlos that it would make sense to have a factory interface for thread creation in C++ (something which I also wrote about in my afore mentioned ROS discourse article). This approach would be more flexible since, for example, it allows creating a thread pool implementation. Ideally all new interfaces should be optional, i.e. it should always be possible to pass nullptr where they are used.
Personally, I think the replacement for std::thread should be located in rcpputils. The fundamental design question here is if the thread configuration functionality should be part of the thread factory interface (like I suggested it in my afore mentioned ROS discourse article), be part of another new interface or directly be a part of the new thread class? What do you think?

Putting all thread configuration related code into a separate ROS2 realtime support package, either an existing one or a new one, and in case of C++ "hiding" the new functionality behind interfaces seems to be a neat approach. Alternatively, a CMake option could be used which allows turning new functionality on or off without changing a line of code, for example see https://github.com/ros2/rcpputils/pull/174/files#diff-1e7de1ae2d059d21e1dd75d5812d5a34b0222cef273b7c3a2af62eb747f9d20a
Another question is where to put other realtime related functionality, like locking memory pages? That functionality should also be optional in some way for systems that don't support, or don't require, it. Moreover, it should remain possible to compile on non-POSIX compliant systems.

It seems that a lot is going on recently in the ROS2 community, in particular with the takeover of OSRC by Google and the new Space ROS working group. I wonder if some other team has also started on implementing thread configuration functionality for ROS 2? Maybe someone with good connections can try to find out? The TSC meeting might be a good place to ask that question?

Best, Martin

@razr @carlossvg @JanStaschulat
(can someone reference Shuhao into this discussion? I don't know his Github user name).

[WideAwakeTN]

In case there is consensus that we should have a C based low level OS & POSIX abstraction I could also adapt my implementations of rclcpp::PIMutex and rclcpp::RecursivePIMutex accordingly (see https://github.com/ros2/rcpputils/blob/e75a51280bb4e0f1c8b72ff7b241c14c1cedb342/src/mutex.cpp ) once we figured out how to organize that abstraction layer. Should not be a big deal.

[carlossvg]

^ FYI @shuhaowu

[smorita-esol]

@WideAwakeTN
Thanks for your reply and for starting the discussion.

I took a look at the pull requests referenced above and if I understood them correctly the thread configuration is implemented in a global way, i.e. there is no distinction possible for different executor instances. I think that this distinction is required and I adressed this functionality in a proposal that I published recently, see https://discourse.ros.org/t/adding-thread-configuration-functionality-for-rclcpp-executors/29753

In my understanding, each ROS parameter belongs to each node, not each process. But, intrinsically, threads belong to each process. So, it seems natural to receive scheduling parameters of the thread pool used by ROS 2 executors at "rcl" level (not as ROS parameters at "rclcpp" level).
As you said, there is no measure to bundle each scheduling parameter to a specific executor. So, we might have to add "name," "ID," or any other item to map each parameter to an executor in the process.
Do you think it is enough to have parameter names (string) so as to make each executor choose the proper ones? (Q1)

I suggested to introduce named executors that read their thread configuration from a ROS parameters YAML. I am not sure, though, if it is better to have the thread configuration in a separate file or integrated into the ROS parameters file. What do you think?

Do you mean that we have to prepare other infrastructures to set scheduling parameters, like ROS parameters? (Q2)
At least, my team's implementation can receive scheduling parameters as command line parameters in YAML format (and file path including the parameters also). And this feature can be used in the ROS launch framework.

Personally, I would prefer to keep all OS specific and POSIX code out of rclcpp, rcpputils and rcl (a "wrongdoing" that I also made initially in my now closed pull request ros2/rcutils#406 , which I closed/discarded after some discussions).

I know there was a pointing out that the rclcpp's implementation should be platform agnostic. But, if we could prepare a fallback implementation for which the environment had not been supported yet, it would still have backward compatibility.
As I mentioned above, for such environments having no rclcpp::thread implementations, we will offer the "fallback" implementation soon.

Putting all thread configuration related code into a separate ROS2 real-time support package, either an existing one or a new one, and "hiding" the new functionality behind interfaces is now my favorite approach. Putting all thread configuration related code into a separate ROS2 real-time support package, either an existing one or a new one, and "hiding" the new functionality behind interfaces is now my favorite approach.
Moreover, other real-time related functionality, like locking memory pages, should be put in the same package like the thread configuration stuff and be optional in some way for systems that don't support the functionality.

As I said in the last meeting, we have to prepare some infrastructures (of course, including your PIMutex) to support real-time capabilities in the ROS 2 framework.
But I'm not sure if we have to prepare another package for real-time application. Could you kindly point out for what purpose we have to separate the real-time capability from the rclcpp? (Q3)

Personally, I think the replacement for std::thread should be located in rcpputils, as long as it contains no OS specific or POSIX code. This goal could be achieved by creating appropriate interface(s). The fundamental design question here is if the thread configuration functionality should be part of the thread factory interface (like I suggested it in my afore mentioned ROS discourse article) or be part of another new interface that is also implemented in the afore mentioned real-time support package? Happy to hear arguments and opinions on that topic.

If necessary, my team will move the rclcpp::thread implementation to rcpputils. Could you kindly address any documents or requirements of ROS 2 to direct us to separate the environment-specific code from rclcpp? (Q4) My team could not find any rule specifying where we should place such code.

It seems that a lot is going on recently in the ROS2 community, in particular with the takeover of OSRC by Google and the new Space ROS working group. I wonder if some other team has also started on implementing thread configuration functionality for ROS 2? Maybe someone with good connections can try to find out? The TSC meeting might be a good place to ask that question?

As I said at the last meeting, I'm not sure if this issue ticket is the correct thread where we discuss this topic. If you have, could you kindly point out a proper thread or place to discuss this topic? Or do we have to invite them here somehow? (Q5)

In case there is consensus that we should have a C based low level OS & POSIX abstraction I could also adapt my implementations of rclcpp::PIMutex and rclcpp::RecursivePIMutex accordingly (see https://github.com/ros2/rcpputils/blob/e75a51280bb4e0f1c8b72ff7b241c14c1cedb342/src/mutex.cpp ) once we figured out how to organize that abstraction layer. Should not be a big deal.

Thanks for your consideration.
If you decided to use the namespace "rclcpp::" for PiMutex, it would be harmonized with our implementation of rclcpp::thread.

[smorita-esol]

As I mentioned above, for such environments having no rclcpp::thread implementations, we will offer the "fallback" implementation soon.

I added the "fallback" above in the commit below.
https://github.com/esol-community/rclcpp/tree/07551e6cf227f510de6201d90776bfb8a0d7151c

[WideAwakeTN]

Regarding Q1 and Q2: I have no concrete plans on how to best implement and parse parameters for executors. In any case, the executors would need some name or ID which can be referenced in a config file. The parsing of thread configurations can happen at the rcl level, like in your implementation and like it also is the case for the node parameters, as far as I know. But I think there should be a discussion on where to store the thread configurations: in a new config file or an ordinary ROS parameters file which could be extended to also contain thread configurations for executors. I bring this point up because it might be handy to add parameter functionality to executors in general, for example to configure the queue size of an internal ring buffer of a particular executor instance. To prevent confusion: I suspect that executor parameters and node parameters will not share the same codebase since executor parameters have different characteristics than node parameters, i.e. in my opinion executor parameters don't have to be accessible through services and they don't change during runtime.

Regarding Q3: it might be nice to have all code that could cause trouble compiling ROS2, since that code is OS dependent, in a separate package. For example, if someone wanted to compile ROS2 on an unsupported system, for example FreeRTOS, that person would only need to change or rewrite that package. A separate package would also prevent adding confusing ifdefs to already existing packages. In case of my PiMutex classes I finally tried to put most of the ifdefs into the respective CMakeLists.txt, but I am also open to put OS dependent code into a separate package.

Regarding Q4: I am not entirely sure if rclcpp or rcpputils is more appropriate for your thread class. Maybe someone else can comment on that. I would move the PiMutex classes to rclcpp, in case it turns our that this namespace is more appropriate.

Regarding Q5: As far as I know "Next generation ROS" at https://discourse.ros.org/c/ng-ros/25 is the best place to propose new stuff. But, I am not sure if the persons in charge, like the maintainers, will take notice. My latest article which proposed to introduce named executors even remained unanswered as of now. There is also a technical steering commitee (TSC), but I don't know how to reach them or how they create technical roadmaps.

I took a look at your thread class implementation. I guess an interface would make sense for the threads, e.g. rclcpp::IThread or rcpputils::IThread. In this way someone could implement a thread class in his own package, for example to support concurrency on an unsupported microcontroller system. The thread interface approach would also fit better with the idea of having a thread factory, which in turn would allow the implementation of thread pools.
Another question: why did you implement a POSIX based thread class in addition to the C++ std based thread class?

Note: what was written above are just my personal opinions and suggestions. I have no official affiliation with OSRF.

[smorita-esol]

@WideAwakeTN

Regarding Q1 and Q2: I have no concrete plans on how to best implement and parse parameters for executors. In any case, the executors would need some name or ID which can be referenced in a config file.

Then, would you agree that we might well just add a string parameter (e.g. ID) for each thread attribute configuration as an initial proposal? (Q6)
With such string parameters, the executors, which are implemented in the future, can decide how to use each thread, which node should belong to, and so on, at least.

The parsing of thread configurations can happen at the rcl level, like in your implementation and like it also is the case for the node parameters, as far as I know. But I think there should be a discussion on where to store the thread configurations: in a new config file or an ordinary ROS parameters file which could be extended to also contain thread configurations for executors. I bring this point up because it might be handy to add parameter functionality to executors in general, for example to configure the queue size of an internal ring buffer of a particular executor instance.

I know we cannot create a fully comprehensive design of configurations related to executors at once. But, only for thread configurations used by the executor, the current parameter set we are discussing seems relatively good as an initial proposal.
If we found other requirements to be added to the configuration parameters for the executor (like the queue size you mentioned), we might have to reconsider how we should treat them like this thread.

To prevent confusion: I suspect that executor parameters and node parameters will not share the same codebase since executor parameters have different characteristics than node parameters, i.e. in my opinion executor parameters don't have to be accessible through services and they don't change during runtime.

I completely agree with you.

Regarding Q3: it might be nice to have all code that could cause trouble compiling ROS2, since that code is OS dependent, in a separate package. For example, if someone wanted to compile ROS2 on an unsupported system, for example FreeRTOS, that person would only need to change or rewrite that package. A separate package would also prevent adding confusing ifdefs to already existing packages. In case of my PiMutex classes I finally tried to put most of the ifdefs into the respective CMakeLists.txt, but I am also open to put OS dependent code into a separate package.

Regarding Q4: I am not entirely sure if rclcpp or rcpputils is more appropriate for your thread class. Maybe someone else can comment on that. I would move the PiMutex classes to rclcpp, in case it turns our that this namespace is more appropriate.

As you can see in the commit, we choose almost the same approach to pack the environment-specific code to avoid unnecessary ifdefs, using CMakeLists.txt.
It sounds nice to separate such environment-specific code from the existing rclpp and/or rcpputils. But, it seems to require a lot of effort and consideration.
So, do you find it enough that we pack the rclcpp::thread into rcpputils as you mentioned before, and share it with ROS community so far? (Q7)
If yes, we will move our rclcpp::thread implementation into rcpputils and then merge your PiMutex into it, also.
After sharing the code as PRs with ROS community, we can continue to discuss how we should separate the environment-specific code from the existing rclcpp/rcpputils then.

Regarding Q5: As far as I know "Next generation ROS" at https://discourse.ros.org/c/ng-ros/25 is the best place to propose new stuff. But, I am not sure if the persons in charge, like the maintainers, will take notice. My latest article which proposed to introduce named executors even remained unanswered as of now. There is also a technical steering commitee (TSC), but I don't know how to reach them or how they create technical roadmaps.

As I mentioned in the 13th March meeting, I will transfer this discussion into ROS Discourse after the topics to be discussed here are agreed with you.

I took a look at your thread class implementation. I guess an interface would make sense for the threads, e.g. rclcpp::IThread or rcpputils::IThread. In this way someone could implement a thread class in his own package, for example to support concurrency on an unsupported microcontroller system. The thread interface approach would also fit better with the idea of having a thread factory, which in turn would allow the implementation of thread pools.

Our intention is to provide an infrastructure for ROS 2 function developers (not application developers) to implement real-time executors. So, we think the thread management features, which use such an interface, should be included in each executor implementation(so such an implementation is not needed).

Another question: why did you implement a POSIX based thread class in addition to the C++ std based thread class?

When our team tried to control the thread priority for each thread controlled by the existing multithread executor, we found there was no measure to control it within the ROS 2 feature in itself.
Except for the thread parameter configurations, we also found existing std::thread has no measure to control the thread attributes. So, we decided to implement rclcpp::thread to have rclcpp have thread attribute control capabilities.

[WideAwakeTN]

Maybe it’s helpful to come up with a somewhat complete set of questions? I tried to collect all the questions that came to my mind so far in the bullet list below.

Topic: How to provide thread configuration functionality in the context of ROS2?

Regarding plain C:

• Should OS specific and POSIX low level code be confined to a separate ROS2 package? Should this package function like an abstraction layer? Does it make more sense to use ifdefs inside the code to support different operating systems or is it better to create multiple variants of this new package for different operating systems?
  The alternative would be putting the code into rclc or some other already existing package (and using ifdefs).
• How to ensure that the new functionality does not hinder microcontroller development for ROS2? This issue suggests to not put OS specific code or POSIX code into rclc.

Regarding C++:

• Should there be a factory to spawn threads? A factory would allow to implement thread pools.
• Should there be an interface for thread class instances?
• How to configure threads? Configure them when they are spawned, e.g. by passing parameters to the factory interface or the constructor, or configure them later?
• Is thread re-configuration required later on? Then the thread class / interface should provide some kind of public configuration functionality.
• What thread properties should be configurable? Presumably CPU core affinity, CPU priority and scheduler type (FIFO or FAIR).
• Which ROS2 C++ core entities should natively support thread configuration? The MultiThreadedExecutor, SingleThreadedExecutor, StaticSingleThreadedExecutor, the standard container, the multithreaded container and the isolated multithreaded container? Note: the SingleThreadedExecutors could spawn a worker thread internally and join it (the join behaviour could actually be optional and the calling thread might alternatively continue... that might be useful functionality).
• Should the new thread functionality be available to all ROS2 C++ components, e.g. the middleware layer, the TF2 library, etc., and if so how?
• Should the new thread functionality be available to application developers, and if so how?
• How to store and read a thread configuration so that it is available to all supported entities? Should the ROS parameters YML be used/extended, a new file be used or some other method chosen?
• Should there be a thread configuration class/object to combine all the different thread configuration parameters? This class could provide an easy way for everyone to load a thread configuration. (Note: internally the configuration class could access some already available data that was parsed during rclcpp::init).
• Should all executor instances get names, similar to node instances, so that they can access their thread configuration automatically? Should executors get general parameter functionality, similar to nodes? Note: executor parameters are expected to not change during runtime and need not be accessible via services.
• How much flexibility do we need?
• Should there be a CMake option to deactivate any new thread related functionality?
• Should POSIX or OS specific code be be kept out of rclcpp and rcpputils?

Regarding Python:

• Does it make any sense to provide thread configuration facilities for Python?

@smorita-esol
You can reuse and/or adapt that list if you like. Mentioning my contribution would be appreciated.

[smorita-esol]

Our team is adding some comments to @WideAwakeTN 's list.
We will throw the list with comments to a ROS Discourse thread, mentioning his contribution.

Could you kindly inform me of any additional comments or items to be discussed before the end of this week? Then, we could append them the list to be shared before submitted.

[smorita-esol]

I've thrown the updated list to a ROS Discourse thread as below.
https://discourse.ros.org/t/adding-thread-attributes-configuration-in-ros-2-framework/30701/1