High latency on multi-process simple pub/sub with DYNAMIC_REUSABLE

[mauropasse]

On rolling, I'm seeing high latency when using DYNAMIC_REUSABLE as history memory policy & zero copy.
In the other hand, latency seems OK when using PREALLOCATED_WITH_REALLOC.
Also latency seems OK when the system is single-process, regardless of the policy (still using loaned msgs).

The system tested uses shared memory to pass a 4MB loaned message between processes:

Multi-process system:
- Process 1: Publisher of 4Mb Loaned messages (10Hz)
- Process 2: Subscription of 4Mb messages
- Platforms: x86 & RaspberryPi4

As reference I'm also showing latency of IPC OFF, i.e. not using Loaned messages with shared memory (normal serialized msg).

Below I provide a CPU flamegraph (from RPi) for both policies:

• Left: DYNAMIC_REUSABLE

  • There seem to be a copy of the 4Mb message on eprosima::fastdds::rtps::UDPChannelResource::perform_listen_operation.

• Right: PREALLOCATED_WITH_REALLOC

The XML config file looks like this for both reader & writer:

  <qos><data_sharing><kind>AUTOMATIC</kind></data_sharing></qos>
  <historyMemoryPolicy>DYNAMIC_REUSABLE</historyMemoryPolicy>
  <!-- other: PREALLOCATED_WITH_REALLOC -->

Since in this case we're aiming for zero-copy using shared memory with loaned messages, I'm guessing this copy is undesired, i.e. a bug? What do you think?