sys.path change in generated __init__.py leads to problems with multiprocessing

[timonegk]

The issue

I am using ros in combination with the python multiprocessing library (in the stable_baselines3 framework). When importing modules that contain python code and compiled python libraries, the library import fails in the spawned processes. Here's why:
0. Setup: a catkin workspace is used, the devel folder is sourced, the src folder contains the code.

1. The module is imported in the main process, before other processes are spawned. The catkin-generated __init__.py in the devel space inserts the source for the module at the beginning of the path. The import of the .so file in the devel space is still working, but a print of sys.path after the import shows the path to the module's source folder at the beginning.
2. A new process is spawned, this process also imports the module. At this point, the source space is at the beginning of sys.path. Therefore, the python import logic finds the module in the source space and never searches the devel space, hence never finding the compiled library.

Code to reproduce

1. Create a catkin workspace. Put a package with a compiled python library in the source folder, for example tf2_py. This package should not already be sourced elsewhere, e.g. from /opt/ros/.
2. Build the workspace and source it.
3. Put the following code in a python file and execute it:

import multiprocessing as mp

def fun():
    import tf2_py

if __name__ == '__main__':
    import tf2_py
    ctx = mp.get_context('spawn')
    process = ctx.Process(target=fun)
    process.start()
    process.join()

4. You will see that the import fails in fun() but not in the main function. Prints of sys.path make the issue clearer.

Possible solutions

As far as I can see, values are only appended to sys.path in the generated __init__.py to be able to use pkgutil.extend_path. Therefore my suggestion would be to remove the added paths from the sys_path variable after this line. I coud create a pull request for that.

Related issues

The issue has already been reported at stable baselines: hill-a/stable-baselines#920. A workaround has also been posted, but I think that cleaning sys.path from the added paths would be cleaner than just appending more at the end as a sustainable solution.