Reason: image not found for macOS packages

[jrbourbeau]

Recently we've seen tests begin to fail on macOS for a few independent project in the Dask ecosystem, all of which are failing with Reason: image not found. For example, over in dask/dask#8472 we're getting (see this CI build):

py.test dask --runslow --cov=dask --cov-report=xml -n4 
ImportError while loading conftest '/Users/runner/work/dask/dask/conftest.py'.
conftest.py:43: in <module>
    import tiledb  # noqa: F401
../../../miniconda3/envs/test-environment/lib/python3.7/site-packages/tiledb/__init__.py:34: in <module>
    ctypes.CDLL(lib_name)
../../../miniconda3/envs/test-environment/lib/python3.7/ctypes/__init__.py:364: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: dlopen(libtiledb.dylib, 6): Library not loaded: @rpath/libcrypto.1.1.dylib
E     Referenced from: /Users/runner/miniconda3/envs/test-environment/lib/libtiledb.dylib
E     Reason: image not found

in coiled/dask-bigquery#31 (see this CI build):

 E            - dlopen(/Users/runner/miniconda3/envs/test-environment/lib/python3.7/site-packages/pyarrow/_parquet.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libssl.1.1.dylib
E             Referenced from: /Users/runner/miniconda3/envs/test-environment/lib/libthrift.0.13.0.dylib
E             Reason: image not found

and a similar issue was reported in dask/distributed#5601.

The errors are similar, but appear to be pointing towards separate libraries not being loaded. I get the sense this may be some sort of packing issue (we're using conda-forge packages in these CI builds) and I'm wondering if this is a know issue, or is folks here have any thoughts on how we may proceed in debugging. Apologies for the very vague issue, happy to dig in more if additional details would be useful.

[jakirkham]

Hmm...not really sure. The last openssl build was from early November (so ~1.5months ago). The other packages mentioned like pyarrow and tiledb haven't really seen any relevant changes recently.

Is it possible things are just really out-of-date in the installer or environment used? Have you tried running conda update --all to start?

[jrbourbeau]

I gave conda update --all a try and, while it did update some packages, it didn't resolve the Reason: image not found error.

Looking at dask-bigquery (which has a relatively simple CI setup), I noticed that using this environment file which specifies only the conda-forge channel, along with this conda-incubator/setup-miniconda GitHub actions setup, which is using mambaforge and strict channel priority, we still end up with a few packages installed from the main channel (including libthrift which the image not found error is pointing to). That mixing of conda channels seems like it might be to blame, but I'm not yet sure why we're pulling in packages from main in the first place

[jakirkham]

I'm not yet sure why we're pulling in packages from main in the first place

Hmm...missed this bit. Is Distributed not using strict channel priority? If so, that should be changed (as not using it tends to result in this mismatch of incompatible packages)

[jrbourbeau]

Distributed is using this condarc file with strict channel priority, and conda-forge at a higher priority than defaults

[jakirkham]

Does this reproduce with Conda?

[jakirkham]

Also is there a reason this says true instead of strict?

[jrbourbeau]

Hmm good catch, I suspect that's an oversight and should be changed to strict. That said, dask-bigquery is using strict already and experiencing the same issue of a few packages being installed from main. FWIW it looks like the conda-incubator/setup-miniconda action may be where the defaults channel is added (xref conda-incubator/setup-miniconda#207)

[jsignell]

Just to say that dask is encountering similar issues even though channel-priority is strict and tiledb is coming from conda-forge.

[jsignell]

I thought adding nodefaults to the env might help dask/dask#8505

[jakirkham]

Is this issue still active?

[jsignell]

Well it seems to have stopped. I did add nodefaults and that seemed to make things better, so I guess we can close and just reopen if it comes up again.

[jakirkham]

Sounds good. Thanks Julia! 😄