Bug 14886

Summary:	Python ctypes find_library() not working correctly.
Product:	buildroot	Reporter:	Pete Morici <pmorici>
Component:	Other	Assignee:	unassigned
Status:	RESOLVED MOVED
Severity:	normal	CC:	buildroot, marcus.hoffmann, yann.morin.1998
Priority:	P5
Version:	2022.02.3
Target Milestone:	---
Hardware:	PC
OS:	All
Host:		Target:
Build:
Attachments:	Python patch to support ctypes when using uclibc libiio patch to fix python bindings

Description Pete Morici 2022-07-01 06:58:53 UTC

The find_library() function in the ctypes Python standard library is not working properly.

  https://docs.python.org/3/library/ctypes.html#finding-shared-libraries

You can reproduce the issue by trying the following in the python shell...

>>> from ctypes.util import find_library
>>> find_library("c")

This should return the string "libc.so.0" but returns nothing, this is the same for any library name I tried even though you can see the libraries are installed and "ldconfig -v" lists the libraries.

This has the effect of causing libraries like pyusb to not work properly because they reply on ctypes & find_library() to find libusb.

Digging a bit deeper on how find_library() works it calls either ldconfig -p, ld, or gcc.  buildroot doesn't have ld or gcc installed so I assume that means it is using "ldconfig -p".  ldconfig is part of the uclibc library and uclibc has caching turned off in its configuration so ldconfig -p just prints a message says, "Library cache disabled" instead of printing a list of libraries and their absolute paths.

https://github.com/python/cpython/blob/main/Lib/ctypes/util.py

There is an option in the uclibc menuconfig to enable the cache (LDSO_CACHE_SUPPORT) but when I did that it didn't seem to change anything.  Any ideas how to fix this?

I'm using the pc_x86_64_efi_defconfig with Python enabled.  No other changes.

Comment 1 Peter Scheie 2022-07-01 16:55:30 UTC

I have seen this bug, too, in building an image for the Beaglebone Black.  It appears in the iio package in the python binding: In iio.py, CDLL calls find_library with the argument "iio" to get the name of the libiio.so.0 library, but fails.  Some folks on #beagle said using find_library() this way was a bad idea because it produces unpredictable results (which I can attest to).  Instead, it was recommended that CDLL be called with libiio.so.0 directly, removing find_libary().  This works and I am using a patch to implement it.  BR currently uses version 0.19 of libiio but even in the most current version, 0.24, it still uses find_library().

I tried find_library() on a few other libs and got the same None result.

Comment 2 Pete Morici 2022-07-11 06:44:50 UTC

The problem is a bunch of dependencies rely on the find_library function so I don't really have a choice short of patching every library.

I think the right way to fix this would be to enable the ldconfig cache mechanism in uclibc or mark libraries that depend on ctypes as dependent on glibc tthat does support the caching mechanism required for find_library

Comment 3 Pete Morici 2022-07-15 00:47:21 UTC

    I was able to solve this by enabling the LDSO_CACHE_SUPPORT option in uclibc via a configuration fragment and then patching the file Lib/ctypes/util.py in the python3 package to change the mapping of "x86_64-64" from "libc6,x86_64" to "libc0".  I wasn't able to find a way to generate the cache file during the build process so you have to run ldconfig on first boot, you could use and init script, before using Python to generate /etc/ls.so.cache and then ctypes will work correctly.

    The crux of the issue is that, on Linux, Python's ctypes module just isn't compatible with anything except glibc because is explicitly looks for "libc6" and other glibc specific items in the output of ldconfig -p which is uses to resolve paths to libraries loaded with ctypes.  If you are using uclibc, of musl you are going to run into this.

Comment 4 Thomas Petazzoni 2022-07-18 22:06:32 UTC

This issue is indeed tricky. I think the right solution would be to patch Python to reimplement the find_library() function in a way that is compatible with Buildroot:

 (1) Walk the library directories to find the matching library.

 (2) Retrieve its SONAME. This might be a bit annoying to do in pure Python without invoking a separate Python library to read ELF files.

A bit annoying to do, but probably the solution that is going to require the least amount of dependencies in Buildroot.

Comment 5 Pete Morici 2022-07-20 00:49:23 UTC

Created attachment 9341 [details]
Python patch to support ctypes when using uclibc

Also set LDSO_CACHE_SUPPORT=y un uclibc configuration.  Then run ldconfig at boot prior to using Python / ctypes if ld.so.cache doesn't exist.

if [[ ! -e "/etc/ld.so.cache" ]]; then ldconfig; fi

Comment 6 Pete Morici 2022-07-20 00:59:57 UTC

My understanding from reading some of the other posts is that the reason the ld.so.cache is disabled is because you can't reliably run it in the fakeroot environment without a bunch of patches.

Comment 7 Marcus Hoffmann 2023-03-31 18:41:01 UTC

Created attachment 9566 [details]
libiio patch to fix python bindings

I've just run into this problem with libiio python bindings as well. (On a glibc system though), the attached (libiio) patch fixes it, though I'm not sure if that's a desired solution.

Comment 8 Yann E. MORIN 2024-06-15 15:03:37 UTC

Thank you for your report.

The issue tracker for the Buildroot project has been moved to
the Gitlab.com issue tracker:
    https://gitlab.com/buildroot.org/buildroot/-/issues

We are taking this opportunity to close old issues in this old
tracker. If you believe your issue is still relevant, please
open one in the new issue tracker.

Thank you!