| Summary: | Multiple chosen python modules are not built due to missing header files in include path | ||
|---|---|---|---|
| Product: | buildroot | Reporter: | Christian Abegg <christian.abegg> |
| Component: | Other | Assignee: | unassigned |
| Status: | RESOLVED WORKSFORME | ||
| Severity: | normal | CC: | buildroot, christian.abegg |
| Priority: | P5 | ||
| Version: | 2014.08 | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Host: | Target: | ||
| Build: | |||
| Attachments: |
.config file used
patch setup.py to print include dir list |
||
Created attachment 5690 [details]
.config file used
Thanks a lot for your detailed bug report. I will try to reproduce this issue and report back. With this qemu-x86_64 defconfig : BR2_x86_64=y BR2_DL_DIR="~/repo/contrib/dl" BR2_TOOLCHAIN_EXTERNAL=y BR2_TARGET_GENERIC_GETTY_PORT="tty1" BR2_LINUX_KERNEL=y BR2_LINUX_KERNEL_CUSTOM_VERSION=y BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="3.13.5" BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux-3.13.config" BR2_PACKAGE_PYTHON=y BR2_PACKAGE_PYTHON_SQLITE=y BR2_TARGET_ROOTFS_EXT2=y # BR2_TARGET_ROOTFS_TAR is not set I see similar messages during the build: Python build finished, but the necessary bits to build these modules were not found: _bsddb _curses _curses_panel _ssl _tkinter bsddb185 bz2 dbm dl gdbm imageop linuxaudiodev ossaudiodev readline sunaudiodev zlib With this build (which is a slightly different config as the submitter showed) sqlite3 can be imported correctly, but ssl cannot (_ssl not found, same thing as reported). So indeed, something seems incorrect here. I will try to investigate further. I did some further investigation: a. the build message "Python build finished, but the necessary bits to build these modules were not found" occurs when a python module requires some external support but this support is not present. For example, when SSL is not selected in the buildroot configuration, then you get this message for the 'ssl' python module. In this case '_ssl' will be in the list. b. if I build your configuration (but use an external toolchain and disable the kernel, htop, vlc, util-linux, and grub, to save build time), I do not see the same issue. I do get a message about some bits not found, but not for ssl, sqlite, nor any of the other modules you selected. This makes sense: when these python modules are selected in the python configuration, the corresponding dependencies like the ssl package, the sqlite package, etc. are built automatically. This makes me believe that there may be a one-time issue due to some particular action you did on your buildroot directory. Have you tried starting from a clean build, i.e. run 'make clean all' Does the problem occur again? Thanks, Thomas Thanks a lot for your reply. a. This seems logical. However, chosen modules in the configuration where showing up in that list. b. I retried multiple builds with the latest version of buildroot (git 526e3d0). What I observed is that when using the buildroot internal toolchain, the python sqlite3 modules were never built. On the other hand, when using an external toolchain (I used Sourcery CodeBench x86/x86_64 2012.09) the python sqlite3 modules were correctly built. So using an external toolchain solves the problem, which is a bit strange. Here are some excerpts of the python build log, using the buildroot toolchain: building '_curses' extension /tmp/buildroot/output/host/usr/bin/i686-buildroot-linux-gnu-gcc -fPIC -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -Os -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I/tmp/buildroot/output/build/python-2.7.8/Include -I/tmp/buildroot/output/build/python-2.7.8 -c /tmp/buildroot/output/build/python-2.7.8/Modules/_cursesmodule.c -o build/temp.linux2-i686-2.7/tmp/buildroot/output/build/python-2.7.8/Modules/_cursesmodule.o using an external toolchain: building '_curses' extension /tmp/buildroot/output/host/usr/bin/i686-pc-linux-gnu-gcc -fPIC -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -Os -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I/tmp/buildroot/output/host/usr/i686-buildroot-linux-gnu/sysroot/usr/include -I/tmp/buildroot/output/build/python-2.7.8/Include -I/tmp/buildroot/output/build/python-2.7.8 -c /tmp/buildroot/output/build/python-2.7.8/Modules/_cursesmodule.c -o build/temp.linux2-i686-2.7/tmp/buildroot/output/build/python-2.7.8/Modules/_cursesmodule.o The difference I see between the internal and external toolchain, is that with the external one python is built with an additional include path: -I/tmp/buildroot/output/host/usr/i686-buildroot-linux-gnu/sysroot/usr/include. In this directory are the sqlite3.h headers among others. You said you used an external toolchain and the python sqlite module built fine. If you use the internal toolchain, does the module also build correctly? I now tried using an internal toolchain: BR2_x86_64=y BR2_DL_DIR="~/repo/contrib/dl" BR2_TOOLCHAIN_BUILDROOT_WCHAR=y BR2_TARGET_GENERIC_GETTY_PORT="tty1" BR2_LINUX_KERNEL=y BR2_LINUX_KERNEL_CUSTOM_VERSION=y BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="3.13.5" BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux-3.13.config" BR2_PACKAGE_PYTHON=y BR2_PACKAGE_PYTHON_SQLITE=y BR2_TARGET_ROOTFS_EXT2=y # BR2_TARGET_ROOTFS_TAR is not set but this seems to build fine too, wrt sqlite. In the build log I also see the mentioned include paths present, for example: building '_sqlite3' extension creating build/temp.linux2-x86_64-2.7/home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8/Modules/_sqlite /home/tdescham/repo/contrib/buildroot-reproduce/output/host/usr/bin/x86_64-buildroot-linux-uclibc-gcc -fPIC -fno-strict-aliasing -pipe -Os -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DMODULE_NAME="sqlite3" -DSQLITE_OMIT_LOAD_EXTENSION=1 -IModules/_sqlite -I/home/tdescham/repo/contrib/buildroot-reproduce/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot/usr/include -I. -IInclude -I./Include -I/home/tdescham/repo/contrib/buildroot-reproduce/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot/usr/include -I/home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8/Include -I/home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8 -c /home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8/Modules/_sqlite/cache.c -o build/temp.linux2-x86_64-2.7/home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8/Modules/_sqlite/cache.o Your defconfig was: BR2_x86_i686=y BR2_TOOLCHAIN_BUILDROOT_GLIBC=y BR2_TOOLCHAIN_BUILDROOT_CXX=y BR2_TARGET_GENERIC_GETTY_PORT="tty1" BR2_LINUX_KERNEL=y BR2_LINUX_KERNEL_DEFCONFIG="i386" BR2_LINUX_KERNEL_INSTALL_TARGET=y BR2_PACKAGE_VLC=y BR2_PACKAGE_PYTHON=y BR2_PACKAGE_PYTHON_PY_PYC=y BR2_PACKAGE_PYTHON_BZIP2=y BR2_PACKAGE_PYTHON_BSDDB=y BR2_PACKAGE_PYTHON_CODECSCJK=y BR2_PACKAGE_PYTHON_CURSES=y BR2_PACKAGE_PYTHON_PYEXPAT=y BR2_PACKAGE_PYTHON_READLINE=y BR2_PACKAGE_PYTHON_SSL=y BR2_PACKAGE_PYTHON_SQLITE=y BR2_PACKAGE_PYTHON_ZLIB=y BR2_PACKAGE_PYTHON_HASHLIB=y BR2_PACKAGE_OPENSSH=y BR2_PACKAGE_HTOP=y BR2_PACKAGE_UTIL_LINUX=y BR2_TARGET_ROOTFS_CPIO=y BR2_TARGET_ROOTFS_EXT2=y BR2_TARGET_GRUB=y BR2_TARGET_GRUB_FS_ISO9660=y so the main differences I see are: - 32-bit - glibc - c++ support I will now make a build with glibc and c++ to see if it makes a difference... Even enabling C++ and glibc works fine here: BR2_x86_64=y BR2_DL_DIR="~/repo/contrib/dl" BR2_TOOLCHAIN_BUILDROOT_GLIBC=y BR2_TOOLCHAIN_BUILDROOT_CXX=y BR2_TARGET_GENERIC_GETTY_PORT="tty1" BR2_LINUX_KERNEL=y BR2_LINUX_KERNEL_CUSTOM_VERSION=y BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="3.13.5" BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux-3.13.config" BR2_PACKAGE_PYTHON=y BR2_PACKAGE_PYTHON_SQLITE=y BR2_TARGET_ROOTFS_EXT2=y # BR2_TARGET_ROOTFS_TAR is not set Build log snippet: building '_sqlite3' extension creating build/temp.linux2-x86_64-2.7/home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8/Modules/_sqlite /home/tdescham/repo/contrib/buildroot-reproduce/output/host/usr/bin/x86_64-buildroot-linux-gnu-gcc -fPIC -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -Os -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DMODULE_NAME="sqlite3" -DSQLITE_OMIT_LOAD_EXTENSION=1 -IModules/_sqlite -I/home/tdescham/repo/contrib/buildroot-reproduce/output/host/usr/x86_64-buildroot-linux-gnu/sysroot/usr/include -I. -IInclude -I./Include -I/home/tdescham/repo/contrib/buildroot-reproduce/output/host/usr/x86_64-buildroot-linux-gnu/sysroot/usr/include -I/home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8/Include -I/home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8 -c /home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8/Modules/_sqlite/cache.c -o build/temp.linux2-x86_64-2.7/home/tdescham/repo/contrib/buildroot-reproduce/output/build/python-2.7.8/Modules/_sqlite/cache.o Could you try any of these working configurations (build-wise) to check if you see something differently? I'm testing this on a recent git revision, I haven't verified if there were changes in this area. In the Python sources, Lib/distutils/command/build_ext.py seems to be responsible for building extensions, see function build_extension. I think it makes sense to debug this further, inspecting the values of ext.include_dirs. Christian, do you have further feedback on this problem? I cannot reproduce the failure here, but since you detected a specific difference, being the different include paths, I think it would be good to debug at your end by adding some debug statements in the Python file I mentioned in comment #7. Hi Thomas,
thanks for the hints and sorry for the delay.
I adapted my config file with your options and still got the same errors. I also tried glibc and uclib and gcc4.8/4.9 but the sqlite extension is still not built. As soon as I switch to an external toolchain, it works correctly.
For the include paths, I added some print statements in the Lib/distutils/command/build_ext.py and the setup.py file (see br-python-2.7.8-setup-py.patch attachment). As seen below, the sysroot include dir isn't received by the python setup script (setup.py):
# With buildroot internal toolchain
inc_dirs = ['.', 'Include', './Include', '/tmp/buildroot-internal/output/build/python-2.7.8/Include', '/tmp/buildroot-internal/output/build/python-2.7.8']
# Using an external toolchain
inc_dirs = ['.', 'Include', './Include', '/tmp/buildroot-external/output/host/usr/i686-buildroot-linux-gnu/sysroot/usr/include', '/tmp/buildroot-external/output/build/python-2.7.8/Include', '/tmp/buildroot-external/output/build/python-2.7.8']
and from the build_ext.py file:
ext.include_dirs = ['Modules/_sqlite', '/tmp/buildroot-external/output/host/usr/i686-buildroot-linux-gnu/sysroot/usr/include']
So somehow the required include dir ('/tmp/buildroot-external/output/host/usr/i686-buildroot-linux-gnu/sysroot/usr/include') isn't forwarded to the python setup.py script who builds the extensions.
From that point, I'm a bit clueless why the sysroot include dir isn't in the dir list when an internal toolchain is used. Where is the next place to look ? Maybe something with the python configure options ?
Created attachment 5738 [details]
patch setup.py to print include dir list
(In reply to comment #9) > From that point, I'm a bit clueless why the sysroot include dir isn't in the > dir list when an internal toolchain is used. Where is the next place to look ? > Maybe something with the python configure options ? There is no need for the sysroot include dir to be explicitly passed to gcc with the internal toolchain, because the Buildroot sysroot is configured to be the natural location for the compiler to look for headers. (In reply to comment #11) > There is no need for the sysroot include dir to be explicitly passed to gcc > with the internal toolchain, because the Buildroot sysroot is configured to be > the natural location for the compiler to look for headers. Ok, thanks for the details. However I'm still wondering why the inc path array in the setup.py script doesn't contain the sysroot include dir when the internal toolchain. What is the next place were I can check why this isn't the case? I tried again today to do a build with an internal toolchain, python and sqlite enabled, and it works fine, and I can import the sqlite3 python module on the target successfully: Welcome to Buildroot buildroot login: root # python -c "import sqlite3" # So, I'm marking this bug as "works for me", since apparently neither Thomas DS nor I have been able to reproduce it so far. |
Python doesn't build some of the requested modules because the setup.py script is unable to locate the respective header files. The affected modules are sqlite3, ssl, zlib and bsddb. The build completes successfully and the target images are correctly generated, but these modules will not work in python. Buildroot target is i686. Buildroot version is 2014.11-git-00330-g6712213. Chosen modules in .config: BR2_PACKAGE_PYTHON_BZIP2=y BR2_PACKAGE_PYTHON_BSDDB=y BR2_PACKAGE_PYTHON_CODECSCJK=y BR2_PACKAGE_PYTHON_CURSES=y BR2_PACKAGE_PYTHON_PYEXPAT=y BR2_PACKAGE_PYTHON_READLINE=y BR2_PACKAGE_PYTHON_SSL=y BR2_PACKAGE_PYTHON_UNICODEDATA=y BR2_PACKAGE_PYTHON_UCS2=y # BR2_PACKAGE_PYTHON_UCS4 is not set BR2_PACKAGE_PYTHON_SQLITE=y BR2_PACKAGE_PYTHON_ZLIB=y BR2_PACKAGE_PYTHON_HASHLIB=y BR2_PACKAGE_SQLITE=y # BR2_PACKAGE_SQLITE_READLINE is not set # BR2_PACKAGE_SQLITE_STAT3 is not set # BR2_PACKAGE_SQLITE_ENABLE_FTS3 is not set # BR2_PACKAGE_SQLITE_ENABLE_UNLOCK_NOTIFY is not set # BR2_PACKAGE_SQLITE_SECURE_DELETE is not set # BR2_PACKAGE_SQLITE_NO_SYNC is not set Relevant build log snippet (python-2.7.8-build.log:2878): Python build finished, but the necessary bits to build these modules were not found: _bsddb _sqlite3 _ssl _tkinter bsddb185 dbm dl gdbm imageop linuxaudiodev nis ossaudiodev sunaudiodev zlib To find the necessary bits, look in setup.py in detect_modules() for the module's name. After booting the target, importing these modules will result in an ImportError: # python -c "import sqlite3" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/sqlite3/__init__.py", line 24, in <module> from dbapi2 import * File "/usr/lib/python2.7/sqlite3/dbapi2.py", line 28, in <module> from _sqlite3 import * ImportError: No module named _sqlite3 Note: sqlite3 library is installed on the target and the sqlite3 shell is working as excepted. # python -c "import ssl" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/ssl.py", line 60, in <module> import _ssl # if we can't import it, let the error propagate ImportError: No module named _ssl As a temporary workaround, copying the sqlite header file in the python include directory before building python permits the module compilation (here for sqlite): $ cp output/build/sqlite-3080500/sqlite3.h output/build/python-2.7.8/Include/ For the other modules, I haven't tested.