Bug 11831

Summary:	python-pyparsing: "No module named 'pyparsing'" while building cantools (SOLVED: use host-python-pyparsing)
Product:	buildroot	Reporter:	Tibor Stolz <tistolz>
Component:	Other	Assignee:	Tibor Stolz <tistolz>
Status:	RESOLVED WORKSFORME
Severity:	normal	CC:	buildroot
Priority:	P5
Version:	2019.02.2
Target Milestone:	---
Hardware:	All
OS:	Linux
Host:		Target:
Build:
Attachments:	python-cantools/Config.in python-cantools/python-cantools.hash python-cantools/python-cantools.mk python-cantools/Config.in python-cantools/python-cantools.hash python-cantools/python-cantools.mk python_ok_bbb_defconfig FIXED python-cantools/python-cantools.mk FIXED python-cantools/Config.in

Description Tibor Stolz 2019-05-03 13:52:22 UTC

Created attachment 7991 [details]
python-cantools/Config.in

Hi all,

I'm trying to include the Python module "cantools" <https://pypi.org/project/cantools/> into my Buildroot installation. The package files are given in the attachment. However, the build step for python-cantools fails with the message:
    ModuleNotFoundError: No module named 'pyparsing'
in a nested import while processing the first line (import cantools) of setup.py.
I can confirm that the python-pyparsing package has been built before, so it should be available to python.

Trying to track this bug down, I first started output/host/bin/python3 and entered the command "import pyparsing", upon which I saw the same error message. However, another package I built was python-bitstruct (using utils/scanpypi), and this became available for "import bitstruct" in an output/host/bin/python3 session.

Furthermore, I rebuilt Buildroot 2019.02.2 (from a freshly extracted tar archive) with only the following packages selected:
    Interpreter languages and scripting --> python3 SELECTED
        core python3 modules: DESELECTED unicodedata
        External python modules: SELECTED python-can, python-spidev, python-pyparsing

On the target, in a python3 session, I get:
    >>> import can
    >>> import pyparsing
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'pyparsing'
    >>> import spidev
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'spidev'
    >>>

So, the "python-can" package seems to install correctly, while "python-pyparsing" and "python-spidev" appear to be not present on the target.

Thanks for your attention,
Tibor Stolz

Comment 1 Tibor Stolz 2019-05-03 13:54:04 UTC

Created attachment 7996 [details]
python-cantools/python-cantools.hash

Comment 2 Tibor Stolz 2019-05-03 13:54:37 UTC

Created attachment 8001 [details]
python-cantools/python-cantools.mk

Comment 3 Tibor Stolz 2019-05-03 13:55:59 UTC

Created attachment 8006 [details]
python-cantools/Config.in

same file, but set content type now correctly

Comment 4 Tibor Stolz 2019-05-03 13:58:42 UTC

Created attachment 8011 [details]
python-cantools/python-cantools.hash

same file, but content type now set correctly

Comment 5 Tibor Stolz 2019-05-03 13:59:57 UTC

Created attachment 8016 [details]
python-cantools/python-cantools.mk

same file, but content type set correctly

Comment 6 Tibor Stolz 2019-05-03 14:08:34 UTC

The build log for python-cantools in detail:

>>> python-cantools 18.0.1 Downloading
cantools-18.0.1.tar.gz: OK (sha256: dc5d265103b8b11d4c7cc21996ad1954325b1f1930837505cabb0b5b137d37ee)
>>> python-cantools 18.0.1 Extracting
gzip -d -c /home/tibor/Buildroot/buildroot-2019.02.2/dl/python-cantools/cantools-18.0.1.tar.gz | tar --strip-components=1 -C /home/tibor/Buildroot/buildroot-2019.02.2/output/build/python-cantools-18.0.1   -xf -
>>> python-cantools 18.0.1 Patching
>>> python-cantools 18.0.1 Configuring
>>> python-cantools 18.0.1 Building
(cd /home/tibor/Buildroot/buildroot-2019.02.2/output/build/python-cantools-18.0.1//; _PYTHON_SYSCONFIGDATA_NAME="_sysconfigdata_m_linux_arm-linux-gnueabihf" PATH="/home/tibor/Buildroot/buildroot-2019.02.2/output/host/bin:/home/tibor/Buildroot/buildroot-2019.02.2/output/host/sbin:/home/tibor/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin" PYTHONPATH="/home/tibor/Buildroot/buildroot-2019.02.2/output/target/usr/lib/python3.7/" PYTHONNOUSERSITE=1 _python_sysroot=/home/tibor/Buildroot/buildroot-2019.02.2/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot _python_prefix=/usr _python_exec_prefix=/usr  /home/tibor/Buildroot/buildroot-2019.02.2/output/host/bin/python setup.py build  )
Traceback (most recent call last):
  File "setup.py", line 4, in <module>
    import cantools
  File "/home/tibor/Buildroot/buildroot-2019.02.2/output/build/python-cantools-18.0.1/cantools/__init__.py", line 6, in <module>
    from . import db
  File "/home/tibor/Buildroot/buildroot-2019.02.2/output/build/python-cantools-18.0.1/cantools/db/__init__.py", line 4, in <module>
    from .database import Database
  File "/home/tibor/Buildroot/buildroot-2019.02.2/output/build/python-cantools-18.0.1/cantools/db/database.py", line 3, in <module>
    from .formats import dbc
  File "/home/tibor/Buildroot/buildroot-2019.02.2/output/build/python-cantools-18.0.1/cantools/db/formats/dbc.py", line 4, in <module>
    from pyparsing import Word
ModuleNotFoundError: No module named 'pyparsing'
package/pkg-generic.mk:238: recipe for target '/home/tibor/Buildroot/buildroot-2019.02.2/output/build/python-cantools-18.0.1/.stamp_built' failed
make[1]: *** [/home/tibor/Buildroot/buildroot-2019.02.2/output/build/python-cantools-18.0.1/.stamp_built] Error 1
Makefile:84: recipe for target '_all' failed
make: *** [_all] Error 2

Comment 7 Tibor Stolz 2019-05-07 14:59:05 UTC

Created attachment 8026 [details]
python_ok_bbb_defconfig

I'm sorry I cannot repeat the problem anymore. Using exactly the same buildroot version and menu settings (except "System hostname" and "System banner"), I get packages python-pyparsing and python-spidev that work fine and can be imported into a python3 session without any issue.

For your reference, I'm attaching the main config file, created using "make savedefconfig".

I'm closing the bug until I hit that behaviour again. (I hope not...)

Comment 8 Tibor Stolz 2019-05-14 11:55:58 UTC

Created attachment 8046 [details]
FIXED python-cantools/python-cantools.mk

Hi all,

The build error for python-cantools persisted, but now I figured out why I ran into it: In order to build correctly, the build-time dependencies must be installed on the *host*! Therefore, the package makefile (python-cantools.mk) must be changed:

- PYTHON_CANTOOLS_DEPENDENCIES = python-bitstruct python-pyparsing
+ PYTHON_CANTOOLS_DEPENDENCIES = host-python-bitstruct host-python-pyparsing

Doing a 'grep -r DEPENDENCIES package/python-*' in the buildroot-2019.02.2 directory revealed that most python library dependencies are in the host-python-* form. Maybe this should be mentioned in the Buildroot user manual?

Best regards,
Tibor Stolz

Comment 9 Tibor Stolz 2019-05-14 12:04:36 UTC

Created attachment 8051 [details]
FIXED python-cantools/Config.in

For anyone interested in building python-cantools v18.0.1 (16 Feb 2018), here is the Config.in file.

You also need packages python-bitstruct and python-pyparsing, which contain the following line at the end of the .mk file:
    $(eval $(host-python-package))
Besides that, the python-bitstruct package is easy to get with utils/scanpypi, and the python-pyparsing package is already available (with host-python-package missing).