Bug 7268

Summary: python 2.7 compilation issue on a Debian/Ubuntu host with multiarch support
Product: buildroot Reporter: David Bachelart <david.bachelart>
Component: OtherAssignee: unassigned
Status: RESOLVED FIXED    
Severity: normal CC: buildroot, yann.morin.1998
Priority: P5    
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Host: Target:
Build:
Attachments: buildroot configuration
Make output
manage multiarch paths only if not cross-compiling

Description David Bachelart 2014-07-07 16:33:49 UTC
Hi,

I am trying to build python for an x86_64 target. 

I am using fresh sources of buildroot (exactly the git revision 5ac8f8ce10027ac8285a7e9cec16e74aa38ee2c1).

To achieve that, I go through following steps :
- make defconfig
- in Target Options, change Target Architecture to x86_64 / generic
- in Toolchain, change C Library to glibc
- in Target Packages / Interpreter languages and scripting, enable python
- make

The following tarball encloses configuration file and make output.

It seems host part of python build goes well. Target part, however, fails building "struct" module (I'm not a python expert... so I may be wrong).

Thanks
Comment 1 David Bachelart 2014-07-07 17:18:26 UTC
Created attachment 5486 [details]
buildroot configuration
Comment 2 David Bachelart 2014-07-07 17:24:42 UTC
Created attachment 5492 [details]
Make output
Comment 3 David Bachelart 2014-07-07 17:26:24 UTC
Hi,

if needed, here is the full make output (5MB, but 797MB uncompressed) :
http://dl.free.fr/eMphhZaBN
Comment 4 Thomas Petazzoni 2014-07-07 17:32:16 UTC
Thanks for the report, I've just started a build with your configuration to try to reproduce the problem.
Comment 5 Thomas Petazzoni 2014-07-07 17:47:44 UTC
The build of your .config works fine here (Debian GNU/Linux 6.0).
Comment 6 Thomas Petazzoni 2014-07-07 19:18:57 UTC
Ok, so the problem is due to the fact that for some reason in your case, Python has added "-I/usr/include/x86_64-linux-gnu" to the CFLAGS used when building target modules, which doesn't make sense. I don't have this flag added when building here, which explains why it works.
Comment 7 Thomas Petazzoni 2014-07-07 19:20:25 UTC
(In reply to comment #3)
> Hi,
> 
> if needed, here is the full make output (5MB, but 797MB uncompressed) :
> http://dl.free.fr/eMphhZaBN

Unfortunately, I'm not able to download it. Could you put it somewhere else?

Thanks.
Comment 8 David Bachelart 2014-07-07 20:38:25 UTC
Thanks for your reply

Here is the download link :
http://we.tl/vSYk0yUI5r

I forgot to say : my host is a Debian 7.5

I'll try to remove this flag tomorrow morning

Regards
Comment 9 David Bachelart 2014-07-08 11:51:23 UTC
Hi,

A few more details about this issue...

A git bisect identifies 7e960dc9 as the first "bad" commit : I manage to compile python using revision 577e52ac, but I do not using revision 7e960dc9.

7e960dc9da56d4a484b5480746aaf617ca491274 is the first bad commit
commit 7e960dc9da56d4a484b5480746aaf617ca491274
Author: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Date:   Thu Feb 13 21:48:53 2014 +0100

    python: bump to 2.7.6
    
    Even though jumping from 2.7.3 to 2.7.6 looks like a minor version
    bump, it is in fact a fairly significant one, because a good number of
    changes to help cross-compilation have been merged into Python
    upstream. Therefore, most of our patches are affected by this change.
    
    In detail, this commit:
    
     * Renames all the patches to follow the naming convention of patches
       in Buildroot: the patch file names should not have any version
       number.
    
     * The patches numbered above 100, that add configuration options to
       disable certain modules of the Python standard library, are only
       renamed and slightly adapted, they didn't change that much.
    
     * The patches numbered below 100 are almost entirely rewritten: many
       of the cross-compilation problems that used to exist in Python
       2.7.3 no longer exist, and the number of remaining problems is
       smaller, and can be fixed with smaller patches.
    
    Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
    Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

:040000 040000 8618414254d7998096c5c87b7774cbd50217cb08 ddcec6a08dbe50e52973f7659789200593fe6861 M	package
Comment 10 David Bachelart 2014-07-08 17:04:14 UTC
Hi,

I finally got the bug. Your input about wrong paths really helped me and saved me a lot of time, thanks a lot for this.

In fact, the main setup.py script has a "add_multiarch_paths" function that adapts standard paths to multiarch requirements. For some reason, it adds host /usr/include/x86_64-linux-gnu to include paths for the target python build. 

I add a patch that makes multiarch management process available only if python is not being cross-compiled. I do not know if multiarch really makes sense in the case of buildroot. This patch fixes my compilation issue, but it may break something for a different architecture, so I hope someone with a lot more experience about python and buildroot architectures than me will read this patch.

Thanks a lot again
David
Comment 11 David Bachelart 2014-07-08 17:05:20 UTC
Created attachment 5504 [details]
manage multiarch paths only if not cross-compiling
Comment 12 Yann E. MORIN 2014-07-08 17:19:34 UTC
David,

Your patch is reversed. ;-)

Care to send the patch to the list, please?

Thanks.
Comment 13 Thomas Petazzoni 2014-07-08 20:20:50 UTC
Patch for python posted at http://patchwork.ozlabs.org/patch/368000/, and patch for python3 posted at http://patchwork.ozlabs.org/patch/368001/.

Thanks a lot David for your bug report and investigation! Once the patches are merged by Peter, we'll mark this bug as fixed. Of course, it would be good if you could formally test the patches, and give your Tested-by on the patches if their work for you.
Comment 14 David Bachelart 2014-07-09 08:56:48 UTC
Hi Thomas,

I managed to compile python 2.7 and python 3 using your patches.

Please forgive me, I'm not familiar with the "Tested-by" thing... How would you do that ?

Thanks
David
Comment 15 Thomas Petazzoni 2014-07-09 11:26:37 UTC
(In reply to comment #14)
> Hi Thomas,
> 
> I managed to compile python 2.7 and python 3 using your patches.
> 
> Please forgive me, I'm not familiar with the "Tested-by" thing... How would you
> do that ?

You would have had to reply to the patches sent to the mailing list, and just add a line like:

Tested-by: John Doe <john.doe@foobar.org>

In the reply.

However, the patches have been committed now, so it's too late :-)