Bug 9

Summary: uClibc dynamic loader broken when PAGE_SIZE != 4KB
Product: uClibc Reporter: Stuart Longland <redhatter>
Component: Shared Library SupportAssignee: unassigned
Status: CLOSED FIXED    
Severity: normal CC: aldot, rep.dot.nop, solar
Priority: P5    
Version: 0.9.30.1   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Host: mipsel-unknown-linux-uclibc Target: mipsel-unknown-linux-uclibc
Build: i686-pc-linux-gnu

Description Stuart Longland 2008-12-27 01:58:27 UTC
Hi,

This is a re-post of a bug that used to be on the old Mantis-based bug tracker, that was identified as affecting mips, powerpc and possibly other architectures as well.

It seems when uClibc tries to load a library, it calls mmap, specifying an offset that violates the host's PAGE_SIZE setting when the PAGE_SIZE is set to something other than 4KB.  This problem is acute on systems such as the Lemote Fulong, where a page size of 16KB is a hardware requirement.

Test Case:

Cross-toolchain was produced using 'crossdev' on Gentoo Linux.  Toolchain versions used:

o gcc-4.1.2
o binutils-2.19
o uClibc-0.9.30
o linux-headers-2.6.27-r2

Cross-compiled ncurses and bash to /usr/mipsel-unknown-linux-uclibc/ then copied the whole SYSROOT to the mipsel target (/home/uclibc directory) using rsync.  Target is a Lemote Fulong minicomputer, running 660MHz Loongson 2E CPU, 1GB RAM, Linux 2.6.23 kernel.

Attempted to chroot into the minimalist environment:
taijia ~ # chroot /home/uclibc/ bin/bash
bin/bash:572: can't map '/lib/libncurses.so.5'
bin/bash:572: can't map '/lib/libncurses.so.5'
bin/bash:572: can't map '/lib/libncurses.so.5'
bin/bash: can't load library 'libncurses.so.5'

Under strace, the following is seen:
taijia ~ # strace chroot /home/uclibc/ bin/bash                                                                                                
execve("/usr/bin/chroot", ["chroot", "/home/uclibc/", "bin/bash"], [/* 43 vars */]) = 0                                                        
svr4_syscall()                          = -1 ERRNO_4045 (Unknown error 4045)                                                                   
uname({sys="Linux", node="taijia", ...}) = 0                                                                                                   
old_mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aac8000                                                     
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)                                                                
open("/etc/ld.so.cache", O_RDONLY)      = 3                                                                                                    
fstat64(3, {st_mode=S_IFREG|0644, st_size=145099, ...}) = 0                                                                                    
old_mmap(NULL, 145099, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2aacc000                                                                              
close(3)                                = 0                                                                                                    
open("/usr/lib/gcc/mipsel-unknown-linux-gnu/4.1.1/libgcc_s.so.1", O_RDONLY) = 3                                                                
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\10\0\1\0\0\0@\32\0"..., 512) = 512                                                                
fstat64(3, {st_mode=S_IFREG|0644, st_size=78076, ...}) = 0                                                                                     
old_mmap(NULL, 330240, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2ab08000                                                      
mprotect(0x2ab1c000, 245760, PROT_NONE) = 0                                                                                                    
old_mmap(0x2ab58000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x10000) = 0x2ab58000                                
close(3)                                = 0                                                                                                    
open("/lib/libc.so.6", O_RDONLY)        = 3                                                                                                    
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\10\0\1\0\0\0\344c\1"..., 512) = 512                                                               
lseek(3, 708, SEEK_SET)                 = 708                                                                                                  
read(3, "\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0"..., 32) = 32                                                                  
fstat64(3, {st_mode=S_IFREG|0755, st_size=1742711, ...}) = 0                                                                                   
old_mmap(NULL, 1674208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2ab5c000                                                     
mprotect(0x2acac000, 245760, PROT_NONE) = 0                                                                                                    
old_mmap(0x2ace8000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14c000) = 0x2ace8000                               
old_mmap(0x2acf4000, 3040, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2acf4000                                      
close(3)                                = 0                                                                                                    
set_thread_area(0x2aad0330)             = 0                                                                                                    
mprotect(0x2ace8000, 32768, PROT_READ)  = 0                                                                                                    
munmap(0x2aacc000, 145099)              = 0                                                                                                    
brk(0)                                  = 0x448000                                                                                             
brk(0x46c000)                           = 0x46c000                                                                                             
chroot("/home/uclibc/")                 = 0                                                                                                    
chdir("/")                              = 0                                                                                                    
execve("bin/bash", ["bin/bash"], [/* 43 vars */]) = 0                                                                                          
svr4_syscall()                          = 4005                                                                                                 
svr4_fork()                             = -1 ERRNO_4078 (Unknown error 4078)                                                                   
svr4_syscall()                          = -1 ERRNO_4090 (Unknown error 4090)                                                                   
stat("/etc/ld.so.cache", {st_mode=S_IFREG|0644, st_size=2636, ...}) = 0                                                                        
open("/etc/ld.so.cache", O_RDONLY)      = 3                                                                                                    
old_mmap(NULL, 2636, PROT_READ, MAP_SHARED, 3, 0) = 0x2aab4000                                                                                 
close(3)                                = 0                                                                                                    
open("/lib/libncurses.so.5", O_RDONLY)  = 3                                                                                                    
fstat(3, {st_mode=S_IFREG|0755, st_size=300444, ...}) = 0                                                                                      
old_mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aab8000                                                     
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\10\0\1\0\0\0\200\207"..., 16384) = 16384                                                          
old_mmap(NULL, 348160, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aac0000                                                               
old_mmap(0x2aac0000, 272316, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0x2aac0000                                                    
old_mmap(0x2ab12000, 10740, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x42000) = -1 EINVAL (Invalid argument)                            
old_mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab18000                                                     
write(2, "", 0)                         = 0                                                                                                    
write(2, "bin/bash", 8bin/bash)                 = 8                                                                                            
write(2, ":", 1:)                        = 1                                                                                                   
write(2, "572", 3572)                      = 3                                                                                                 
write(2, ": can\'t map \'", 13: can't map ')         = 13                                                                                      
write(2, "/lib/libncurses.so.5", 20/lib/libncurses.so.5)    = 20                                                                               
write(2, "\'\n", 2'                                                                                                                            
)                     = 2                                                                                                                      
munmap(0x2ab18000, 16384)               = 0                                                                                                    
munmap(0x2aac0000, 348160)              = 0                                                                                                    
close(3)                                = 0                                                                                                    
munmap(0x2aab8000, 16384)               = 0                                                                                                    
open("/lib/libncurses.so.5", O_RDONLY)  = 3                                                                                                    
fstat(3, {st_mode=S_IFREG|0755, st_size=300444, ...}) = 0                                                                                      
old_mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aab8000                                                     
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\10\0\1\0\0\0\200\207"..., 16384) = 16384                                                          
old_mmap(NULL, 348160, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aac0000                                                               
old_mmap(0x2aac0000, 272316, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0x2aac0000                                                    
old_mmap(0x2ab12000, 10740, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x42000) = -1 EINVAL (Invalid argument)                            
old_mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab18000                                                     
write(2, "", 0)                         = 0                                                                                                    
write(2, "bin/bash", 8bin/bash)                 = 8                                                                                            
write(2, ":", 1:)                        = 1                                                                                                   
write(2, "572", 3572)                      = 3                                                                                                 
write(2, ": can\'t map \'", 13: can't map ')         = 13                                                                                      
write(2, "/lib/libncurses.so.5", 20/lib/libncurses.so.5)    = 20
write(2, "\'\n", 2'
)                     = 2
munmap(0x2ab18000, 16384)               = 0
munmap(0x2aac0000, 348160)              = 0
close(3)                                = 0
munmap(0x2aab8000, 16384)               = 0
open("/lib/libncurses.so.5", O_RDONLY)  = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=300444, ...}) = 0
old_mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aab8000
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\10\0\1\0\0\0\200\207"..., 16384) = 16384
old_mmap(NULL, 348160, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aac0000
old_mmap(0x2aac0000, 272316, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0x2aac0000
old_mmap(0x2ab12000, 10740, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x42000) = -1 EINVAL (Invalid argument)
old_mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab18000
write(2, "", 0)                         = 0
write(2, "bin/bash", 8bin/bash)                 = 8
write(2, ":", 1:)                        = 1
write(2, "572", 3572)                      = 3
write(2, ": can\'t map \'", 13: can't map ')         = 13
write(2, "/lib/libncurses.so.5", 20/lib/libncurses.so.5)    = 20
write(2, "\'\n", 2'
)                     = 2
munmap(0x2ab18000, 16384)               = 0
munmap(0x2aac0000, 348160)              = 0
close(3)                                = 0
munmap(0x2aab8000, 16384)               = 0
open("/usr/lib/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
old_mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aab8000
write(2, "", 0)                         = 0
write(2, "bin/bash", 8bin/bash)                 = 8
write(2, ": can\'t load library \'", 22: can't load library ') = 22
write(2, "libncurses.so.5", 15libncurses.so.5)         = 15
write(2, "\'\n", 2'
)                     = 2
munmap(0x2aab8000, 16384)               = 0
exit(16)                                = ?
Process 14292 detached

The lines of interest seem to be these ones:
old_mmap(0x2ab12000, 10740, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x42000) = -1 EINVAL (Invalid argument)

According to the mmap(2) manpage:
SYNOPSIS
       #include <sys/mman.h>

       void *mmap(void *addr, size_t length, int prot, int flags,
                  int fd, off_t offset);
       int munmap(void *addr, size_t length);

DESCRIPTION
[...]
       The contents of a file mapping (as opposed to an anonymous mapping; see
       MAP_ANONYMOUS  below),  are  initialized using length bytes starting at
       offset offset in the file (or other object) referred  to  by  the  file
       descriptor  fd.  offset must be a multiple of the page size as returned
       by sysconf(_SC_PAGE_SIZE).

Now... 0x42000, divided by 4096 gives 66, which works fine.  0x42000 divided by 16384 however, gives 16.5, hence the -EINVAL return value.

I'll try these binaries on another host (with a 4KB page size) in a moment, but for now, it is broken on systems with a 16KB or larger page size.
Comment 1 Bernhard 2009-01-09 12:25:10 UTC
I believe this is fixed on the 0_9_30 branch. Please confirm.
Comment 2 Stuart Longland 2009-01-10 04:28:21 UTC
(In reply to comment #1)
> I believe this is fixed on the 0_9_30 branch. Please confirm.
> 

Well, either the 0_9_30 CVS branch and the 0.9.30 tarball are different, or something was missed, because my Fulong begs to differ:

taijia ~ # ls -l /home/uclibc/lib/libc.so.0
lrwxrwxrwx 1 root root 19 Dec 27 11:49 /home/uclibc/lib/libc.so.0 -> libuClibc-0.9.30.so
taijia ~ # chroot /home/uclibc/
/bin/bash:572: can't map '/lib/libncurses.so.5'
/bin/bash:572: can't map '/lib/libncurses.so.5'
/bin/bash:572: can't map '/lib/libncurses.so.5'
/bin/bash: can't load library 'libncurses.so.5'
taijia ~ # uname -a
Linux taijia 2.6.23.14-mipsgit-20080206-lm2e #1 Wed Feb 20 23:28:06 EST 2008 mips64 ICT Loongson-2 V0.2  FPU V0.1 lemote-fulong GNU/Linux
taijia ~ # zgrep PAGE_SIZE /proc/config.gz
# CONFIG_PAGE_SIZE_4KB is not set
# CONFIG_PAGE_SIZE_8KB is not set
CONFIG_PAGE_SIZE_16KB=y
# CONFIG_PAGE_SIZE_64KB is not set
Comment 3 Bernhard 2009-01-10 11:41:29 UTC
> Well, either the 0_9_30 CVS branch and the 0.9.30 tarball are different, or
> something was missed, because my Fulong begs to differ:

Did you use the branch or the 0.9.30 release tarball?

If the latter, see comment 1
Please clarify which svn revision of the branch you're using.
TIA,
Comment 4 Bernhard Reutner-Fischer 2009-03-02 23:55:04 UTC
Fixed in 0.9.30.1, closing.