Bug 14701

Summary:	Multiple unresolved symbols while building kernel module package
Product:	buildroot	Reporter:	José Pekkarinen <jose.pekkarinen>
Component:	Other	Assignee:	unassigned
Status:	RESOLVED INVALID
Severity:	normal	CC:	buildroot
Priority:	P5
Version:	2021.11.1
Target Milestone:	---
Hardware:	All
OS:	Linux
Host:		Target:
Build:

Description José Pekkarinen 2022-03-28 06:11:42 UTC

Hi,

I'm trying to build a package for sgx driver:

https://github.com/intel/linux-sgx

The package looks like:

sgx-driver.mk:

################################################################################
#
# sgx-driver
#
################################################################################

SGX_DRIVER_VERSION = 2.14
SGX_DRIVER_SOURCE = sgx_driver_$(SGX_DRIVER_VERSION).tar.gz
SGX_DRIVER_SITE = https://github.com/intel/linux-sgx-driver/archive/refs/tags
SGX_DRIVER_LICENSE = BSD-3-Clause, GPL-2.0
SGX_DRIVER_LICENSE_FILES = License.txt

$(eval $(kernel-module))
$(eval $(generic-package))

Config.in:

comment "sgx-driver needs a Linux kernel to be built"
        depends on !BR2_LINUX_KERNEL

config BR2_PACKAGE_SGX_DRIVER
        bool "sgx-driver"
        depends on BR2_LINUX_KERNEL
        help
          Intel sgx driver for secured containers.

          https://github.com/intel/linux-sgx-driver

When building with a valid config, the build process
stops like this:

>>> sgx-driver 2.14 Building kernel module(s)
PATH="/usr/src/output/br_admin/output_x86_upxtreme/host/bin:/usr/src/output/br_admin/output_x86_upxtreme/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" PKG_CONFIG="/usr/src/output/br_admin/output_x86_upxtreme/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/usr/src/output/br_admin/output_x86_upxtreme/host/lib/pkgconfig:/usr/src/output/br_admin/output_x86_upxtreme/host/share/pkgconfig" BR_BINARIES_DIR=/usr/src/output/br_admin/output_x86_upxtreme/images KCFLAGS=-Wno-attribute-alias /usr/bin/make -j9 -C /usr/src/output/br_admin/output_x86_upxtreme/build/linux-tc-x86-5.15-sec HOSTCC="/usr/src/output/br_admin/output_x86_upxtreme/host/bin/ccache /usr/bin/gcc -O2 -I/usr/src/output/br_admin/output_x86_upxtreme/host/include -L/usr/src/output/br_admin/output_x86_upxtreme/host/lib -Wl,-rpath,/usr/src/output/br_admin/output_x86_upxtreme/host/lib" ARCH=x86_64 INSTALL_MOD_PATH=/usr/src/output/br_admin/output_x86_upxtreme/target CROSS_COMPILE="/usr/src/output/br_admin/output_x86_upxtreme/host/bin/x86_64-TII-linux-gnu-" DEPMOD=/usr/src/output/br_admin/output_x86_upxtreme/host/sbin/depmod INSTALL_MOD_STRIP=1  PWD=/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/. M=/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/. modules
make[2]: Entering directory '/usr/src/output/br_admin/output_x86_upxtreme/build/linux-tc-x86-5.15-sec'
  CC [M]  /usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./sgx_main.o
  CC [M]  /usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./sgx_page_cache.o
  CC [M]  /usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./sgx_ioctl.o
  CC [M]  /usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./sgx_vma.o
  CC [M]  /usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./sgx_util.o
  CC [M]  /usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./sgx_encl.o
  CC [M]  /usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./sgx_encl2.o
  LD [M]  /usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.o
  MODPOST /usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./Module.symvers
ERROR: modpost: "flush_work" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
ERROR: modpost: "put_pid" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
ERROR: modpost: "up_read" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
ERROR: modpost: "boot_cpu_data" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
ERROR: modpost: "system_freezing_cnt" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
ERROR: modpost: "mmu_notifier_register" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
ERROR: modpost: "__warn_printk" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
ERROR: modpost: "__x86_indirect_alt_call_rbx" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
ERROR: modpost: "__x86_indirect_thunk_rcx" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
ERROR: modpost: "alloc_pages" [/usr/src/output/br_admin/output_x86_upxtreme/build/sgx-driver-2.14/./isgx.ko] undefined!
WARNING: modpost: suppressed 65 unresolved symbol warnings because there were too many)
WARNING: modpost: modpost: Found 2 writable function pointers.

While the symbols can be found in the original
source code of the kernel. Am I missing something
in the package to resolv the symbols?

Thanks!

José.

Comment 1 José Pekkarinen 2022-03-29 06:07:59 UTC

Is there any way I can set in buildroot KBUILD_MODPOST_WARN?
According to the kernel documentation it will set all errors
as warning, and this is for testing purposes only, so no matter
if it boils out.

Comment 2 José Pekkarinen 2022-03-29 06:21:17 UTC

as a workaround I have added to the package the following line:

SGX_DRIVER_MAKE_OPTS=KBUILD_MODPOST_WARN=1

That allows to build and test stuff. I'm uncertain this is a
kernel bug instead of a buildroot one by now.

Comment 3 José Pekkarinen 2022-03-29 07:23:20 UTC

This is the list of "missing" symbols:

depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __put_page
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __check_object_size
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol vmf_insert_pfn
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol misc_deregister
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol alloc_workqueue
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol radix_tree_insert
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __x86_indirect_alt_call_rcx
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol platform_driver_unregister
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol on_each_cpu_cond_mask
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol queue_work_on
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol stackleak_track_stack
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol _find_first_bit
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol radix_tree_lookup
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __x86_indirect_alt_call_rax
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol finish_wait
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol ioremap_cache
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol iounmap
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol _find_first_zero_bit
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol set_freezable
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol zap_vma_ptes
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol ex_handler_default
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol prepare_to_wait_event
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol kthread_should_stop
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __wake_up
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol _raw_spin_lock
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol msleep_interruptible
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol wake_up_process
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __refrigerator
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __x86_indirect_thunk_rax
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol _printk
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol schedule
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol refcount_warn_saturate
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol smp_call_function
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __free_pages
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol _dev_info
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol find_vma
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol page_offset_base
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __cond_resched
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol radix_tree_delete
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol init_wait_entry
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol shmem_file_setup
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __list_add_valid
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol fput
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol radix_tree_next_chunk
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol platform_device_unregister
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol destroy_workqueue
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __x86_indirect_thunk_rbx
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol shmem_read_mapping_page_gfp
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol apply_to_page_range
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol mmu_notifier_unregister
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __cpu_online_mask
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __list_del_entry_valid
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol kthread_stop
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __mutex_init
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol freezing_slow_path
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol current_task
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol misc_register
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol _copy_to_user
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol down_read
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __platform_driver_register
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol kthread_create_on_node
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol set_page_dirty
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol vmemmap_base
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __ubsan_handle_shift_out_of_bounds
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol platform_device_register_full
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol alloc_pages
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __x86_indirect_thunk_rcx
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __x86_indirect_alt_call_rbx
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol __warn_printk
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol mmu_notifier_register
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol system_freezing_cnt
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol boot_cpu_data
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol up_read
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol put_pid
depmod: WARNING: /usr/src/output/br_admin/output_x86_upxtreme/target/lib/modules/5.15.21-hardened1/extra/isgx.ko needs unknown symbol flush_work

Comment 4 José Pekkarinen 2022-03-30 05:37:38 UTC

I want to highlight that this may be a buildroot issue, and not
a driver, or a linux one. This is because I tried the same kernel
version on my laptop:

$ uname -r
5.15.21-hardened1

And this is the building output of the module:

 make
make -C /lib/modules/5.15.21-hardened1/build M=/home/pekkari/linux-sgx-driver-sgx_driver_2.14 modules
make[1]: Entering directory '/usr/src/linux-headers-5.15.21-hardened1'
  CC [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/sgx_main.o
  CC [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/sgx_page_cache.o
  CC [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/sgx_ioctl.o
  CC [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/sgx_vma.o
  CC [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/sgx_util.o
  CC [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/sgx_encl.o
  CC [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/sgx_encl2.o
  LD [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/isgx.o
  MODPOST /home/pekkari/linux-sgx-driver-sgx_driver_2.14/Module.symvers
WARNING: modpost: modpost: Found 2 writable function pointers.
To see full details build your kernel with:
'make CONFIG_DEBUG_WRITABLE_FUNCTION_POINTERS_VERBOSE=y'
  CC [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/isgx.mod.o
  LD [M]  /home/pekkari/linux-sgx-driver-sgx_driver_2.14/isgx.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.15.21-hardened1'

and installing and loading the module I can see in dmesg:

[  233.145965] isgx: loading out-of-tree module taints kernel.
[  233.146006] isgx: module verification failed: signature and/or required key missing - tainting kernel
[  233.146559] intel_sgx: the CPU is missing SGX

So symbols are in place, kernel is not wrong, module either, but
buildroot configures the linker in a way the symbols are not found.

Thanks!

Comment 5 José Pekkarinen 2022-04-07 05:18:13 UTC

Kernel was trimminig unused symbols, removing the config
to trim them makes the module build properly. Sorry!