Buildroot seems to have NEON as a hard dependency for vc4, this prevents bcm2835 based targets to not build with vc4 mesa enabled. However as it turns out, it doesn't require neon, but it does have some run-time enabled neon fast paths. So it is completely possible to build this for bcm2835 boards.
Fixed by https://git.buildroot.org/buildroot/commit/?id=350cb0d32ece533b9723a5f3ca6fbf7e6f071c90. Thanks!