Relative dir for local::lib doesn't work

[averms]

I have Perl 5.34 with local::lib installed through my Linux package manager. The following variables are exported

export PATH="perl_modules/bin${PATH:+:${PATH}}"
export PERL5LIB="perl_modules/lib/perl5"
export PERL_LOCAL_LIB_ROOT="perl_modules"
export PERL_MB_OPT="--install_base perl_modules"
export PERL_MM_OPT="INSTALL_BASE=perl_modules"

This is so I can have a perl_modules folder relative to my project folder, analagous to npm and node_modules. However, Installing anything with cpan installs to a perl_modules directory inside the build folder, i.e., ~/.cpan/build/CPAN-2.29-0/perl_modules.