Using cpanm to Install Perl Modules in a Conda Environment

As a devoted Perl programmer, I find myself making heavy use of the fantastic Perl module repository, CPAN. And I do my best to keep up with all the latest goings-on with Perl.

I also have recently gotten hooked on conda, a neat tool which combines package management and environment management. If you haven't checked that out yet, I suggest you give it a look to see if it suits your needs.

TL;DR

(Full post after this tl;dr)

# envname is the name of your (existing) conda env
conda activate envname
# cpanm is available on either conda-forge or
# bioconda, so add conda-forge (or look up
# how to set up bioconda if you need tools
# for bioinformatics)
conda config --add channels conda-forge
# Install cpanm
conda install perl-app-cpanminus
# packagename is the name of the Perl package you want
env PERL5LIB="" PERL_LOCAL_LIB_ROOT="" PERL_MM_OPT="" PERL_MB_OPT="" $CONDA_PREFIX/bin/cpanm packagename

If that didn't work:

# Make sure gcc is installed
# Replace gcc with gxx if you need to compile C++
conda install -n envname gcc_impl_linux-64
conda activate envname
declare perlarchloc=$($CONDA_PREFIX/bin/perl -V::installarchlib: | sed "s/[' ]//g")
for conffile in "Config_heavy.pl" "CORE/config.h" "Config.pm"
do
    sed -i -E 's%/tmp/build/[a-zA-Z0-9]+/perl_[0-9]+/_build_env%'"$CONDA_PREFIX"'%g' "$perlarchloc"/"$conffile"
done
unset perlarchloc

If that still didn't work, jump to the last section.

/tl;dr

Installing Perl and Python Packages

Conda is written in Python, and I think because of that there is a slight bias towards things in Python working well. Case in point, installing a python module which hasn't already been packaged in a conda channel is relatively simple:

# envname is the name of your (existing) conda env
conda activate envname
# Make sure pip is installed
conda install pip
# packagename is the name of the python package you want
pip install packagename

Cool. Now, how about a Perl package?

# envname is the name of your (existing) conda env
conda activate envname
# cpanm is available on either conda-forge or
# bioconda, so add conda-forge (or look up
# how to set up bioconda if you need tools
# for bioinformatics)
conda config --add channels conda-forge
# Install cpanm
conda install perl-app-cpanminus
# packagename is the name of the Perl package you want
cpanm packagename

If this worked for you, great! If it didn't, well let's see what could have gone wrong.

Problem 1: local::lib or other environment variables

A lot of us like to use local::lib for installing Perl packages locally, without needing to mess with system-level Perl. This works great in general. Not so much when you're letting conda take care of things.

There are a few reasons you want to use conda, and if you're reading this, you probably already know why (go ahead and look up more about conda if you really don't know how useful it can be). And as such, it's preferable for you to install your Perl modules for an environment only in that environment.

Maybe your environment uses a different version of perl than your system (or another environment). Maybe you share the conda environment with many users, and you want any user of it to have access to some specific module(s).

Unfortunately, having local::lib set up might mean that the cpanm command from earlier installed the module only for you, outside of the conda env:

$ perldoc -l packagename
$HOME/perl5/lib/perl5/packagename/packagename.pm

Ugh.

To avoid this, and any other environment variables that affect Perl modules from interfering, you might try this:

env PERL5LIB="" PERL_LOCAL_LIB_ROOT="" PERL_MM_OPT="" PERL_MB_OPT="" $CONDA_PREFIX/bin/cpanm packagename

The above worked for me, and its a bit ugly but I needed a solution in the moment. If you know of a better way, let me know in the comments and I'll update the post.

Problem 2: Modules using XS

It turns out that installing a module that uses XS (as many modules that need the speed of C, or access to Perl internals do) in this way does not work. It's a known bug in conda, too.

What happens is that the the configuration for cpanm retains some paths relative to the directory in which it was built, rather than to the conda environment (as you'll see in the last comment in that bug report).

That comment also explains how to fix the issue, but I will reproduce that solution here, a bit more generalized (this assumes the version of Perl you installed is 5.26.2, so make sure you check that first):

# Make sure gcc is installed
# Replace gcc with gxx if you need to compile C++
conda install -n envname gcc_impl_linux-64
conda activate envname
declare perlarchloc=$($CONDA_PREFIX/bin/perl -V::installarchlib: | sed "s/[' ]//g")
for conffile in "Config_heavy.pl" "CORE/config.h" "Config.pm"
do
    sed -i -E 's%/tmp/build/[a-zA-Z0-9]+/perl_[0-9]+/_build_env%'"$CONDA_PREFIX"'%g' "$perlarchloc"/"$conffile"
done
unset perlarchloc

Now just rerun your cpanm command(s) and that should work! Unless...

Problem 2a: compilerroot?!

Turns out that on some configurations, a variable, $compilerroot, is used to set the location of the compiler in the files above. It's not easy to automate (for me), so I suggest you manually open the "Config_heavy.pl" and "Config.pm" files under "$CONDA_PREFIX"/lib/5.26.2/x86_64-linux-thread-multi/ and look for the following line:

my $compilerroot = "";
# Some other code is here

Change that assignment, and remove the if/else code following that line that conditionally sets $compilerroot too. Make sure you use the correct path for your environment's conda prefix (ie, the value of $CONDA_PREFIX when your environment is active):

my $compilerroot = "/your/conda/env/prefix";

# Don't change this code below
local *_ = \my $a;

Now try using cpanm :).

I hope this helped someone. It took me a few hours before I figured this out. In retrospect, it might have been easier to just package the missing perl modules for conda myself!

Edit: Feb 20, 2021. Updated the code block showing how to change the paths in the conda perl headers. Now instead of the perl version being hardcoded, use the location of the arch lib location given by the conda perl installation. Also updated post so that call to conda cpanm use the $CONDA_PREFIX environment variable.