Recommendations to fix broken Minicondas Installations on Ibex

Installation Location

It has come to our attention that some users have been installing Miniconda in their home directories and are having problems. While this is a common practice for many software packages, keeping in view the recent issues associated with it, we strongly recommend against installing Miniconda or Miniforge in your home directory.

Why Not Install it in the Home Directory?

Disk Space: Installing Miniconda in the home directory can consume a significant amount of disk space, which is limited on ibex. This can lead to storage issues and slow down your system.
Permissions: Home directories often have restrictive permissions, which can cause problems when Miniconda tries to create and manage conda environments and packages.
Speed: The home directory is moderately fast compared to the new filesystem on Ibex, the WekaIO. Conda package management requires supporting a large number of small files which can easily overwhelm the parallel shared filesystems. WekaIO, on the other hand, is a purpose-built filesystem that can support high IOP use cases like conda installation.

Alternative Recommendation

Instead of installing Miniconda in your home directory, we recommend using /ibex/user/$USER directory which is the WekaIO filesystem. This will help avoid the issues mentioned above and ensure better compatibility and performance. If you don’t have this directory, please email ibex@hpc.kaust.edu.sa for get one created for you. This directory will have a quota applied to it so use the space responsibly.

Python 3.11 Compatibility

The current Ibex OS is CentOS; as our last knowledge update in September 2021, CentOS 8 was the latest version available. Thus, Ibex users have been experiencing compatibility issues with Miniconda when running Python 3.11 (the latest version of Python).

There is an imminent upgrade of a more modern Operating System planned for Ibex. Schedule for this will be communicated to our user soon and has potential to fix this issue – testing is in progress.

To address these current issues with Miniconda, we suggest the following alternatives:

Downgrade Miniconda: If you have an existing installation of Miniconda, consider downgrading your conda version to 23.5.2
Check conda version:
conda --version
If you have a higher version installed, please downgrade to the recommended version.
conda install conda=23.5.2 python=3.9
Use WekaIO: A high-performance and scalable file system tailored for high IOPs workloads, WekaIO provides an ideal infrastructure for installing Miniconda, ensuring fast and reliable data access and management for data-intensive applications and workloads.

We appreciate your attention to these important installation guidelines and compatibility concerns. If you have any questions or require further assistance, please don't hesitate to contact us on ibex@hpc.kaust.edu.sa

Mambaforge installation on WekaIO - Example

This script automates the installation of Mambaforge (Provides same commands as Miniconda) with user-specified options, sets up environment variables, configures Mambaforge behavior, and provides instructions for activating Mambaforge. It is designed for use in environments where Conda package management is needed. On ibex /ibex/user/$USER, you will find your folder for the WekaIO partition.

# Define the log file path
log_file="/ibex/user/$USER/mambaforge_installation.log"

# Function to log messages
log() {
    local message="$1"
    logger -t "Mambaforge installation" "$message"
    echo "$(date): $message" >> "$log_file"
}

# Initialize the log file if it doesn't exist
touch "$log_file"
log "Mambaforge installation script started"

# Check if the directory /ibex/user/$USER exists
if [ ! -d "/ibex/user/$USER" ]; then
    echo "Please send an email to ibex@kaust.edu.sa to create a WekaIO Directory for you"
    exit 1
fi

# install Mambaforge in user's WekaIO directory
export PREFIX=/ibex/user/$USER

# Check the output of the "which conda" command
conda_path=$(which conda 2>/dev/null)

# Check if conda is installed
if [ -n "$conda_path" ]; then

    # Get the main conda directory
    grandparent_dir=$(dirname "$(dirname "$conda_path")")
    log "Backing up old conda envs..."
    cp -r $grandparent_dir/envs $PREFIX/conda_envs_backup
    echo "Conda environments were backed up to $PREFIX/conda_envs_backup"
    log "Backup done"
    log "Deleting old conda version installed at: $grandparent_dir"
    rm -rf "$grandparent_dir"
else
    log "No previous conda version installed, resuming..."
fi

#Download and install latest Mambaforge
wget --quiet https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh
log "Installing Mambaforge..."
bash Mambaforge-Linux-x86_64.sh -b -p $PREFIX/mambaforge
source $PREFIX/mambaforge/bin/activate
conda deactivate
conda init bash
rm Mambaforge-Linux-x86_64.sh
log "Mambaforge installation completed."

# Create the conda_cache directory if it doesn't exist
if [ ! -d "$PREFIX/conda_cache" ]; then
    log "Creating conda_cache directory at $PREFIX/conda_cache"
    mkdir -p "$PREFIX/conda_cache"
fi

# Update the CONDA_PKGS_DIRS variable in ~/.bashrc
RELOCATE_CACHE="export CONDA_PKGS_DIRS=${PREFIX}/conda_cache"
if ! grep -qF "$RELOCATE_CACHE" ~/.bashrc; then
  # If the line is not found, append it to ~/.bashrc
  echo "$RELOCATE_CACHE" >> ~/.bashrc
fi

# source ~/.bashrc to load modifications
source ~/.bashrc

# make sure that base environment is not active by default
conda config --set auto_activate_base false

# move the envs installation path to weka
conda config --add envs_dirs $PREFIX/conda-environments

# second source required for update to conda config file to take effect
source ~/.bashrc

log "Mambaforge installation script completed."