SEML on ibex

Create a virtual environment.

This step assume you have installed miniconda following the instructions here:
https://github.com/kaust-rccl/ibex-miniconda-install

The first to clone the git repo is to create a corresponding conda environment for SEML; let's create this environment in the scratch partition for the user and add some useful libraries.

cd /ibex/scratch/$USER git clone https://github.com/TUM-DAML/seml.git cd seml export ENV_PREFIX=$PWD/env

Option 1: pure conda environment

This option will create an environment that runs SEML 0.3.5 with python 3.9. SEML package is not updated on the conda channel, so it needs very specifically python 3.9.

mamba env create -f environment.yaml -p $ENV_PREFIX --force conda activate $ENV_PREFIX

 

Here is the environment.yaml file :

name: null channels: - conda-forge - defaults dependencies: - _libgcc_mutex=0.1 - _openmp_mutex=4.5 - aiohttp=3.8.3 - aiosignal=1.3.1 - alsa-lib=1.2.8 - anyio=3.6.2 - argon2-cffi=21.3.0 - argon2-cffi-bindings=21.2.0 - asttokens=2.1.0 - async-timeout=4.0.2 - attr=2.5.1 - attrs=22.1.0 - babel=2.11.0 - backcall=0.2.0 - backports=1.0 - backports.functools_lru_cache=1.6.4 - beautifulsoup4=4.11.1 - bleach=5.0.1 - bokeh=3.0.1 - boost-cpp=1.74.0 - brotli=1.0.9 - brotli-bin=1.0.9 - brotlipy=0.7.0 - bzip2=1.0.8 - c-ares=1.18.1 - ca-certificates=2022.9.24 - certifi=2022.9.24 - cffi=1.15.1 - charset-normalizer=2.1.1 - colorama=0.4.6 - contourpy=1.0.6 - cryptography=38.0.3 - cycler=0.11.0 - dbus=1.13.6 - debugpy=1.6.3 - decorator=5.1.1 - defusedxml=0.7.1 - docopt=0.6.2 - entrypoints=0.4 - executing=1.2.0 - expat=2.5.0 - fftw=3.3.10 - flit-core=3.8.0 - font-ttf-dejavu-sans-mono=2.37 - font-ttf-inconsolata=3.000 - font-ttf-source-code-pro=2.038 - font-ttf-ubuntu=0.83 - fontconfig=2.14.1 - fonts-conda-ecosystem=1 - fonts-conda-forge=1 - fonttools=4.38.0 - freetype=2.12.1 - frozenlist=1.3.3 - gettext=0.21.1 - gitdb=4.0.9 - gitpython=3.1.29 - glib=2.74.1 - glib-tools=2.74.1 - gst-plugins-base=1.21.2 - gstreamer=1.21.2 - gstreamer-orc=0.4.33 - icu=70.1 - idna=3.4 - importlib-metadata=5.0.0 - importlib_metadata=5.0.0 - importlib_resources=5.10.0 - ipykernel=6.17.1 - ipython=8.6.0 - ipython_genutils=0.2.0 - jack=1.9.21 - jedi=0.18.1 - jinja2=3.1.2 - jpeg=9e - json5=0.9.5 - jsonpickle=1.5.1 - jsonschema=4.17.0 - jupyter-server-proxy=3.2.2 - jupyter_client=7.4.5 - jupyter_core=5.0.0 - jupyter_server=1.23.1 - jupyterlab=3.5.0 - jupyterlab-nvdashboard=0.7.0 - jupyterlab_pygments=0.2.2 - jupyterlab_server=2.16.3 - keyutils=1.6.1 - kiwisolver=1.4.4 - krb5=1.19.3 - lame=3.100 - lcms2=2.14 - ld_impl_linux-64=2.39 - lerc=4.0.0 - libabseil=20220623.0 - libblas=3.9.0 - libbrotlicommon=1.0.9 - libbrotlidec=1.0.9 - libbrotlienc=1.0.9 - libcap=2.66 - libcblas=3.9.0 - libclang=15.0.6 - libclang13=15.0.6 - libcups=2.3.3 - libcurl=7.86.0 - libdb=6.2.32 - libdeflate=1.14 - libedit=3.1.20191231 - libev=4.33 - libevent=2.1.10 - libffi=3.4.2 - libflac=1.4.2 - libgcc-ng=12.2.0 - libgcrypt=1.10.1 - libgfortran-ng=12.2.0 - libgfortran5=12.2.0 - libglib=2.74.1 - libgomp=12.2.0 - libgpg-error=1.45 - libiconv=1.17 - liblapack=3.9.0 - libllvm15=15.0.6 - libnghttp2=1.47.0 - libnsl=2.0.0 - libogg=1.3.4 - libopenblas=0.3.21 - libopus=1.3.1 - libpng=1.6.39 - libpq=15.1 - libsndfile=1.1.0 - libsodium=1.0.18 - libsqlite=3.40.0 - libssh2=1.10.0 - libstdcxx-ng=12.2.0 - libsystemd0=252 - libtiff=4.4.0 - libtool=2.4.6 - libudev1=252 - libuuid=2.32.1 - libvorbis=1.3.7 - libwebp-base=1.2.4 - libxcb=1.13 - libxkbcommon=1.0.3 - libxml2=2.10.3 - libzlib=1.2.13 - lz4-c=1.9.3 - markupsafe=2.1.1 - matplotlib=3.6.2 - matplotlib-base=3.6.2 - matplotlib-inline=0.1.6 - mistune=2.0.4 - mongodb=6.0.2 - mpg123=1.31.1 - multidict=6.0.2 - munch=2.5.0 - munkres=1.1.4 - mysql-common=8.0.31 - mysql-libs=8.0.31 - nbclassic=0.4.8 - nbclient=0.7.0 - nbconvert=7.2.4 - nbconvert-core=7.2.4 - nbconvert-pandoc=7.2.4 - nbformat=5.7.0 - ncurses=6.3 - nest-asyncio=1.5.6 - notebook=6.5.2 - notebook-shim=0.2.2 - nspr=4.35 - nss=3.82 - numpy=1.23.4 - openjpeg=2.5.0 - openssl=3.0.7 - packaging=21.3 - pandas=1.5.1 - pandoc=2.19.2 - pandocfilters=1.5.0 - parso=0.8.3 - pcre=8.45 - pcre2=10.40 - pexpect=4.8.0 - pickleshare=0.7.5 - pillow=9.2.0 - pip=22.3.1 - pkgutil-resolve-name=1.3.10 - platformdirs=2.5.2 - ply=3.11 - prometheus_client=0.15.0 - prompt-toolkit=3.0.32 - psutil=5.9.4 - pthread-stubs=0.4 - ptyprocess=0.7.0 - pulseaudio=16.1 - pure_eval=0.2.2 - py-cpuinfo=9.0.0 - pycparser=2.21 - pygments=2.13.0 - pymongo=3.13.0 - pynvml=11.4.1 - pyopenssl=22.1.0 - pyparsing=3.0.9 - pyqt=5.15.7 - pyqt5-sip=12.11.0 - pyrsistent=0.19.2 - pysocks=1.7.1 - python=3.9.13 - python-dateutil=2.8.2 - python-fastjsonschema=2.16.2 - python_abi=3.9 - pytz=2022.6 - pyyaml=6.0 - pyzmq=24.0.1 - qt-main=5.15.6 - readline=8.1.2 - requests=2.28.1 - sacred=0.8.2 - seml=0.3.5 - send2trash=1.8.0 - setuptools=65.5.1 - simpervisor=0.4 - sip=6.7.5 - six=1.16.0 - smmap=3.0.5 - snappy=1.1.9 - sniffio=1.3.0 - soupsieve=2.3.2.post1 - sqlite=3.40.0 - stack_data=0.6.1 - terminado=0.17.0 - tinycss2=1.2.1 - tk=8.6.12 - toml=0.10.2 - tomli=2.0.1 - tornado=6.2 - tqdm=4.64.1 - traitlets=5.5.0 - typing-extensions=4.4.0 - typing_extensions=4.4.0 - tzdata=2022f - unicodedata2=15.0.0 - urllib3=1.26.11 - wcwidth=0.2.5 - webencodings=0.5.1 - websocket-client=1.4.2 - wheel=0.38.4 - wrapt=1.14.1 - xcb-util=0.4.0 - xcb-util-image=0.4.0 - xcb-util-keysyms=0.4.0 - xcb-util-renderutil=0.3.9 - xcb-util-wm=0.4.1 - xorg-libxau=1.0.9 - xorg-libxdmcp=1.1.3 - xyzservices=2022.9.0 - xz=5.2.6 - yaml=0.2.5 - yaml-cpp=0.7.0 - yarl=1.8.1 - zeromq=4.3.4 - zipp=3.10.0 - zstd=1.5.2

 

Option 2: Use conda and pip to install the latest SEML version

This option will create a conda environment with the latest version of SEML that comes from pip installation and python 3.11. However, it will also need an extra step to fix a numpy bug inside of a library. Brace yourself; we start building the conda environment.

 

This environment. yaml looks like this :

 

Fix the bug for numpy:

If you try to run SEML using this environment, you will get all your experiments killed and the following error related to numpy:

 

To fix it, you will have to edit the file numpy that s on that site package and issue your favorite text editor:

 

Then you have to change line 139, and you will also see a warning about the same bug, note that this is the correct line if obj.dtype == "object":

 

Setup MongoDB using a container

You can do this step on any folder, even your home directory

Clone the git repository containing the Singularity definition file to create the image:

Submit a job to make the singularity image, and the following is an example of the job script

Successful completion should result in creating a singularity image file mongo.sif.

Launch the MongoDB

You can now copy the image file anywhere you want. The jobs are managed by SEML using MongoDB. A CPU partition must be assigned to the mongo database you will use. You also need a data folder to bind the instance to someplace.

If the mongo instance is killed due to time limits, you will have to launch another, but you may lose the files stored in them.

Create a user on the database.

Inside the out file generated by the script, you will find the IP address you need to connect to the mongo instance; in this example, use head to see the IP_ADDRESS where the database is running.

 

Use an interactive session to try the connection and access the mongosh interface

 

After the resources are allocated, you will see the output like this below:

 

Here you see the assigned port 27017, the database name test, and your shell command prompt from mongo test>. Next, you have to create a user and a password in the database using the following line

 

This command will create the user barradd with the password test1 and give it roles in the admin and test databases. You can modify this parameter as you like or create a new database for your project.

After this step, you can stop this interactive session

Running the SEML configuration and setup

The next step is to create the corresponding setup for SEML. You need to activate the conda environment,

 

Since the SEML library is active, it provides a CLI to use it. So first you type seml configure

Here you have to provide the corresponding data; the host is the IP address; you can leave the default port (unless you change it in previous steps), the user, passwords, and the database name you created during the last step.

Changing the setup for ibex

As indicated by the authors, you can explore several options by looking at the file in the GitHub repo examples/example_config.yaml.

 

Change the example_config.yaml file

Here, we work with the examples/tutorial/example_config.yaml file that does not contains the whole explanation about the configuration settings. To be very explicit with the interface, you can put the complete path we have on the $ENV_PREFIX variable to add the conda environment that SEML will use.

 

Change the slurm_template.sh

A necessary step is that We need to edit the slur template that is going to be used by the library to launch the job arrays. This will be deep in the libraries of the environment:

Then we modify line 1 to #!/bin/bash --login and line 14 and change it to line 15 to source $CONDA_BASE/bin/activate

 

Run the experiments

Then you should be able to start your experiments as indicated on the tutorial demo with the corresponding bash commands.

 

It should look like this:

All the output (successful or not) will be on the logs folders seml/examples/logs