SEML on ibex
Create a virtual environment.
This step assume you have installed miniconda following the instructions here:
https://github.com/kaust-rccl/ibex-miniconda-install
The first to clone the git repo is to create a corresponding conda environment for SEML; let's create this environment in the scratch partition for the user and add some useful libraries.
cd /ibex/scratch/$USER
git clone https://github.com/TUM-DAML/seml.git
cd seml
export ENV_PREFIX=$PWD/env
Option 1: pure conda environment
This option will create an environment that runs SEML 0.3.5 with python 3.9. SEML package is not updated on the conda channel, so it needs very specifically python 3.9.
mamba env create -f environment.yaml -p $ENV_PREFIX --force
conda activate $ENV_PREFIX
Here is the environment.yaml file :
name: null
channels:
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1
- _openmp_mutex=4.5
- aiohttp=3.8.3
- aiosignal=1.3.1
- alsa-lib=1.2.8
- anyio=3.6.2
- argon2-cffi=21.3.0
- argon2-cffi-bindings=21.2.0
- asttokens=2.1.0
- async-timeout=4.0.2
- attr=2.5.1
- attrs=22.1.0
- babel=2.11.0
- backcall=0.2.0
- backports=1.0
- backports.functools_lru_cache=1.6.4
- beautifulsoup4=4.11.1
- bleach=5.0.1
- bokeh=3.0.1
- boost-cpp=1.74.0
- brotli=1.0.9
- brotli-bin=1.0.9
- brotlipy=0.7.0
- bzip2=1.0.8
- c-ares=1.18.1
- ca-certificates=2022.9.24
- certifi=2022.9.24
- cffi=1.15.1
- charset-normalizer=2.1.1
- colorama=0.4.6
- contourpy=1.0.6
- cryptography=38.0.3
- cycler=0.11.0
- dbus=1.13.6
- debugpy=1.6.3
- decorator=5.1.1
- defusedxml=0.7.1
- docopt=0.6.2
- entrypoints=0.4
- executing=1.2.0
- expat=2.5.0
- fftw=3.3.10
- flit-core=3.8.0
- font-ttf-dejavu-sans-mono=2.37
- font-ttf-inconsolata=3.000
- font-ttf-source-code-pro=2.038
- font-ttf-ubuntu=0.83
- fontconfig=2.14.1
- fonts-conda-ecosystem=1
- fonts-conda-forge=1
- fonttools=4.38.0
- freetype=2.12.1
- frozenlist=1.3.3
- gettext=0.21.1
- gitdb=4.0.9
- gitpython=3.1.29
- glib=2.74.1
- glib-tools=2.74.1
- gst-plugins-base=1.21.2
- gstreamer=1.21.2
- gstreamer-orc=0.4.33
- icu=70.1
- idna=3.4
- importlib-metadata=5.0.0
- importlib_metadata=5.0.0
- importlib_resources=5.10.0
- ipykernel=6.17.1
- ipython=8.6.0
- ipython_genutils=0.2.0
- jack=1.9.21
- jedi=0.18.1
- jinja2=3.1.2
- jpeg=9e
- json5=0.9.5
- jsonpickle=1.5.1
- jsonschema=4.17.0
- jupyter-server-proxy=3.2.2
- jupyter_client=7.4.5
- jupyter_core=5.0.0
- jupyter_server=1.23.1
- jupyterlab=3.5.0
- jupyterlab-nvdashboard=0.7.0
- jupyterlab_pygments=0.2.2
- jupyterlab_server=2.16.3
- keyutils=1.6.1
- kiwisolver=1.4.4
- krb5=1.19.3
- lame=3.100
- lcms2=2.14
- ld_impl_linux-64=2.39
- lerc=4.0.0
- libabseil=20220623.0
- libblas=3.9.0
- libbrotlicommon=1.0.9
- libbrotlidec=1.0.9
- libbrotlienc=1.0.9
- libcap=2.66
- libcblas=3.9.0
- libclang=15.0.6
- libclang13=15.0.6
- libcups=2.3.3
- libcurl=7.86.0
- libdb=6.2.32
- libdeflate=1.14
- libedit=3.1.20191231
- libev=4.33
- libevent=2.1.10
- libffi=3.4.2
- libflac=1.4.2
- libgcc-ng=12.2.0
- libgcrypt=1.10.1
- libgfortran-ng=12.2.0
- libgfortran5=12.2.0
- libglib=2.74.1
- libgomp=12.2.0
- libgpg-error=1.45
- libiconv=1.17
- liblapack=3.9.0
- libllvm15=15.0.6
- libnghttp2=1.47.0
- libnsl=2.0.0
- libogg=1.3.4
- libopenblas=0.3.21
- libopus=1.3.1
- libpng=1.6.39
- libpq=15.1
- libsndfile=1.1.0
- libsodium=1.0.18
- libsqlite=3.40.0
- libssh2=1.10.0
- libstdcxx-ng=12.2.0
- libsystemd0=252
- libtiff=4.4.0
- libtool=2.4.6
- libudev1=252
- libuuid=2.32.1
- libvorbis=1.3.7
- libwebp-base=1.2.4
- libxcb=1.13
- libxkbcommon=1.0.3
- libxml2=2.10.3
- libzlib=1.2.13
- lz4-c=1.9.3
- markupsafe=2.1.1
- matplotlib=3.6.2
- matplotlib-base=3.6.2
- matplotlib-inline=0.1.6
- mistune=2.0.4
- mongodb=6.0.2
- mpg123=1.31.1
- multidict=6.0.2
- munch=2.5.0
- munkres=1.1.4
- mysql-common=8.0.31
- mysql-libs=8.0.31
- nbclassic=0.4.8
- nbclient=0.7.0
- nbconvert=7.2.4
- nbconvert-core=7.2.4
- nbconvert-pandoc=7.2.4
- nbformat=5.7.0
- ncurses=6.3
- nest-asyncio=1.5.6
- notebook=6.5.2
- notebook-shim=0.2.2
- nspr=4.35
- nss=3.82
- numpy=1.23.4
- openjpeg=2.5.0
- openssl=3.0.7
- packaging=21.3
- pandas=1.5.1
- pandoc=2.19.2
- pandocfilters=1.5.0
- parso=0.8.3
- pcre=8.45
- pcre2=10.40
- pexpect=4.8.0
- pickleshare=0.7.5
- pillow=9.2.0
- pip=22.3.1
- pkgutil-resolve-name=1.3.10
- platformdirs=2.5.2
- ply=3.11
- prometheus_client=0.15.0
- prompt-toolkit=3.0.32
- psutil=5.9.4
- pthread-stubs=0.4
- ptyprocess=0.7.0
- pulseaudio=16.1
- pure_eval=0.2.2
- py-cpuinfo=9.0.0
- pycparser=2.21
- pygments=2.13.0
- pymongo=3.13.0
- pynvml=11.4.1
- pyopenssl=22.1.0
- pyparsing=3.0.9
- pyqt=5.15.7
- pyqt5-sip=12.11.0
- pyrsistent=0.19.2
- pysocks=1.7.1
- python=3.9.13
- python-dateutil=2.8.2
- python-fastjsonschema=2.16.2
- python_abi=3.9
- pytz=2022.6
- pyyaml=6.0
- pyzmq=24.0.1
- qt-main=5.15.6
- readline=8.1.2
- requests=2.28.1
- sacred=0.8.2
- seml=0.3.5
- send2trash=1.8.0
- setuptools=65.5.1
- simpervisor=0.4
- sip=6.7.5
- six=1.16.0
- smmap=3.0.5
- snappy=1.1.9
- sniffio=1.3.0
- soupsieve=2.3.2.post1
- sqlite=3.40.0
- stack_data=0.6.1
- terminado=0.17.0
- tinycss2=1.2.1
- tk=8.6.12
- toml=0.10.2
- tomli=2.0.1
- tornado=6.2
- tqdm=4.64.1
- traitlets=5.5.0
- typing-extensions=4.4.0
- typing_extensions=4.4.0
- tzdata=2022f
- unicodedata2=15.0.0
- urllib3=1.26.11
- wcwidth=0.2.5
- webencodings=0.5.1
- websocket-client=1.4.2
- wheel=0.38.4
- wrapt=1.14.1
- xcb-util=0.4.0
- xcb-util-image=0.4.0
- xcb-util-keysyms=0.4.0
- xcb-util-renderutil=0.3.9
- xcb-util-wm=0.4.1
- xorg-libxau=1.0.9
- xorg-libxdmcp=1.1.3
- xyzservices=2022.9.0
- xz=5.2.6
- yaml=0.2.5
- yaml-cpp=0.7.0
- yarl=1.8.1
- zeromq=4.3.4
- zipp=3.10.0
- zstd=1.5.2
Option 2: Use conda and pip to install the latest SEML version
This option will create a conda environment with the latest version of SEML that comes from pip installation and python 3.11. However, it will also need an extra step to fix a numpy bug inside of a library. Brace yourself; we start building the conda environment.
This environment. yaml looks like this :
Fix the bug for numpy:
If you try to run SEML using this environment, you will get all your experiments killed and the following error related to numpy:
To fix it, you will have to edit the file numpy that s on that site package and issue your favorite text editor:
Then you have to change line 139, and you will also see a warning about the same bug, note that this is the correct line if obj.dtype == "object":
Setup MongoDB using a container
You can do this step on any folder, even your home directory
Clone the git repository containing the Singularity definition file to create the image:
Submit a job to make the singularity image, and the following is an example of the job script
Successful completion should result in creating a singularity image file mongo.sif
.
Launch the MongoDB
You can now copy the image file anywhere you want. The jobs are managed by SEML using MongoDB. A CPU partition must be assigned to the mongo database you will use. You also need a data folder to bind the instance to someplace.
If the mongo instance is killed due to time limits, you will have to launch another, but you may lose the files stored in them.
Create a user on the database.
Inside the out file generated by the script, you will find the IP address you need to connect to the mongo instance; in this example, use head
to see the IP_ADDRESS
where the database is running.
Use an interactive session to try the connection and access the mongosh
interface
After the resources are allocated, you will see the output like this below:
Here you see the assigned port 27017
, the database name test,
and your shell command prompt from mongo test>
. Next, you have to create a user and a password in the database using the following line
This command will create the user barradd
with the password test1
and give it roles in the admin
and test
databases. You can modify this parameter as you like or create a new database for your project.
After this step, you can stop this interactive session
Running the SEML configuration and setup
The next step is to create the corresponding setup for SEML. You need to activate the conda environment,
Since the SEML library is active, it provides a CLI to use it. So first you type seml configure
Here you have to provide the corresponding data; the host
is the IP address; you can leave the default port (unless you change it in previous steps), the user, passwords, and the database name you created during the last step.
Changing the setup for ibex
As indicated by the authors, you can explore several options by looking at the file in the GitHub repo examples/example_config.yaml
.
Change the example_config.yaml
file
Here, we work with the examples/tutorial/example_config.yaml
file that does not contains the whole explanation about the configuration settings. To be very explicit with the interface, you can put the complete path we have on the $ENV_PREFIX
variable to add the conda environment that SEML will use.
Change the slurm_template.sh
A necessary step is that We need to edit the slur template that is going to be used by the library to launch the job arrays. This will be deep in the libraries of the environment:
Then we modify line 1 to #!/bin/bash --login
and line 14 and change it to line 15 to source $CONDA_BASE/bin/activate
Run the experiments
Then you should be able to start your experiments as indicated on the tutorial demo with the corresponding bash commands.
It should look like this:
All the output (successful or not) will be on the logs folders seml/examples/logs