CarlaSim runs as a client server application. Since CarlaSim supports limited Operating systems, and requires root access to install, it is unviable to install it on bare metal on a HPC system like Ibex which is shared. Besides, only the server side requires an installation such as that.

Moreover, vulkan drivers are not available on Ibex GPU node and we will rely on opengl to run our server as headless i.e. with no rendering support.

Building Server environment

For server side we use Docker image published by CarlaSim on DockerHub. Since Docker container platform is not available on Ibex. Instead we will use Singularity container platform to run a CarlaSim container using the same Docker image from DockerHub.

Here is how you fetch the relevant image on Ibex:

module load singularity
mkdir -p $HOME/tmpdir
export SINGULARITY_TMPDIR=$HOME/tmpdir
singularity pull docker://carlasim/carla:0.9.5

Once completed, the above commands result in a Singularity Image File or a carla_0.9.5.sif which Singularity container runtime understand.

Running the server

The following jobscript should start a Carla server: 

#!/bin/bash

#SBATCH --job-name=carla_server
#SBATCH --gpus=1
#SBATCH --gpus-per-node=1
#SBATCH --constraint=gtx1080ti
#SBATCH --mem=50G
#SBATCH --cpus-per-task=4
#SBATCH --time=01:00:00


module load singularity
export DISPLAY=
# set image path to where your image is located
export IMAGE=/ibex/scratch/shaima0d/tickets/37094/carla_0.9.5.sif

#Setting the port is optional. Set it if another user process is using the default 2000 and 2001 ports.
export HOST=$(/bin/hostname)
export PORT=11011
echo "Starting Carla server on $HOST:$PORT" > server_info.${SLURM_JOBID}.txt

SINGULARITYENV_SDL_VIDEODRIVER=offscreen SINGULARITYENV_CUDA_VISIBLE_DEIVCE=$CUDA_VISIBLE_DEVICES singularity exec --nv carla_0.9.5.sif /home/carla/CarlaUE4.sh -opengl -carla-rpc-port=11011

  • At times, you may notice that the server doesn’t start and crashes with Segmentation Fault. In such situation try allocating the whole node and resubmit the server job. We are investigating the issue why the CUDA_VISIBLE_DEVICES are not honored.

  • In case you see that the requested port is busy, try to change the port, its an arbitrary number. Try port number > 10000 to keep away from the commonly used ones.

Build Client environment

You can create a conda environment:

conda env create -f environment.yml

where the environment.yml looks as follows:

name: carla_py3
channels:
  - conda-forge
  - intel
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - ca-certificates=2021.5.30=ha878542_0
  - certifi=2021.5.30=py36h5fab9bb_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - libblas=3.9.0=11_linux64_openblas
  - libcblas=3.9.0=11_linux64_openblas
  - libffi=3.3=h58526e2_2
  - libgcc-ng=11.1.0=hc902ee8_8
  - libgfortran-ng=11.1.0=h69a702a_8
  - libgfortran5=11.1.0=h6c583b3_8
  - libgomp=11.1.0=hc902ee8_8
  - libjpeg-turbo=2.1.0=h7f98852_0
  - liblapack=3.9.0=11_linux64_openblas
  - libopenblas=0.3.17=pthreads_h8fe5266_1
  - libstdcxx-ng=11.1.0=h56837e0_8
  - ncurses=6.2=h58526e2_4
  - numpy=1.19.5=py36hfc0c790_2
  - openssl=1.1.1k=h7f98852_0
  - pip=21.2.4=pyhd8ed1ab_0
  - python=3.6.13=hffdb5ce_0_cpython
  - python_abi=3.6=2_cp36m
  - readline=8.1=h46c0cb4_0
  - setuptools=49.6.0=py36h5fab9bb_3
  - sqlite=3.36.0=h9cd32fc_0
  - tk=8.6.10=h21135ba_1
  - wheel=0.37.0=pyhd8ed1ab_0
  - xz=5.2.5=h516909a_1
  - zlib=1.2.11=h516909a_1010
  - pip:
    - carla==0.9.5
    - gputil==1.4.0
    - psutil==5.8.0
    - py-cpuinfo==8.0.0
    - pygame==2.0.1
    - python-tr==0.1.2

Running the client

To connect to the server, you can either start an interactive job or submit a python script in a SLURM job script. The following is a jobscript running a performance benchmark packaged in CarlaSim’s Github repository:

#!/bin/bash
#SBATCH --gpus=1
#SBATCH --time=01:00:00
#SBATCH --mem=100G


source ~/miniconda3/bin/activate carla_py3

HOST=$(cat server_info.txt | cut -d ":" -f 1)
PORT=$(cat server_info.txt | cut -d ":" -f 2)

echo "python  ${PWD}/carla/PythonAPI/util/performance_benchmark.py --host=$HOST --port=${PORT} "
srun -u python  ${PWD}/carla/PythonAPI/util/performance_benchmark.py --host=$HOST --port=${PORT}