CLUSTER

Step 1: Find a client for the SFTP/SCP

Find a client for the SFTP/SCP protocol:

• CyberDuck
• WinSCP
• FileZilla (WARNING: you must pay attention to not install a web browser toolbar like My Search Dial)

Step 2: Log on the server

Log on our server with the following information

• Host : bioinfo.sb-roscoff.fr
• Username and password from your ABiMS account
• Port : 22
• Protocol : SFTP

Step 3: Upload your data on the server

1. You must change the remote directory: double-click on the path and change "/home" -> "/projet" (or another path)

   

2. Drag and Drop your files in the remote directory

RULES

• Work only in the /projet or /scratch spaces
• Respect this spaces structure (see Usage section),

SSH connection

You must connect on our server by ssh, through a terminal :

ssh -Y <login>@bioinfo.sb-roscoff.fr

/projet

cdprojet

Each /projet space must respect the following structure :

projet_utilisateur/
├── projet1
│   ├── input
│   ├── finalresult
│   ├── script
│   ├── archive
│  
└── projet2
    ├── input
    ├── finalresult
    ├── script
    ├── archive

The backup system works with inclusion. For the /projet spaces that are on the nz server, only the following directories are backuped!

• finalresult/
• script/
• archive(s)/

For the recent nz11 server, no backup has been yet implemented!

/scratch

cdscratch

Since March 2020, ABiMS has set up a new Slurm cluster in parallel with the current SGE cluster. Slurm is, like SGE, a job planner. It is more modern, but the major difference is that it requires you to manage RAM. This is currently our biggest problem with the "old" one. The memory required by jobs under SGE regularly exceeds the capacity of the node and causes major errors. Slurm forces users to declare the requested RAM and kills the job if the threshold is exceeded. This allows us to save on infrastructure maintenance. The first servers/nodes in this new cluster were initially 4 nodes of 256 Core-Threads for 256 GB of RAM. The nodes of the old cluster are added to the new computing infrastructure. Currently, there are more CPUs available on Slurm than on SGE. There is no official doc for the moment but as the new cluster is based on the one implemented for the IFB, we can easily direct you to the IFB docs :)

Request an account

To be allow to submit jobs on Slurm, a Slurm Account is required.
If you want to test this new cluster Slurm, let us know via the support.abims@sb-roscoff.fr

Login node

The login node is now slum0 but you still need to login to ssh/n221 first, than to slurm0.

$ ssh login@ssh.sb-roscoff.fr 
$ ssh slurm0

However, you have to go through ssh.sb-roscoff.fr beforehand. slurm0 is not accessible from outside for the moment.

The documentation

There's some docs we've built as part of the IFB Core Cluster.
You will find these links in today's message when you log in to slurm0.
There's classic doc, training slides and a small 5 minute video that's really didactic.

SGE2SLURM

• qsub is replaced by sbatch. There are subtleties in the syntax of the instructions to declare CPU numbers, ...
• qlogin disappears. If you want to run a fast calculation without sbatch, you just prefix your command with srun from the slurm0 server.

srun hostname # you can observe that you reach one of the cluster node
srun ln --help
srun --mem=8GB foobar -p yo

• Now you need to reserve the RAM for your jobs. By default, it is 2Gb. If the job exceeds the requested RAM, it is killed. It is restrictive but it protects the servers from memory overflow and therefore, we gain stability. It becomes an iterative process to properly estimate neither too high nor too low RAM consumption.

The Partitions and default values

The tools

We are now installing the tool through this central GitLab repository: https://ifb-elixirfr.gitlab.io/cluster/doc/conda/
The tools deploied at the same time on the: IFB Core Cluster, ABiMS, the IGBMC and on the Strasbourg University Datacenter.
What's new is that all Conda env and Singularity images can be loaded via module:

$ module load trinity/2.8.5
$ module load trinity # loads the latest version 

To know all the available

$ module avail
$ module avail trinity/ 

Self-completion works

$ module load tri [TAB]

/!\ The tools that used to be in the PATH are no longer in the PATH. You can load a specific module to grab them:

$ module load _genome2-legacy

But be aware that a lot of tool won't work because they need some library on the system, we didn't and don't want to install. Conda <3 <3 <3

Training slides

Please have a look at our last training material: [pdf]

RULES

• Never run calculation on the master node (n221)
• At the end of a qlogin, it's imperative to log out,
• Job-array have to be ran only on short.q, 
  • If jobs are too long, input data have to be cut (option -n for atomic_blast).

QSUB

1. Write a qsub script

   Script example (sge_blast.sh). The first 4 lines from the header are mandatory.

   #!/bin/bash
   #$ -S /bin/bash
   #$ -V
   #$ -cwd
   ## Queue settings (can be set at runtime in the qsub submition line)
   #$ -q long.q
   ## [Optional] to ask n processors
   #$ -pe thread 4
   ## [Optional] to get job status by email
   #$ -M <votre-adresse>@sb-roscoff.fr
   #$ -m bea
   ## Ask for at least 60GB of memory for the job
   #$ -l mem_free=60G
   ## Kill the job if it eats 20GB per slot (i.e. thread) - here it means 20 GB*4 threads=80GB
   #$ -l h_vmem=20G 
   

    

   # Exemple pour l'utilisation de blast :
   blastp -query test_seq.fasta -out test_seq.blast -db /db/blast/all/uniprot_swissprot -num_threads $NSLOTS

   Parameters	Usage	Mandatory
   -S	Command interpreter to be used	Yes
   -V	Export these environment variables	Yes
   -cwd	Use current working directory	Yes
   -q short.q	Queue + [Group/Node] needed See our ressources to choose a queue among short.q, long.q and infinite.q For a specific group: -q long.q@@bigcpu For a specific node: -q long.q@n80	Yes
   -pe thread X	For multithread software to require n CPUs. The number MUST be the same as the one set in the software parameters (-num_threads) The variable $NSLOTS can be used in the program arguments to set the number of CPU according the the SGE setting -pe thread	No
   -M	Notify these e-mail addresses	No
   -m bea	Send mail notification events at the (b)egin, at the (e)nd, when (a)bort, (s)uspend a job and (n)one	No
   -l mem_free=XG	Only submit the job if there is enough memory	No
   -l h_vmem=XG	Kill the job if it eats more than XG of memory per slot/thread	No

2. Submit your script using QSUB

   If you already set your -q options within the script (see above)

   qsub sge_blast.sh

   Or to set -q options at runtime

   qsub -q long.q -pe thread 10 sge_blast.sh

QLOGIN

• Connection to a queue :

qlogin -q qlogin.q

• Connection to a node :

qlogin -q qlogin.q@n72

• Connection to a group :

qlogin -q qlogin.q@@bigcpu

• To quit a qlogin connection :

exit

You can also close your window to quit the connection.

QDEL

• Delete a job :

qdel <job id>

• Delete all my jobs :

qdel -u <login>

Information about jobs

• Runing and waiting jobs list :

qstat

• To see only my job list :

\qstat

• Nodes status :

qhost

Parallel computing allow to use several processor (CPU) for one job. The gain is not linear and often saturates around 8 or 16 threads but you can save a lot of time compare to one CPU.

Parallel computing can be available depending on the software and/or the input.

• Some software allow to use several CPU because they were developed for that purpose and some not: See Thread and MPI methods.
• If the input can be split and be processed separally, thus you can cut it in 100 files and launch 100 jobs.

ABiMS provide two environments depending on the software you will use:

• Thread: spread jobs within one node  - Common
• MPI: spead jobs across all nodes - Rare

Environnement Thread

It's the more efficient method because all the exchanges of information take place within one hardware host. But the maximum number of CPU < the maximum number of CPU in the node.

The number of slots available across the cluster is 800 slots

qsub

Argument to use in the qsub script:

#$ -pe thread <number_of_cpu> 

To book the slots as soon as they are relase -R y

software

To get the parameter within the software, display the help.

The variable $NSLOTS can be used in the program arguments to set the number of CPU according the the SGE setting -pe thread

Examples

• Blast : -a <number_of_cpu>     note: it's more efficient to split the input query file
• TCoffee : -n_core <number_of_cpu>    note: the default behaviour of TCoffe is to use all the CPU available. So please, set the number of CPU
• Mira : -GE:not=<nombre de coeurs>

Environnement MPI (implémentation avec la bibliothèque openMPI)

This method allow to submit parts of the job across the whole cluster. Thereby, there is more resource available. But this method is not as efficient as the Thread method because the information are exchanged through the network. Those software are also very rare.

qsub

Argument to use in the qsub script:

#$  -pe openmpi <number_of_cpu> 

To book the slots as soon as they are relase -R y

The software must be launched though mpirun:

mpirun -np <nb core> <full_path_of_the_program> <arg program>

Example of qsub script:

#!/bin/bash
#$ -S /bin/bash 
#$ -cwd # Export des variables
#$ -V
#$ -q long.q
#$ -pe openmpi 8
mpirun -np 8 /usr/local/genome2/bin/phyml-mpi -i input.phy --datatype "nt" --model 012345 --bootstrap 10 --search BEST --alpha e -f m --pinv e -o tlr --nclasses 4

The qsub command:

qsub  ./phyml-mpi.sh

Software

Example of program which allow mpi:

• clustalw : sequence alignment
• raxmlHPC-MPI : phylogeny
• phyml-mpi : phylogeny