Tools

Conda

Why

Most of the tools need some requirements. Some need lot of requirements: Python or R libraries in specific version, the last brand new compilator, or simply a newer one (GCC > 4.4.7), ... Often, thoses dependencies are not compatibles with what we already have on our system or between them.

What

Conda is an open source package, dependency and environment manager for any language: Python, R, Ruby, Lua, Scala, Java, Javascript, C/ C++, FORTRAN. Miniconda is a small “bootstrap” version that includes only conda, Python, and the packages they depend on. Over 720 scientific packages and their dependencies can be installed individually from the Continuum repository with the “conda install” command.

At ABiMS, it will allow us to install tools within some dedicated and isolated environments. Specially those which require some newer version of GCC 4.4.7.

Note that all software are not provided by Conda. Meanwhile, ABiMS is contributing to add package in Conda through the GitHub repository Bioconda.

How

We installed Miniconda based Python 3.5.

Conda is not directly in the system $PATH to avoid troubles with existing system libraries and binairies.
We will always try to provide some isolated environments around the software

Using a Conda environment is quite simple. Firstly you have to activate the environment, secondly use the wanted program(s), and finally restore the default environment.

Example

In this example, we will use programs from stacks-1.47.

In a script, a terminal or a qlogin, we just have to type the following and program(s) from the Conda env. will appear in the our $PATH.
```
source $CONDA3/activate stacks-1.47
```
Now we can use programs from the Conda environment. Here we display the version of cstacks:
```
cstacks --version # Must return cstacks 1.47
```
When we have finished, we go back in the default environment with the following command:
```
source $CONDA3/deactivate stacks-1.47
```

To stack 2 or more envronment:

[lecorguille@n221 ~]$ source $CONDA3/activate beast2-2.5.0
(beast2-2.5.0) [lecorguille@n221 ~]$ source $CONDA3/activate --stack beagle-5.1_24Aug19.3e8
(beagle-5.1_24Aug19.3e8) [lecorguille@n221 ~]$ which beagle
/usr/local/genome2/conda3/envs/beagle-5.1_24Aug19.3e8/bin/beagle
(beagle-5.1_24Aug19.3e8) [lecorguille@n221 ~]$ which beast
/usr/local/genome2/conda3/envs/beast2-2.5.0/bin/beast

Cons

Because, those environments isotate the software. The other Python, R or Perl libraries which are installed on the system are not availables within the Conda environment. If it is an issue for you, let us know support.abims@sb-roscoff.fr

Tips and Tricks

Before using those commands, you need to type in your terminal/qlogin one of the following command:

$ conda3activate #for miniconda3

When finished, do not forget to go back to unload the environment.

$ conda3deactivate #for miniconda3

To list all the Conda environments already installed on ABiMS

$ conda env list

To list all the tools within a specific Conda environment installed on ABiMS

$ source $CONDA3/activare eba2017_rnaseq_denovo
$ conda list

To know if a tools is avalable though Conda

$ conda search mu_tool

Install your own instance of Conda

Some software need write access into their install directory (RepeatMasker, Homer, ...).
For them, we have to install miniconda in the user home directory.

Installation

1- https://bioconda.github.io/user/install.html#install-conda
We should avoid to add Conda in the .bashrc at the end of the installation
2- https://bioconda.github.io/user/install.html#set-up-channels
There will be a warning when addng default. It's harmless.
3-

~/miniconda3/bin/conda create -n repeatmasker-4.0.9_p2 repeatmasker=4.0.9_p2

Usage

source ~/miniconda3/bin/activate repeatmasker-4.0.9_p2

RStudio

A new Rstudio instance

Since April 2020, ABiMS is providing a new RStudio instance : https://rstudio.sb-roscoff.fr

/!\ When logging in to the site, you will certainly get a security warning linked to our self-signed SSL certificate. We are aware of this, but currently the CNRS cannot provide us with a valid certificate. But be sure that your passwords are encrypted between us.

Note that this instance is the result of a strong collaboration with the IFB Core Cluster Task Force. It is a clone of the IFB rstudio server.

The server that now hosts this instance is newer and more powerful than the previous one.
The application was installed using Ansible and will be updated more regularly.
The packages are installed simultaneously on the ABiMS and IFB servers via a centralized GitLab repository.
You will be able to access the same R deployment, with the same packages via RStudio and via SSH usage. This will make it easier to switch from one interface to the other.
As the mechanisms and packages are shared between the IFB and ABiMS, we'll now share to a group of experts in case of problems .

R <-> RStudio

You can now switch easily between R-Cli and RStudio since RStudio is using the R-Cli installed for SSH purpose.

From the Slurm cluster

[login@slurm0 ~]$ module avail r/
------------------------- /shared/software/modulefiles -------------------------
r/3.5.1  r/3.6.3

[login@slurm0 ~]$ module load r/3.6.3

From the SGE cluster

[login@n221 ~]$ source $CONDA3/activate r-3.6.3

Tips and Tricks

Working directory

By default, RStudio working directory will be your $HOME, but we suggest you to work in your project workspace instead. It can be easily done as shown in the following picture:

Python Virtual Env

Why

Some Python software or packages need some other requirements... Furthermore, thoses dependencies are not compatibles with what we already have on our system or between them: upgrade of NumPy, downgrade of SciPy, ...

What

Python Virtualenv propose to build some isolated environments around a tool of a package.

How

From qsub / qlogin / a terminal: Activate and deactivate a Virtualenv environment

$ source $TOOLS/dadi_venv/bin/activate
(dadi_venv_1.7.0)$ source $TOOLS/dadi_venv/bin/activate
(dadi_venv_1.7.0)$ deactivate
$

Tips and Tricks

To list all the Conda environments already installed

$ ls -d $TOOLS/*_venv*

Atomic Blast +

Description

Atomic Blast+ is implementation of the NCBI Blast+ algorithm that can (1) split a multifasta file in sub-file, (2) generate a SGE script and (3) submit this script as an array-job to use several nodes in parallel. This will gain speed of execution and optimally use the cluster capacity.

Use

$ atomicblastplus.py -p blastn -i input.fasta -o myproject/tmp/blast/input_vs_nr -d /db/blast/all/nr -e 1e-4 --verbose

Help

$ atomicblastplus.py --help

Darkhorse

Blast versus NR

Because Darkhorse need a good synchronization beetween its indexes and the version of NR which was used for the indexation.

Please use the NR Blast indexes at this location:

/db/off_biomaj/darkhorse/nr/blast

The Darkhorse config file

The configuration is located at this location:

/usr/local/genome2/DarkHorse/darkhorse1.cfg

Qiime / Qiime 2

Qiime databanks are available in

/db/off_biomaj/qiime/

Qiime 1.9.1

source $CONDA2/activate qiime-1.9.1

And then read the documentation here: http://qiime.org/scripts/

Qiime 2

source $CONDA3/activate qiime2-2018.2

And then read the documentation here: https://docs.qiime2.org/2018.2/

Interproscan

Interproscan is installed in /opt/abims/interproscan. It can be run in standalone mode or in cluster mode. The precalculed match lookup service is also installed. Using Interproscan is quite simple. Running interproscan.sh without parameter gives usage instructions (it can take few seconds)

interproscan.sh

tl;dr

interproscan.sh -i test_proteins.fasta -f tsv,gff3 -mode cluster -clusterrunid iprscantest

Where:

-i: input file
-f: output formats (tsv, xml, json, gff3, html, svg). Multiple formats can be used. Look at -o, -b or -d options to change output file names/directory.
-mode cluster: use the cluster through sge.
-clusterrunid: mandatory with -mode cluster; given name of the job. Must be unique.

This command will produce test_proteins.fasta.tsv and test_proteins.fasta.gff3 in the current directory.

For huge sequence files, please consult the following page:

https://github.com/ebi-pf-team/interproscan/wiki/ImprovingPerformance, or
/opt/abims/interproscan/interproscan.wiki/interproscan.wiki/ImprovingPerformance.md

Standalone mode

Just follow the 'How to run' wiki page

https://github.com/ebi-pf-team/interproscan/wiki/HowToRun, or
/opt/abims/interproscan/interproscan.wiki/interproscan.wiki/HowToRun.md

Detail of Parameters

Welcome to InterProScan-5.24-63.0
usage: java -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:+AggressiveOpts
            -XX:+UseFastAccessorMethods -Xms128M -Xmx2048M -jar
            interproscan-5.jar


Please give us your feedback by sending an email to

interhelp@ebi.ac.uk

 -appl,--applications             Optional, comma separated list
                                            of analyses.  If this option
                                            is not set, ALL analyses will
                                            be run.
 -b,--output-file-base    Optional, base output filename
                                            (relative or absolute path).
                                            Note that this option, the
                                            --output-dir (-d) option and
                                            the --outfile (-o) option are
                                            mutually exclusive.  The
                                            appropriate file extension for
                                            the output format(s) will be
                                            appended automatically. By
                                            default the input file
                                            path/name will be used.
 -cpu,--cpu                            Optional, number of cores for
                                            inteproscan.
 -d,--output-dir                Optional, output directory.
                                            Note that this option, the
                                            --outfile (-o) option and the
                                            --output-file-base (-b) option
                                            are mutually exclusive. The
                                            output filename(s) are the
                                            same as the input filename,
                                            with the appropriate file
                                            extension(s) for the output
                                            format(s) appended
                                            automatically .
 -dp,--disable-precalc                      Optional.  Disables use of the
                                            precalculated match lookup
                                            service.  All match
                                            calculations will be run
                                            locally.
 -dra,--disable-residue-annot               Optional, excludes sites from
                                            the XML, JSON output
 -f,--formats               Optional, case-insensitive,
                                            comma separated list of output
                                            formats. Supported formats are
                                            TSV, XML, JSON, GFF3, HTML and
                                            SVG. Default for protein
                                            sequences are TSV, XML and
                                            GFF3, or for nucleotide
                                            sequences GFF3 and XML.
 -goterms,--goterms                         Optional, switch on lookup of
                                            corresponding Gene Ontology
                                            annotation (IMPLIES -iprlookup
                                            option)
 -help,--help                               Optional, display help
                                            information
 -i,--input                Optional, path to fasta file
                                            that should be loaded on
                                            Master startup. Alternatively,
                                            in CONVERT mode, the
                                            InterProScan 5 XML file to
                                            convert.
 -iprlookup,--iprlookup                     Also include lookup of
                                            corresponding InterPro
                                            annotation in the TSV and GFF3
                                            output formats.
 -ms,--minsize                Optional, minimum nucleotide
                                            size of ORF to report. Will
                                            only be considered if n is
                                            specified as a sequence type.
                                            Please be aware of the fact
                                            that if you specify a too
                                            short value it might be that
                                            the analysis takes a very long
                                            time!
 -o,--outfile     Optional explicit output file
                                            name (relative or absolute
                                            path).  Note that this option,
                                            the --output-dir (-d) option
                                            and the --output-file-base
                                            (-b) option are mutually
                                            exclusive. If this option is
                                            given, you MUST specify a
                                            single output format using the
                                            -f option.  The output file
                                            name will not be modified.
                                            Note that specifying an output
                                            file name using this option
                                            OVERWRITES ANY EXISTING FILE.
 -pa,--pathways                             Optional, switch on lookup of
                                            corresponding Pathway
                                            annotation (IMPLIES -iprlookup
                                            option)
 -t,--seqtype                Optional, the type of the
                                            input sequences (dna/rna (n)
                                            or protein (p)).  The default
                                            sequence type is protein.
 -T,--tempdir                     Optional, specify temporary
                                            file directory (relative or
                                            absolute path). The default
                                            location is temp/.
 -version,--version                         Optional, display version
                                            number
Copyright © EMBL European Bioinformatics Institute, Hinxton, Cambridge,
UK. (http://www.ebi.ac.uk) The InterProScan software itself is provided
under the Apache License, Version 2.0
(http://www.apache.org/licenses/LICENSE-2.0.html). Third party components
(e.g. member database binaries and models) are subject to separate
licensing - please see the individual member database websites for
details.

Available analyses:
                      TIGRFAM (15.0) : TIGRFAMs are protein families based on Hidden Markov Models or HMMs
                         SFLD (2) : SFLDs are protein families based on Hidden Markov Models or HMMs
                      Phobius (1.01) : A combined transmembrane topology and signal peptide predictor
        SignalP_GRAM_NEGATIVE (4.1) : SignalP (organism type gram-negative prokaryotes) predicts the presence and location of signal peptide cleavage sites in amino acid sequences for gram-negative prokaryotes.
                  SUPERFAMILY (1.75) : SUPERFAMILY is a database of structural and functional annotation for all proteins and genomes.
                      PANTHER (11.1) : The PANTHER (Protein ANalysis THrough Evolutionary Relationships) Classification System is a unique resource that classifies genes by their functions, using published scientific experimental evidence and evolutionary relationships to predict function even in the absence of direct experimental evidence.
                       Gene3D (4.1.0) : Structural assignment for whole genes and genomes using the CATH domain structure database
                        Hamap (201701.18) : High-quality Automated and Manual Annotation of Microbial Proteomes
              ProSiteProfiles (20.132) : PROSITE consists of documentation entries describing protein domains, families and functional sites as well as associated patterns and profiles to identify them
                        Coils (2.2.1) : Prediction of Coiled Coil Regions in Proteins
                        SMART (7.1) : SMART allows the identification and analysis of domain architectures based on Hidden Markov Models or HMMs
                          CDD (3.14) : Prediction of CDD domains in Proteins
                       PRINTS (42.0) : A fingerprint is a group of conserved motifs used to characterise a protein family
              ProSitePatterns (20.132) : PROSITE consists of documentation entries describing protein domains, families and functional sites as well as associated patterns and profiles to identify them
                  SignalP_EUK (4.1) : SignalP (organism type eukaryotes) predicts the presence and location of signal peptide cleavage sites in amino acid sequences for eukaryotes.
                         Pfam (31.0) : A large collection of protein families, each represented by multiple sequence alignments and hidden Markov models (HMMs)
                       ProDom (2006.1) : ProDom is a comprehensive set of protein domain families automatically generated from the UniProt Knowledge Database.
                   MobiDBLite (1.0) : Prediction of disordered domains Regions in Proteins
        SignalP_GRAM_POSITIVE (4.1) : SignalP (organism type gram-positive prokaryotes) predicts the presence and location of signal peptide cleavage sites in amino acid sequences for gram-positive prokaryotes.
                        PIRSF (3.02) : The PIRSF concept is being used as a guiding principle to provide comprehensive and non-overlapping clustering of UniProtKB sequences into a hierarchical order to reflect their evolutionary relationships.
                        TMHMM (2.0c) : Prediction of transmembrane helices in proteins

Cluster mode

Same as standalone mode, but with the following additional parameters in the command:

interproscan.sh ... -mode cluster -clusterrunid "myUniqJobName"

Jobs will be submitted on long.q.

Example

interproscan.sh -i test_proteins.fasta -f tsv -mode cluster -clusterrunid iprscantest

Huge Sequence Files

From https://github.com/ebi-pf-team/interproscan/wiki/ImprovingPerformance

If your FASTA input files contains a large number of sequences then you may consider splitting your input into smaller chunks (depends on resources, but batches of 10,000 protein sequences is a suggested starting point). You can then submit the smaller input files to InterProScan and process the results afterwards. For DNA/RNA sequences a much smaller number is suggested (e.g. 1,000 sequences). However for improved performance you could translate these using an external tool and then submit the necessary protein sequences instead, see running nucleic acid sequences for more information.

Disable precalculed match lookup service

interproscan.sh ... -dp

Additional ressources

https://github.com/ebi-pf-team/interproscan/wiki/HowToRun https://github.com/ebi-pf-team/interproscan/wiki/ClusterMode https://github.com/ebi-pf-team/interproscan/wiki/ImprovingPerformance

You are here

Tools

Why

What

How

Example

Cons

Tips and Tricks

To list all the Conda environments already installed on ABiMS

To list all the tools within a specific Conda environment installed on ABiMS

To know if a tools is avalable though Conda

Install your own instance of Conda

Installation

Usage

A new Rstudio instance

R <-> RStudio

From the Slurm cluster

From the SGE cluster

Tips and Tricks

Working directory

Why

What

How

From qsub / qlogin / a terminal: Activate and deactivate a Virtualenv environment

Tips and Tricks

To list all the Conda environments already installed

Description

Use

Help

Blast versus NR

The Darkhorse config file

Qiime 1.9.1

Qiime 2

tl;dr

Standalone mode

Cluster mode

Example

Huge Sequence Files

Disable precalculed match lookup service

Additional ressources