View source for Galaxy Tutorial 2012 Extras

<div class="emphasisbox">Items shown on this page were part of earlier drafts of the [[Galaxy Tutorial 2012|2012 Summer School Galaxy workshop]].  They were moved here either in the interests of time, or because they no longer fit well with other content.  However, these sections are still informative.</div>

== Defining Genomes to Trackster ==

From the {{GalaxyWikiLink|Learn/Visualization#Setup_for_Local_Instances|Learn/Visualization wiki page}}

<div class="indent">
Trackster needs chrom/contig length information for builds to be able to be visualized, either in the form of <tt>.len</tt> files or custom user builds. The former option allows all users to be able to use the build. Trackster looks for the <tt>.len</tt> files in the <tt>tool-data/shared/ucsc/chrom/</tt> directory by default, but this location can be configured in the <tt>universe_wsgi.ini</tt> file:

 # Directory where chrom len files are kept, currently mainly used by trackster
 #len_file_path = tool-data/shared/ucsc/chrom

For example, to enable visualization of hg18, a file called hg18.len must exist in the previously specified folder. The contents of the <tt>.len</tt> file is tab-delimited and specifies the length of each chrom/contig of that build, each on a separate line, eg:

 chr1    27473282
 chr2    38882233
 ...
 chrX    28883322

To populate this directory with common UCSC builds, run the following commands:

<syntaxhighlight lang="bash">mkdir ./tool-data/shared/ucsc/chrom/
python ./cron/build_chrom_db.py ./tool-data/shared/ucsc/chrom/</syntaxhighlight>

To display a build's genome data (i.e. bases/nucleotides) when sufficiently zoomed in: (a) create a 2bit file from the genome's fasta file using <tt>faToTwoBit</tt>, which is available from UCSC's tools (source, executables and (b) add an entry to the <tt>tool-data/twobit.loc</tt> file to indicate that where the build's twobit file is located.

Finally, you will need to make sure your <tt>datatypes_conf.xml</tt> file contains the necessary datatype converters that GTB uses, such as <tt>bam_to_bai.xml</tt>, <tt>bed_to_tabix.xml</tt>, etc. If you have not made local changes to <tt>datatypes_conf.xml</tt>, then simply copy everything from <tt>datatypes_conf.xml.sample</tt> into <tt>datatypes_conf.xml</tt> to make sure you have the latest copy.
</div>

'''Note:''' the above script will run for almost an hour

After restarting Galaxy, Trackster will now know about the size of all chromosomes for many genome builds.



== Adding a new tool ==
<div class="indent">
'''Note:'''  This was the "create a tool" example used in previous years.  This year I switched the time to cover the Galaxy Tool Shed instead.  However, this is such a nice example that I couldn't delete it altogether.  ''Dave C.''
</div>
The [http://samtools.sourceforge.net/ SAM format] contains 12 required fields representing the read and the position it maps to, followed by a variable number of optional fields of the form <tt>''CODE:TYPE:VALUE''</tt>. Suppose we want to filter by these optional fields. We can whip up a quick command line tool in Python (or perl, or awk, or...) to do this.

Let's first create a directory for our new tool:

 $ <span class="enter">mkdir tools/gmod_2012</span>

=== The tool command (script) ===

And then using a [[Linux Text Editors|text editor]], create <tt>tools/gmod_2012/sam_filter.py</tt> containing:

<syntaxhighlight lang="python">#!/usr/bin/env python

"""
Usage: sam_filter.py input_file output_file flag value
"""

import sys

out = open( sys.argv[2], "w" )

for line in open( sys.argv[1] ):
    # Strip end of line and split on tabs
    fields = line.rstrip( "\r\n" ).split( "\t" )
    # Look only at optional fields
    write_line = False
    for field in fields[12:]:
        flag, type, value = field.split( ":" )
        if flag == sys.argv[3] and value == sys.argv[4]:
            write_line = True
    # If any optional field matched, keep the read
    if write_line:
        out.write( line )</syntaxhighlight>

=== The tool wrapper ===

Next, we need to create the tool configuration. Edit the file <tt>tools/gmod_2012/sam_filter.xml</tt> and start with the following skeleton:

<syntaxhighlight lang="xml"><tool id="sam_filter_1" name="SAM Filter">
    <command interpreter="python">

    </command>
    <inputs>

    </inputs>
    <outputs>

    </outputs>
</tool></syntaxhighlight>

First, let's define the output. This tool has a single output, of type <tt>sam</tt>, so we modify the configuration to contain:

<syntaxhighlight lang="xml"><tool id="sam_filter_1" name="SAM Filter">
    <command interpreter="python">

    </command>
    <inputs>

    </inputs>
    <outputs>
        <data name="output1" format="sam" />
    </outputs>
</tool></syntaxhighlight>

The name can be anything, but it will be used later to identify the output file in the command line. Second, let's define the following inputs

* An input dataset of type <tt>sam</tt>
* An input flag, which is selected from a predefined set
* An input value, which can be any text

The resulting configuration:

<syntaxhighlight lang="xml"><tool id="sam_filter_1" name="SAM Filter">
    <command interpreter="python">

    </command>
    <inputs>
        <param type="data" format="sam" name="input1" label="File to filter"/>
        <param type="select" name="flag" label="Optional field to filter on">
            <option value="NM">Edit Distance</option>
            <option value="MD">Mismatching positions / bases</option>
	    <option value="AS">Alignment score</option>
        </param>
	<param type="text" name="value" label="Value to require for flag"/>
    </inputs>
    <outputs>
        <data name="output1" format="sam" />
    </outputs>
</tool></syntaxhighlight>

Finally, we define how to construct our command line based on values for the inputs. The command line is a template, where we can substitute in the value of each input (filenames in the case of datasets). Thus our final tool configuration is:

<syntaxhighlight lang="xml"><tool id="sam_filter_1" name="SAM Filter">
    <command interpreter="python">
        sam_filter.py $input1 $output1 $flag $value
    </command>
    <inputs>
        <param type="data" format="sam" name="input1" label="File to filter"/>
        <param type="select" name="flag" label="Optional field to filter on">
            <option value="NM">Edit Distance</option>
            <option value="MD">Mismatching positions / bases</option>
	    <option value="AS">Alignment score</option>
        </param>
	<param type="text" name="value" label="Value to require for flag"/>
    </inputs>
    <outputs>
        <data name="output1" format="sam" />
    </outputs>
</tool></syntaxhighlight>

We now need to modify <tt>tool_conf.xml</tt> to register our new tool and run Galaxy. Modify the top of <tt>tool_conf.xml</tt> to look like:

<syntaxhighlight lang="xml"><?xml version="1.0"?>
<toolbox>
  <section name="GMOD 2012 Course Tools" id="gmod_2012">
    <tool file="gmod_2012/sam_filter.xml"/>
  </section>
  ...</syntaxhighlight>

and run Galaxy using

 $ <span class="enter">sh run.sh --reload</span>

=== Running the new tool ===

Return to the Galaxy web interface and from the '''Tools''' panel and ''select'' '''GMOD 2012 Course Tools &rarr; SAM Filter'''.

The dataset '''3: Map with BWA...''' should already be '''selected'''. Because we defined the type of data the tool can accept, Galaxy knows this is the only dataset in our history that is valid.

''Select'' '''Edit distance''' for the '''flag''', ''enter'' '''0''' for the '''value''', and ''click'' '''Execute'''.

[[Image:GalaxyExtras_SamFilter_1.png]]

We know have a new dataset '''4: Sam Filter on data 3''' which contains only reads that mapped exactly to the reference.

[[Image:GalaxyExtras_SamFilter_2.png]]


== Configuring Galaxy (2) ==

=== Tool dependencies ===

In the example above, we added a new tool to the default Galaxy installation.  This included, adding the new tool to the Tools panel.  However, many of the tools that are listed in the default Tools panel need additional software installed before they will work (we are currently working on an enhanced dependency management system to make this easier).

Suppose we'd like to analyze some Illumina datasets. We see that the '''Map with BWA''' tool requires us to install [http://bio-bwa.sourceforge.net/ BWA] (surprise!).  To save time, BWA is already compiled on the AMI, so let's copy it to a location on the <tt>PATH</tt>:

 $ <span class="enter">sudo cp /home/ubuntu/Galaxy/BWA/bwa-0.6.2/bwa /usr/local/bin</span>

Now <tt>bwa</tt> can be run from the command line:

 $ <span class="enter">bwa</span>

 Program: bwa (alignment via Burrows-Wheeler transformation)
 Version: 0.6.2-r126
 Contact: Heng Li <lh3@sanger.ac.uk>

 ...

The Galaxy tool also requires a <tt>loc</tt> file which contains the locations of indexes for locally stored genome builds. We don't have any, so we can just use the empty sample:

 $ <span class="enter">cp tool-data/bwa_index.loc.sample tool-data/bwa_index.loc</span>

and then start Galaxy

 $ <span class="enter">sh run.sh --reload</span>

== A second example with NGS data ==

Having made these changes and restarted Galaxy, access it at {{YourUrlGalaxy}}.

<!--
And the <tt> sh ./run.sh</tt> fails.  Type this magic:
<pre class="enter"> sudo ln -s /var/run/postgresql/.s.PGSQL.5432 /tmp/
 sh ./run.sh</pre>

We don't know why.
-->

You will notice that your history has been lost. This is the result of moving to Postgres. On startup, Galaxy will again have created a new database from scratch.

==== 1. Upload datasets ====

We will again use the '''Get Data &rarr; Upload File''' tool to upload data into Galaxy. You can enter multiple URLs into the '''URL / Text''' box. ''Enter:''

<pre>
ftp://ftp.gmod.org/pub/gmod/Courses/2012/SummerSchool/Galaxy/phiX174_genome.fa
ftp://ftp.gmod.org/pub/gmod/Courses/2012/SummerSchool/Galaxy/phiX174_reads.fastqsanger
</pre>
and ''click'' '''Execute''':

[[Image:GalaxyExtras_SecondAnalysis_1.png]]

After uploading the datasets, ''expand them'' in the '''History'''. We can see that Galaxy has successfully detected the file formats of both files.

[[Image:GalaxyExtras_SecondAnalysis_2.png]]

However, we need to make one modification. The FASTQ format is ill-defined, and Galaxy needs to know how the quality scores are scaled. ''Click'' the '''pencil icon for dataset 2''', and ''change'' the '''datatype''' from '''fastq''' to '''fastqsanger''' (be careful, there are many similar choices here). Finally, ''click'' '''Save'''.

=== 2. Run BWA to map reads ===

In the '''Tools''' panel, ''select'' '''NGS: Mapping &rarr; Map with BWA for Illumina'''.

''Change'' the value of the first parameter to '''Use one from the history''' and make sure that '''1: ftp: // ftp.gmod.or...4_genome.fa''' is '''selected'''.

Make sure that for parameter '''FASTQ file''' that '''2: ftp: // ftp.gmod.or...fastqsanger''' is selected.

''Click'' '''Execute'''.

[[Image:GalaxyExtras_SecondAnalysis_BWASettings.png]]

A new dataset is generated containing the mapped reads in {{GlossaryLink|SAM|SAM format}}.

[[Image:GalaxyExtras_SecondAnalysis_3.png]]

== Making genome / alignment data available to Galaxy ==

'''Note:''' We probably won't do this section during class, due to time constraints.  The process for loading genome builds into Galaxy will also be changing shortly.

Galaxy is designed so that a single Galaxy instance can work with many genomes simultaneously. To use certain tools, Galaxy needs to be configured to know where to find data for those genomes. As an example, let's create a new fake genome and configure some data for it.

Every genome in Galaxy needs to have a unique identifier (called the ''build identifier'' or ''dbkey''). For our example, the identifier will be <tt>a_example_1</tt>.

=== Python package management tools ===

'''DO NOT DO THIS NOW''': This has already been done on the AMI.

First, install a few packages that will allow us to index our data. At the terminal, enter:

<tt>&nbsp;&nbsp;$ <span class="enter">sudo apt-get install python-dev python-setuptools python-pip</span></tt>

=== bx-python ===

'''Do not do this now either.'''  This has already been done on your image.

This ''would'' install some support files needed by Python. Next type

<tt>&nbsp;&nbsp;$ <span class="enter">sudo pip install bx-python</span></tt>

This would install the <tt>bx-python</tt> package, a collection of scripts and Python libraries for working with genomic and comparative genomic data.

=== Get datasets for our genome ===

Now, we will download the datasets for our example genome:

 $ <span class="enter">mkdir tool-data/a_example_1</span>
 $ <span class="enter">cd tool-data/a_example_1</span>
 $ <span class="enter">wget ftp://ftp.gmod.org/pub/gmod/Courses/2012/SummerSchool/Galaxy/a_example_1.maf</span>
 $ <span class="enter">wget ftp://ftp.gmod.org/pub/gmod/Courses/2012/SummerSchool/Galaxy/a_example_1.2bit</span>
<div style="font-size: 80%">
::Note: These files are also available on the image in <tt>/home/ubuntu/Galaxy/Data/</tt>.</div>

Next, we will use the script <tt>maf_build_index.py</tt> (put in <tt>/usr/local/bin/</tt> when <tt>bx-python</tt> was installed) to create a binary index that allows fast selection of alignments from a MAF file:

 $ <span class="enter">maf_build_index.py a_example_1.maf</span>
 $ <span class="enter">ls</span>
 a_example_1.2bit  a_example_1.maf  a_example_1.maf.index
 $ <span class="enter">cd ~/Galaxy/galaxy-dist</span>

=== Edit configuration files ===

We now need to edit several configuration files to inform Galaxy of the existence of our genome, and of these associated datasets.

'''NOTE:''' these are TAB separated files, and it is important that the tabs are in the right places. If you have trouble cutting and pasting from the wiki, you can cut and paste from this text file instead:
 ftp://ftp.gmod.org/pub/gmod/Courses/2012/SummerSchool/Galaxy/info.txt

Add this line to <tt>tool-data/shared/ucsc/builds.txt</tt>
<pre class="enter">
a_example_1	Example genome
</pre>

Add this to <tt>tool-data/alignseq.loc</tt>
<pre class="enter">
seq	a_example_1	/home/ubuntu/Galaxy/galaxy-dist/tool-data/a_example_1/a_example_1.2bit
</pre>

Add this to <tt>tool-data/maf_index.loc</tt>
<pre class="enter">
a_example_1 with 3 other species	a_example_1_3way	a_example_1	a_example_1,apiMel2,n_vitripennis_20100409,dm3	/home/ubuntu/Galaxy/galaxy-dist/tool-data/a_example_1/a_example_1.maf
</pre>

Now, stop and start your running Galaxy with
 <span class="enter"><Control-C></span>
 $ <span class="enter">sh run.sh --reload</span>

=== Create dataset using new genome build ===

Now let's see if our new data is available in Galaxy. Click '''Get Data &rarr; Upload File'''.

Paste the following into the '''URL/Text''' box:

<pre>
scaffold0	450	650
scaffold0	2000	3000
</pre>

Set the '''File Format''' to '''bed''', make sure '''Convert spaces to tabs''' is '''selected''', and click in the '''Genome:''' box. You should be able to find your genome by typing '''example'''.

=== Extract sequence corresponding to these intervals ===

In the '''Tools''' menu, ''click'' '''Fetch Sequences &rarr; Extract Genomic DNA'''.

[[Image:GalaxyExtras_Fetch_1.png]]

The tool interface should open in the main panel. The default values are probably fine (make sure the dataset you just created is selected). ''Click'' '''Execute'''. A new dataset will be created containing the DNA for your regions of interest.

[[Image:GalaxyExtras_Fetch_ExtractedDNA.png]]

=== Extract multiple alignments corresponding to these intervals ===

In the '''Tools''' menu, click '''Fetch Alignments &rarr; Extract MAF Blocks'''.

[[Image:GalaxyExtras_Fetch_2.png]]

The tool interface will open in the main panel. Again, the defaults should be reasonable, but ''click'' '''Select All''' to do alignments to all other species. ''Click'' '''Execute'''. A new dataset will be created containing alignments corresponding to your intervals of interest.

[[Image:GalaxyExtras_Fetch_3.png]]

[[Category:Galaxy]]
[[Category:Tutorials]]