Revision as of 18:20, 13 October 2011 (view source) Anthony Cros (Talk | contribs) (→‎More exercises with MartConfigurator) ← Older edit		Revision as of 18:20, 13 October 2011 (view source) Anthony Cros (Talk | contribs) (→‎More exercises with MartConfigurator) Newer edit →
Line 399:		Line 399:
	## Right-click on the newly created tab --> ''Set GUI type'' --> ''MartWizard''		## Right-click on the newly created tab --> ''Set GUI type'' --> ''MartWizard''
	# Use remote mart (URL Mart) Using backward compatibility (from previous BioMart versions: <= 0.7):		# Use remote mart (URL Mart) Using backward compatibility (from previous BioMart versions: <= 0.7):
−	## In the ''Source'' view: ''Add Mart'' (see [[Add Mart]])	+	## In the ''Source'' view: ''Add Mart'' (see [[=== Add Mart ===]])
	## Select ''URL Mart'' --> ''Next''		## Select ''URL Mart'' --> ''Next''
	## Input the following values:		## Input the following values:
Line 411:		Line 411:
	Backward compatibility is run in the background in order to convert a mart configuration in 0.7 format to one in 0.8		Backward compatibility is run in the background in order to convert a mart configuration in 0.7 format to one in 0.8
	# Add access points for URL mart:		# Add access points for URL mart:
−	## Add an access point to the "Form" GUI tab [[Creating Access Point]]	+	## Add an access point to the "Form" GUI tab [[=== Creating Access Point ===]]
−	## Add an access point to the "Wizard" GUI tab [[Creating Access Point]]	+	## Add an access point to the "Wizard" GUI tab [[=== Creating Access Point ===]]
	* deploy		* deploy
	* exam chromosome filter (singleSelect), gene type filter (multiSelect), how to edit dropdown options for those filters		* exam chromosome filter (singleSelect), gene type filter (multiSelect), how to edit dropdown options for those filters

---

Revision as of 18:20, 13 October 2011

{{#icon: Biomart250.png|BioMart|200|BioMart}} {{#icon: 170px-October2011Logo.png|October 2011 - Toronto

2009 GMOD Summer School - Toronto, Canada}}

BioMart Session 2011 GMOD Community Meeting & Ontario Institute for Cancer Research Toronto, Canada 14 October 2011 Junjun Zhang, Elena Rivkin and Anthony Cros

__NOTITLE__

This tutorial walks you through how to install and configure a local installation of BioMart.

System overview and installation

Prerequisites for BioMart:

1. Software: Java 1.6, Ant and SVN client
2. OS: Linux, Mac & Windows
3. Server: min. 1 GB memory, 3 GB for better performance

Add a section here for installing VirtualBox or VMware and start up the VM

1. where to download virtualbox etc.
2. where to get VM image (download? or get it from USB key)

Downloading & Installing BioMart

Two components are necessary for this tutorial: MartBuilder is installed form an older version of BioMart (0.7), and MartConfigurator is installed from the most recent version (0.8) (until all features are fully ported to the new version)

Installation of MartBuilder and MartConfigurator has already been done on the VM image under ~/biomart_0_7.template and ~/biomart_0_8.template, but we are going to do once together for demonstration purposes.

Refer to BioMart Documentation for more information.

Biomart 0.7 (MartJ -> MartBuilder + MartRunner)

MartJ contains applications necessary to create a Mart: MarBuilder and MartRunner.

Check Section 2 in Installing BioMart for more information.

Download & extract tarball content with:

 $ mkdir ~/biomart_0_7 # arbitrary name of course
 $ cd ~/biomart_0_7
 $ wget ftp://anonymous@ftp.ebi.ac.uk/pub/software/biomart/martj_current/martj-bin.tgz # or get it from MartJ
 $ tar zxvf martj-bin.tgz # creates "martj" directory
 $ cd martj
 $ ls bin/martbuilder.sh bin/martrunner.sh # what we care about

Biomart 0.8 (MartConfigurator)

Checkout latest release using SVN:

 $ mkdir ~/biomart_0_8 # arbitrary name of course
 $ cd ~/biomart_0_8
 $ svn co https://code.oicr.on.ca/svn/biomart/biomart-java/branches/oct_3_2011 # creates "oct_3_2011" directory
 $ mv oct_3_2011 martconfigurator # friendlier name
 $ cd martconfigurator
 $ ant # build project with ant; in the future, you may use: ant clean dist
 $ ls dist/scripts/martconfigurator.sh dist/scripts/biomart-server.sh # what we care about

Building mart

This section will show you how to create a (simplified) mart containing 1 dataset based on the VEGA database, using MartBuilder.

Describing mart (MartBuilder)

Creating/loading sample mart

Start MartBuilder

From the directory ~/biomart_0>7/martj, issue:

 $ bin/martbuilder.sh

MartBuilder should open with an empty mart.

Add a schema

Choose Schema->Add to open the dialog to add a schema.

Click the Test button to ensure we can connect to the database. Click the Add button in order to proceed with the dataset description.

You should now see the source database show in MartBuilder.

Note that if you did not specify a schema when creating your database, your tables will be in the default schema for your platform:

1. MySQL: has no notion of schema, database acts as a schema
2. PostGreSQL: public
3. Microsoft SQL Server: dbo
4. Oracle & DB2: the username of the user who created the database

Choose dataset

We are going to create a dataset based on the tables "gene" and "transcript" (as main and submain tables respectively, as described in presentation)

Materializing mart (MartBuilder->MartRunner)

Your dataset does not actually exist yet. In order to create it, you need to generate the SQL for it then run that SQL against your database.

BioMart offers a tool (MartRunner) that does that for you, using JDBC. The SQL used is as ANSI-compliant as possible, with some exceptions based on the RDBMS in use.

1. We now going to transform the source data into target dataset, but before that, we have to create a target database:

 $ MY_DESTINATION_DATABASE=mini_hsap_vega_mart
 $ mysql -hlocalhost -P3306 -ubiomart -pbiomart -e "create database $MY_DESTINATION_DATABASE"

In theory one could materialize a mart in the same database as the source one provided there is no table name conflicts, but this is strongly discouraged.

Also we have to have MartRunner running. Lets run it over a local port (any free one will do).

2. Start martrunner with:

 $ MY_MART_RUNNER_PORT=9876
 $ bin/martrunner.sh $MY_MART_RUNNER_PORT

MartBuilder will send the materialization SQL to MartRunner through that port (in this example, 9876), and MartRunner will execute the transformation SQL.

Note that MartRunner may be run from another machine, as we specify its connection parameters to MartBuilder (see below).

We go back to MartBuilder, and click on the Build Mart button to pop up the following dialog:

3. Monitor progress using the Monitor MartRunner progress

Your schema now contains a mart with a complete dataset ready for configuring with MartConfigurator.

Configuring mart (MartConfigurator) - basic

This section will show you how to configure the created VEGA mart using MartConfigurator

Start MartConfigurator

Start MartConfigurator with the following command in the directory of your installation:

 $ dist/scripts/martconfigurator.sh

Brief intro to the main interfaces of the MC

• screenshot here

Add Mart

1. Wizard step 1/4:

2. Wizard step 2/4:

3. Wizard step 3/4:

4. Wizard step 4/4:

5. Done!

Creating Access Point

Customizing configuration

Deploying BioMart Server (Jetty)

In order to deploy the application on Jetty, one has 2 options:

• GUI approach: Using the Start Server button in the upper right corner
• CLI approach:
  • Save the the current state (referred to as registry in MartConfigurator)
  • Manually start the server:

 $ dist/scripts/biomart-server.sh start

• GUI approach:
  • Using the Start Server button will result in prompting a dialog for saving the registry.
  • The server will be deployed on your local machine, port 9000 by default (unless specified differently in .biomart.properties file).
  • Your web page should open automatically when the local server is ready.

Stopping the server

• GUI approach: Using the Stop Server button in the upper right corner
• CLI approach:

 $ dist/scripts/biomart-server.sh stop

Using deployed application

The interface is intuitive, typical interaction involve choose a number of attributes and filters and obtaining results accordingly.

Configuring mart (MartConfigurator) - more advanced

From the configuration editor (see previous section)

Configuration editor

Description

• north: Configuration tree
  • Show sources view: where one can modify config in a way that will affect all access points
  • Show access point view: where one can modify individual access point's configs specifically
  • Show both: most typical usage
    • north west: source view
    • north east: access point view

• south: Editor frame
  • south west: property names
  • south east: property values

Notes

Only properties whose name appears in blue font can be modified.

Each view as a Find search box that allows to quickly find attributes/filters based on their names

Also very convenient, the attributes and filters in each view offer an Show in the [opposite view] item in their context menu. It filters the opposite view side to show the counterpart filter/attribute

Containers

Typical properties that can be changed:

• displayname
• description
• hide: whether attribute should be hidden or not
• enableselectall: offer checkbox allowing selection of all attributes/filters in the given container

Attributes

Typical properties that can be changed:

• displayname
• description
• hide
• linkouturl: to make result values linkable to an external resource
• datatype: String, Integer, Float, Boolean (useful mostly for reordering)

Filters

Typical properties that can be changed:

• displayname
• description
• type (covered in the next section)
• qualifier: whether to use "=", "<=", ">=", "LIKE" operations
• spliton: separator for filter list values (see below)

Filters types

Regular filters

Typically used to (suprisingly) filter data based on values of interest (for instance a specific gene, or location, or type, ...)

Filter lists

They are considered filter themselves, but actually contain a list of regular filters

Linking

Set Filter typesSet Attribute URLSelect GUI type

Creating Links between sources

If two data sources contain common information (e.g. a Gene/Protein ID), this can be used to create a link, allowing filters and attributes from one data source to appear in the other. These are called “pointer attributes” and “pointer filters,” and the attribute or filter to which they point is called the “target. ”To add a pointer to an access point, double click on that access point in the portal tab to edit it. In the top left corner of the editing window, click on the Import from sources button.

You will be given a list of the existing data sources to choose which one you would like to make an access point for. After giving the new access point a name of your choice, it will appear in the GUI tab. Double-clicking on the access point icon will open a new window that allows you to modify the access point. References

More exercises with MartConfigurator

1. Rename 'default' GUI tab to "Form":
   1. Right-click on the tab --> Rename
   2. enter new name: "Form"
2. Add new GUI wizard:
   1. Click on the "+" sign next to the latest GUI tab (currently just renamed "Form")
   2. Enter name of the new tab: "Wizard"
   3. Right-click on the newly created tab --> Set GUI type --> MartWizard
3. Use remote mart (URL Mart) Using backward compatibility (from previous BioMart versions: <= 0.7):
   1. In the Source view: Add Mart (see === Add Mart ===)
   2. Select URL Mart --> Next
   3. Input the following values:
      1. Protocol: http
      2. Host: www.biomart.org
      3. Port: 80
      4. Path: /biomart/martservice
   4. Choose source: vega (url) --> Next
      1. Select all 3 datasets
      2. Uncheck import each datasets to individual marts, one dataset per mart --> Finish

Backward compatibility is run in the background in order to convert a mart configuration in 0.7 format to one in 0.8

1. Add access points for URL mart:
   1. Add an access point to the "Form" GUI tab === Creating Access Point ===
   2. Add an access point to the "Wizard" GUI tab === Creating Access Point ===

• deploy
• exam chromosome filter (singleSelect), gene type filter (multiSelect), how to edit dropdown options for those filters
• exam Multiple Chromosomal Regions (Chr:Start:End:Strand) filter
• create attribute list
• add pathway dataset via URL data source
• drag&drop pathway name attribute to vega AP under Form GUI tab, this will trigger link creation (detailed below)
• link creation between pathway and vega gene. use ensembl gene id as the attribute/filter pair to build the link
• deploy the server and select pathway name together with other attribute from vega gene dataset, exam the result
• drag&drop pathway name filter to vega AP under Wizard GUI tab, then deploy the sever and choose pathway filter then exam result

Querying a BioMart server via REST API

MetaData queries

http://localhost:9000/martservice/marts

http://localhost:9000/martservice/datasets?mart=gene_vega

http://localhost:9000/martservice/accesspoints?datasets=&mart=gene_vega

http://localhost:9000/martservice/attributes?dataset=hsapiens_gene_vega&mart=gene_vega

http://localhost:9000/martservice/filters?dataset=hsapiens_gene_vega&mart=gene_vega

Data query

<xml> <Query processor="TSV" header="true" limit="-1" client="webbrowser"> <Dataset name="hsapiens_gene_vega" config="gene_vega_ap"> <Filter name="chromosomal_region" value="2:1000000:2000000:1,4:9000000:11000000:-1"/> <Filter name="biotype" value="protein_coding"/> <Attribute name="vega_gene_id"/> <Attribute name="vega_transcript_id"/> <Attribute name="vega_translation_id"/> <Attribute name="chromosome_name"/> <Attribute name="start_position"/> <Attribute name="end_position"/> <Attribute name="strand"/> <Attribute name="band"/> </Dataset> </Query> </xml>

Paste this piece of XML in a web browse address as:

http://localhost:9000/martservice/results?query=paste_query_xml_string_here

To read more about BioMart, refer to the recent articles describing the BioMart software and its applications.

1. Zhang J, Haider S, Baran J, Cros A, Guberman JM, Hsu J, Liang Y, Yao L, Kasprzyk A. BioMart: a data federation framework for large collaborative projects. Database (Oxford). 2011 Sep 19:2011:bar038.

2.Guberman JM, Ai J, Baran J., et al. BioMart Central Portal: An Open Database Network for the Biological Community. Database (Oxford). 2011 Sep18;2011:bar041.

3. Zhang J, Baran J, Guberman JM, Haider, S, Hsu J, Liang Y, Rivkin E, Wang J, Whitty B, Wong-Erasmus M, Yao L, Kasprzyk A. International Cancer Genome Consortium Data Portal - a One-stop Shop for Cancer Genomics Data. Database (Oxford). 2011 Sep19;2011:bar026.