bigPsl Track Format

The bigPsl format stores alignments between two sequences just as PSL files do; however, bigPsl files are compressed and indexed as bigBeds. PSL files are converted to bigPsl files using the program bedToBigBed, run with the -as option to pull in a special autoSql (.as) file that defines the fields of the bigPsl.

The bigPsl files are in an indexed binary format. The main advantage of this format is that only those portions of the file needed to display a particular region are transferred to the Genome Browser server. Because of this, bigPsl files have considerably faster display performance than regular PSL files when working with large data sets. The bigPsl file remains on your local web-accessible server (http, https or ftp), not on the UCSC server, and only the portion needed for the currently displayed chromosomal position is locally cached as a "sparse file". If you do not have access to a web-accessible server and need hosting space for your bigPsl files, please see the Hosting section of the Track Hub Help documentation.

bigPSL format definition

The following autoSql definition is used to specify bigPsl alignment files. This definition, contained in the file bigPsl.as, is pulled in when the bedToBigBed utility is run with the -as=bigPsl.as option.

table bigPsl
"bigPsl pairwise alignment"  
    ( 
    string chrom;       	"Reference sequence chromosome or scaffold"
    uint   chromStart;  	"Start position in chromosome"
    uint   chromEnd;    	"End position in chromosome"
    string name;        	"Name or ID of item, ideally both human readable and unique"
    uint score;         	"Score (0-1000)"
    char[1] strand;     	"+ or - indicates whether the query aligns to the + or - strand on the reference"
    uint thickStart;    	"Start of where display should be thick (start codon)"
    uint thickEnd;      	"End of where display should be thick (stop codon)"
    uint reserved;       	"RGB value (use R,G,B string in input file)"
    int blockCount;     	"Number of blocks"
    int[blockCount] blockSizes; "Comma separated list of block sizes"
    int[blockCount] chromStarts;"Start positions relative to chromStart"

    uint    oChromStart;	"Start position in other chromosome"
    uint    oChromEnd;  	"End position in other chromosome"
    char[1] oStrand;    	"+ or -, - means that psl was reversed into BED-compatible coordinates" 
    uint    oChromSize; 	"Size of other chromosome."
    int[blockCount] oChromStarts;"Start positions relative to oChromStart or from oChromStart+oChromSize depending on strand"
    
    lstring  oSequence;  	"Sequence on other chrom (or empty)"
    string   oCDS;       	"CDS in NCBI format"
  
    uint    chromSize;		"Size of target chromosome"
  
    uint match;        		"Number of bases matched."
    uint misMatch; 		"Number of bases that don't match "
    uint repMatch; 		"Number of bases that match but are part of repeats "
    uint nCount;   		"Number of 'N' bases "
    uint seqType;   		"0=empty, 1=nucleotide, 2=amino_acid"
    )

The value of the oStrand field indicates whether or not the stored psl data should be reverse-complemented before it is outputted or displayed. This is necessary because the bigPsl file stores reference coordinates on the positive strand, as required by the BED format. The strand field indicates whether the positions in oChromStarts are listed from the chromosome beginning (+) or end (-).

Note that the bedToBigBed utility uses a substantial amount of memory: approximately 25% more RAM than the uncompressed BED input file.

Creating a bigPsl track

To create a bigPsl track, follow these steps:

Step 1. If you already have a PSL file created using BLAT or another tool, skip to Step 2. Otherwise, download the example PSL file bigPsl.psl for the human GRCh38/hg38 assembly. You will also want to download the files bigPsl.fa and bigPsl.cds if you would like to use the alternate options described in Step 4, below.

Step 2. Download the bedToBigBed and pslToBigPsl programs from the binary utilities directory.

Step 3. Use the fetchChromSizes script from the same directory to create a chrom.sizes file for the UCSC database with which you are working (e.g., hg38). Alternatively, you can download the chrom.sizes file for any assembly hosted at UCSC from our downloads page (click on "Full data set" for any assembly). For example, the hg38.chrom.sizes file for the hg38 database is located at http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes.

Step 4. Use the pslToBigPsl utility to create a bigPsl file in bed12+13 format that contains the 25 fields described in the bigPsl format definition above. The file must include the 13 extra fields: oChromStart, oChromEnd, oStrand, oChromSize, oChromStarts, oSequence, oCDS, chromSize, match, misMatch, repMatch, nCount, and seqType.

pslToBigPsl bigPsl.psl stdout | sort -k1,1 -k2,2n > bigPsl.txt

If you are using your own PSL file, you may have corresponding FASTA and CDS files that accompany it. You can provide these files as input to pslToBigPsl to generate a more informative bigPsl file:

pslToBigPsl bigPsl.psl -cds=bigPsl.cds -fa=bigPsl.fa stdout | sort -k1,1 -k2,2n > bigPsl.txt

Step 5. Create the binary indexed bigPsl file from your sorted bigPsl input file using the bedToBigBed utility:

bedToBigBed -as=bigPsl.as -type=bed12+13 -tab bigPsl.txt chrom.sizes bigPsl.bb

Step 6. Move the newly created bigPsl file (bigPsl.bb) to a web-accessible http, https, or ftp location.

Step 7. Construct a custom track using a single track line. Note that any of the track attributes listed here are applicable to tracks of type bigBed. The most basic version of the track line will look something like this:

track type=bigPsl name="My Big Psl" description="Some mRNAs Discovered from Data from My Lab" bigDataUrl=http://myorg.edu/mylab/myBigPsl.bb

Step 8. Paste the custom track line into the text box on the custom track management page.

The bedToBigBed program can be run with several additional options. For a full list of the available options, type bedToBigBed (with no arguments) on the command line to display the usage message.

Examples

Example #1

In this example, you will create a bigPsl custom track using an existing bigPsl file, bigPsl.bb, located on the UCSC Genome Browser http server. This file contains data for the hg38 assembly.

To create a custom track using this bigPsl file:

Construct a track line that references the file:

track type=bigPsl name="bigPsl Example One" description="A bigPsl file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigPsl.bb

Paste the track line into the custom track management page for the human assembly hg38.
Click the "submit" button.

Custom tracks can also be loaded via one URL line. The link below loads the same bigPsl track, and sets additional display parameters in the URL:

http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&position=chr1:10000-200000&hgct_customText=track%20type=bigPsl%20name=Example%20bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigPsl.bb%20visibility=pack

After this example bigPsl is loaded in the Genome Browser, click on a track item in the Genome Browser's track display. Note that the details page displays information about the alignment, similar that which is available for PSL tracks, as well as links that display the browser position of the alignment and other detailed information about the alignment.

Example #2

In this example, you will create your own bigPsl file from an existing bigPsl input file.

Save the example bed12+13 file bigPsl.txt to your computer (Step 4 in Creating a bigPsl track, above).
Download the bedToBigBed utility (Step 2, above).
Save the hg38.chrom.sizes text file to your computer. This file contains the chrom.sizes for the human (hg38) assembly (Step 3, above).
Save the autoSql file bigPsl.as to your computer.

Run the bedToBigBed utility to create the bigPsl output file (Step 5, above):

bedToBigBed -as=bigPsl.as -type=bed12+13 -tab  bigPsl.txt hg38.chrom.sizes bigPsl.bb

Place the newly created bigPsl file (bigPsl.bb) on a web-accessible server (Step 6, above).
Construct a track line that points to the bigPsl file (Step 7, above).
Create the custom track on the human assembly hg38 (Dec. 2013), and view it in the Genome Browser (Step 8, above).

Sharing your data with others

If you would like to share your bigPsl data track with a colleague, learn how to create a URL by looking at Example 11 on this page.

Extracting data from the bigPsl format

Because bigPsl files are an extension of bigBed files, which are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory.

bigBedToBed — converts a bigBed file to ASCII BED format.
bigBedSummary — extracts summary information from a bigBed file.
bigBedInfo — prints out information about a bigBed file.

As with all UCSC Genome Browser programs, simply type the program name (with no parameters) at the command line to view the usage statement.

Troubleshooting

If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) in your input file before running the bedToBigBed program.