Posts | Pascal GP Martin

Plotting SRA database growth

Mon, 06 Dec 2021 00:00:00 +0000

The Sequence Read Archive is NCBI’s database for high throughput sequencing data and the largest public repository for such data.
The data on the growth of the database is available here

Load packages

suppressPackageStartupMessages(library(magrittr))
suppressPackageStartupMessages(library(httr))
suppressPackageStartupMessages(library(ggplot2))

Function to make nicer axis labels

scientific_10 <- function(x) {
xx <- dplyr::case_when(x>=0 & x<=100 ~ as.character(x),
x>100 & log10(x)%%1==0 ~ gsub(".+e\\+", "10^", scales::scientific_format()(x)),
TRUE ~ gsub("e\\+", " %*% 10^", scales::scientific_format()(x)))
parse(text = xx)
}

Download the data in R

sraurl <- "https://www.ncbi.nlm.nih.gov/Traces/sra/sra_stat.cgi"
sra <- httr::GET(url=sraurl) %>%
httr::content(encoding="UTF-8")
sra$date <- as.Date(sra$date, format = "%m/%d/%Y")

Plot with linear scale

sra %>%
dplyr::select(date, bases, open_access_bases) %>%
tidyr::pivot_longer(cols = 2:3, names_to = "type", values_to = "terabases") %>%
dplyr::mutate(terabases = terabases / 1e12) %>%
dplyr::mutate(type = dplyr::case_when(type == "bases" ~ "total bases",
type == "open_access_bases" ~ "open access bases")) %>%
ggplot(mapping = aes(x = date, y = terabases, color = type)) +
geom_line(size = 1.5) +
scale_color_manual(name = NULL,
values = c("total bases" = "darkred",
"open access bases" = "darkblue")) +
scale_y_continuous(labels = scientific_10 ) +
labs(x = "Year", y = expression( Terabases~(10^12) ),
title = "Growth of Sequence Read Archive database") +
theme_bw() +
NULL

Growth of Sequence Read Archive database

Plot with log scale

sra %>%
dplyr::select(date, bases, open_access_bases) %>%
tidyr::pivot_longer(cols = 2:3, names_to = "type", values_to = "bases") %>%
dplyr::mutate(type = dplyr::case_when(type == "bases" ~ "total bases",
type == "open_access_bases" ~ "open access bases")) %>%
ggplot(mapping = aes(x = date, y = bases, color = type)) +
geom_line(size = 1.5) +
scale_color_manual(name = NULL,
values = c("total bases" = "darkblue",
"open access bases" = "darkred")) +
scale_y_log10(labels = scientific_10 ) +
labs(x = "Year", y = "bases" ,
title = "Growth of Sequence Read Archive database") +
theme_bw() +
NULL

Growth (log) of Sequence Read Archive database

Blastn output format 6

Fri, 12 Mar 2021 00:00:00 +0000

See also:

Blast documentation
Blast on Metagenomics Wiki
blastn -help and check -outfmt formatting option

Prepare the database

Only needed if reference sequences are going to be used frequently

makeblastdb \
-in myRefSeqs.fasta \
-out "My/Folder/myRefSeqs_blastdb" \
-dbtype "nucl" \
--parse_seqids \
-logfile My/Folder/logFile.log

Run Blast

-num_alignment default is 250. Modify it if more alignments are expected
-outfmt sets the output format

blastn \
-task megablast \
-query InputSequence.fasta \
-db "My/Folder/myRefSeqss_blastdb" \
-out My/Output/Folder/InputSeq_vs_RefSeqs.res \
-outfmt 6 \
-num_alignments 10000000

Run blast with 2 fasta sequences:

blastn \
-query genes.fasta \
-subject genome.fasta \
-outfmt 6

Default value for -outfmt 6 corresponds to -outfmt "6 qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore"

Output

Column headers:
qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore
In R:

c("qseqid","sseqid", "pident", "length",
"mismatch", "gapopen", "qstart", "qend",
"sstart", "send", "evalue", "bitscore")

Col	name	Description
1.	qseqid	query (e.g., unknown gene) sequence id
2.	sseqid	subject (e.g., reference genome) sequence id
3.	pident	percentage of identical matches
4.	length	alignment length (sequence overlap)
5.	mismatch	number of mismatches
6.	gapopen	number of gap openings
7.	qstart	start of alignment in query
8.	qend	end of alignment in query
9.	sstart	start of alignment in subject
10.	send	end of alignment in subject
11.	evalue	expect value
12.	bitscore	bit score

Generate random genes as GRanges

Sun, 09 Dec 2018 00:00:00 +0000

Generating random genome annotations

For different purposes, I often need to create random gene/features annotations. I usually need these annotations as a GRanges object (see the GenomicRanges package).
I searched a bit on the internet but couldn’t really find what I was looking for.

I want to be able to control the following parameters:

Number of genes
Gene sizes (~gaussian distribution with upper and lower bounds as parameters)
Number of chromosomes
Chromosomes' length

I will assign to each chromosome a number of gene that is proportional to the chromosome length (so all chromosome have ~equal gene densities).

First I need 2 functions:

A function that generates random (and unique) gene names
A function that generates gene lengths (or ‘sizes’) within defined boundaries

Function to generate random gene names

I basically copy/pasted a solution found on this stackoverflow post. I just modified it slightly to ony allow letters (and not numbers) in gene names in order to simplify the definition of the length (number of characters) of gene names, which will be automatically defined based on the gene number.

idGenerator <- function(n, lengthId) {
idList <- stringi::stri_rand_strings(n,
lengthId,
pattern = "[A-Za-z]")
while(any(duplicated(idList))) {
idList[which(duplicated(idList))] <-
stringi::stri_rand_strings(sum(duplicated(idList), na.rm = TRUE),
lengthId,
pattern = "[A-Za-z]")
}
return(idList)
}

Function to generate random gene width/sizes

Gene size distribution heavily depends on the species. For example here is the distribution of gene annotations for 6 species:

#Load libraries
library(tidyverse)
library(TxDb.Hsapiens.UCSC.hg38.knownGene)
library(TxDb.Athaliana.BioMart.plantsmart25)
library(TxDb.Dmelanogaster.UCSC.dm6.ensGene)
library(TxDb.Mmusculus.UCSC.mm10.ensGene)
library(TxDb.Scerevisiae.UCSC.sacCer3.sgdGene)
library(TxDb.Celegans.UCSC.ce6.ensGene)
txdb <- list()
txdb[["hg38"]] <- TxDb.Hsapiens.UCSC.hg38.knownGene
txdb[["mm10"]] <- TxDb.Mmusculus.UCSC.mm10.ensGene
txdb[["dm6"]] <- TxDb.Dmelanogaster.UCSC.dm6.ensGene
txdb[["ce6"]] <- TxDb.Celegans.UCSC.ce6.ensGene
txdb[["tair10"]] <- TxDb.Athaliana.BioMart.plantsmart25
txdb[["sacSer3"]] <- TxDb.Scerevisiae.UCSC.sacCer3.sgdGene
#Get gene size and arange as a data.frame
gnw <- reshape2::melt(lapply(txdb,
function(x) width(genes(x))),
value.name = "GeneSize") %>%
dplyr::mutate(Species = factor(L1,
levels = names(txdb),
ordered = TRUE))
# Personalize colors and labels
mycols = viridis::viridis(8)[-c(1,8)]
names(mycols) <- names(txdb)
mylabs <- c("Homo sapiens (hg38)", "Mus musculus (mm10)",
"Drosophila melaogaster (dm6)", "Caenorhabditis elegans (cer6)",
"Arabidopsis thaliana (TAIR10)", "Saccharomyces cerevisiae (sacCer3)")
names(mylabs) <- names(txdb)
# Plot distributions:
ggplot(gnw, aes(x=log10(GeneSize), fill=Species)) +
geom_histogram(aes(y=..density..),
colour="black", fill="lightgrey") +
geom_density(alpha=.3) +
scale_fill_manual(values = mycols,
labels = mylabs) +
scale_x_continuous(breaks = c(2, 4, 6),
labels = c("100bp", "10Kb", "1Mbp")) +
labs(x = "Gene Size (log scale)",
y = "Density") +
theme_bw() +
theme(panel.grid.minor = element_blank()) +
facet_wrap(vars(Species), nrow=3)

Gene Size Distribution in 6 species

Although it probably does not reflect a “natural” situation, I decided to define gene sizes based on a Gaussian distribution with a choice of a minimum and a maximum gene size.

widthGenerator <- function(n, minWidth, maxWidth) {
if (maxWidth<=minWidth) stop("maxWidth should be < minWidth")
meanWidth <- minWidth + sqrt(maxWidth*minWidth) # take geometrical mean to limit the impact of high maxWidth
sdWidth <- floor((maxWidth-meanWidth)/3)
wdList <- round(rnorm(n, meanWidth, sdWidth))
isOutOfRange <- (wdList<minWidth | wdList>maxWidth)
while(any(isOutOfRange)) {
wdList[isOutOfRange] <- round(rnorm(sum(isOutOfRange),
meanWidth,
sdWidth))
isOutOfRange <- (wdList<minWidth | wdList>maxWidth)
}
return(wdList)
}

Here is what we generate with some relatively usual parameters (gene size betwen 100bp and 100Kb):

df <- data.frame(GeneSize = widthGenerator(1e4, 100, 1e5))
ggplot(df, aes(x=log10(GeneSize))) +
geom_histogram(aes(y=..density..),
colour="black", fill="lightgrey") +
geom_density(alpha=.3, fill = "darkcyan") +
scale_x_continuous(limits = c(1.2, 6.8),
breaks = c(2, 3, 4, 5, 6),
labels = c("100bp", "1kb", "10kb", "100kb", "1Mb")) +
labs(x = "Gene Size (log scale)",
y = "Density") +
theme_bw() +
theme(panel.grid.minor = element_blank())

Function to generate random genes

Now the function that generates random genes as GRanges, using the 2 previous functions above:

CreateGR <- function(ng=1e4, nchr=1, chromlength=1e4, widthMin=100, widthMax=1e4) {
## ng: number of genes
## nchr: number of chromosomes
## withMin/WidthMax: min/max gene size
## chromLength: vector of chromosome lengths
# Check
if (length(chromlength) != nchr) {
stop("length(chromlength) is not equal to number of chromosomes (nchr)")
}
#Number of genes on each chromosome
##Proportion of the genome for each chromosome
ChromProp <- chromlength/sum(chromlength)
##Number of genes per chromosome
GeneNumberByChr <- round(ChromProp*ng,0)
if (sum(GeneNumberByChr) != ng) {
GeneNumberByChr[1] <- GeneNumberByChr[1] + ng - sum(GeneNumberByChr)
}
#Chromosomes
chrNames <- paste0("chr", 1:nchr)
gnChr <- S4Vectors::Rle(rep(chrNames,
times=GeneNumberByChr))
#Strands
gnStrand <- S4Vectors::Rle(sample(c("+", "-"),
size = ng,
replace = TRUE))
#Ranges
##Gene names
gnLength <- ceiling(log(ng) / log(26)) + 1
gnNames <- idGenerator(ng, gnLength)
## start values (even with MaxWidth the genes won't go beyond chromosome borders)
gnStart <- unlist(mapply(sample,
lapply(chromlength,
function(x){(widthMax+1):(x-widthMax-1)}),
GeneNumberByChr,
list("replace" = TRUE)),
use.names = FALSE)
#Gene widths
gnWidth <- widthGenerator(ng,
maxWidth=widthMax,
minWidth=widthMin)
GR <- GenomicRanges::GRanges(seqnames = gnChr,
ranges = IRanges::IRanges(start = gnStart,
width = gnWidth,
names = gnNames),
strand = gnStrand)
GenomeInfoDb::seqinfo(GR) <- GenomeInfoDb::Seqinfo(seqnames = chrNames,
seqlengths = chromlength,
isCircular = rep(FALSE, length(chrNames)),
genome = "mock")
return(GR)
}

Using the following parameters, the function generates a GRanges in ~30 nanoseconds:

chromSizes <- c(30427671, 19698289, 23459830, 18585056, 26975502) # from A. thaliana genome
CreateGR(1e5, nchr=5, chromlength=chromSizes)

## GRanges object with 100000 ranges and 0 metadata columns:
## seqnames ranges strand
## <Rle> <IRanges> <Rle>
## SAmvd chr1 29194029-29199925 -
## sPWlf chr1 29738130-29742506 +
## OeOyx chr1 4094438-4099100 -
## iDJdJ chr1 25695446-25695917 -
## QrgNb chr1 7424826-7430143 -
## ... ... ... ...
## BYNer chr5 4638035-4640168 -
## Wqhxl chr5 23896448-23903443 +
## NwlQQ chr5 21284011-21286015 -
## mwQvN chr5 9237160-9244238 +
## obPRS chr5 8099106-8101112 -
## -------
## seqinfo: 5 sequences from mock genome

Nicer scatter plots in GGgally ggpairs-ggduo

Wed, 14 Nov 2018 00:00:00 +0000

GGally is an extension of ggplot2. I use it mainly as a replacement/refinement of pairs in base R.

Here, I want to replace the default scatter plot used to compare 2 continuous variables with something nicer, especially adapted to plotting large number of points.

Load the library and some RNA-seq data:

library(GGally)
library(airway)
data(airway)

The airway dataset contains data for 4 cell lines treated or not with dexamethasone:

colData(airway)[,1:3]

	SampleName	cell	dex
SRR1039508	GSM1275862	N61311	untrt
SRR1039509	GSM1275863	N61311	trt
SRR1039512	GSM1275866	N052611	untrt
SRR1039513	GSM1275867	N052611	trt
SRR1039516	GSM1275870	N080611	untrt
SRR1039517	GSM1275871	N080611	trt
SRR1039520	GSM1275874	N061011	untrt
SRR1039521	GSM1275875	N061011	trt

Let’s keep a fraction of the data for simplicity

df <- as.data.frame(assays(airway)$counts[,1:4]) #first 4 columns
df <- df[rowSums(df)>4,] #keep genes with some counts
set.seed(123)
df <- df[sample.int(nrow(df),1e3),] #random 1K genes

default behavior of GGally::ggpair:

GGally::ggpairs(log2(df+1))

Here is a better function to make the scatterplots in the lower triangle. It is largely based on this blog post and implements:

viridis coloring of points based on expression level (practically based on effect size captured by the first principal component). I added some code to get consistent coloring even if orientation of the first PC differs between pairs of columns.
alpha based on inverse of 2D density, i.e. low alpha when there are lots of points and higher when points are more scattered

Also, the use of GGally:eval_data_col is described here:

GGscatterPlot <- function(data, mapping, ...,
method = "spearman") {
#Get correlation coefficient
x <- GGally::eval_data_col(data, mapping$x)
y <- GGally::eval_data_col(data, mapping$y)
cor <- cor(x, y, method = method)
#Assemble data frame
df <- data.frame(x = x, y = y)
# PCA
nonNull <- x!=0 & y!=0
dfpc <- prcomp(~x+y, df[nonNull,])
df$cols <- predict(dfpc, df)[,1]
# Define the direction of color range based on PC1 orientation:
dfsum <- x+y
colDirection <- ifelse(dfsum[which.max(df$cols)] <
dfsum[which.min(df$cols)],
1,
-1)
#Get 2D density for alpha
dens2D <- MASS::kde2d(df$x, df$y)
df$density <- fields::interp.surface(dens2D ,
df[,c("x", "y")])
if (any(df$density==0)) {
mini2D = min(df$density[df$density!=0]) #smallest non zero value
df$density[df$density==0] <- mini2D
}
#Prepare plot
pp <- ggplot(df, aes(x=x, y=y, color = cols, alpha = 1/density)) +
ggplot2::geom_point(shape=16, show.legend = FALSE) +
ggplot2::scale_color_viridis_c(direction = colDirection) +
# scale_color_gradient(low = "#0091ff", high = "#f0650e") +
ggplot2::scale_alpha(range = c(.05, .6)) +
ggplot2::geom_abline(intercept = 0, slope = 1, col="darkred") +
ggplot2::geom_label(
data = data.frame(
xlabel = min(x, na.rm = TRUE),
ylabel = max(y, na.rm = TRUE),
lab = round(cor, digits = 3)),
mapping = ggplot2::aes(x = xlabel,
y = ylabel,
label = lab),
hjust = 0, vjust = 1,
size = 3, fontface = "bold",
inherit.aes = FALSE # do not inherit anything from the ...
) +
theme_minimal()
return(pp)
}

Now use in ggpairs (adjust correlation method to be consistent with defaults in GGscatterPlot):

GGally::ggpairs(log2(df+1),
1:4,
lower = list(continuous = GGscatterPlot),
upper = list(continuous = wrap("cor", method= "spearman")))

Now with Pearson correlation coefficient (and removing the correlation coefficient in the upper triangle since we already have them in the lower triangle):

GGally::ggpairs(log2(df+1),
1:4,
lower = list(continuous = wrap(GGscatterPlot, method="pearson")),
upper = "blank") +
theme_minimal() +
theme(panel.grid.minor = element_blank())

We can use the upper triangle to plot other info since the scatter plots include the correlation coefficient.
I illustrate it here with the correlation coefficient for genes with >15 exons vs those with <15 exons:

exonNumber <- elementNROWS(rowRanges(airway[rownames(df),]))
df$MoreThan15Exons <- ifelse(exonNumber>15,
">15ex", "<15ex")
df[,1:4] <- log2(df[,1:4]+1)
GGally::ggpairs(df,
1:4,
lower = list(continuous = wrap(GGscatterPlot, method="pearson")),
upper = list(continuous = wrap(ggally_cor, alignPercent = 0.8),
mapping = ggplot2::aes(color = MoreThan15Exons))) +
theme_minimal() +
theme(panel.grid.minor = element_blank())

The GGscatterPlot function can also be used with GGally::ggduo to study the link between 2 sets of columns/samples. Here an example comparing control vs DEX samples:

GGally::ggduo(df,
c(1,3),
c(2,4),
types = list(continuous = GGscatterPlot)) +
theme_minimal() +
theme(panel.grid.minor = element_blank())