Sunday, 22 November 2020

CSV to VCF converter

Today (close to 2021), many companies and research organisations are still working with Excel spreadsheets or similar versions. Despite the fact that spreadsheets were (and still are) excellent tools for providing reactive computational models, the thing is that several major issues concern the statistical and computing community. The main problems found when working with spreadsheets are related to disorganization and errors. Simple research on Google can give you hundreds of examples but I would like to share one I found on BBC News called Excel: Why using Microsoft's tool caused Covid-19 results to be lost in which 16,000 positive corona-virus cases were unreported in the UK owing to a spreadsheets formatting error.

For this learning project, I have been asked to convert a Comma-Separated Value (CSV) file into Variant Call Format. A CSV file is a simple text file that uses coma to separate values. This format has been used for decades to store tabular data in which lines have the same number of fields. Excel spreadsheets or similar can interpret this data and show you as a data frame (what we see when we open a simple Excel document). 

If we open a CSV file with any text editor we will see something like this:

Domain,Origin,Gene,GRCh38 coordinates;ref/alt allele,Transcript,CDS change and protein change,Predicted consequences,Population germline allele frequency (1KG | gnomAD),Genomics England germline allele frequency,VAF,Alt allele/total read depth,Genotype,COSMIC ID,ClinVar ID,ClinVar review status,ClinVar clinical significance,Gene-level actionability,Variant-level actionability,Gene mode of action,PharmGKB_ID

2,somatic,AXIN2,17:65558019;C>T,ENST00000307078,c.602G>A;p.(Arg201Lys),missense_variant, -  | <0.00005,,0.46,46/99,,,,,,,,tumor suppressor gene,

2,somatic,FLT4,5:180630768;C>T,ENST00000261937,c.187G>A;p.(Gly63Arg),missense_variant, -  | <0.00005,,0.53,34/64,,,,,,,,oncogene,

2,somatic,PTPRB,12:70539697;T>A,ENST00000334414,c.5706A>T;p.(Lys1902Asn),missense_variant,- | -,,0.49,51/105,,,,,,,,tumor suppressor gene,

2,somatic,WT1,11:32396364;G>T,ENST00000332351,c.1142C>A;p.(Ser381*),stop_gained,- | -,,0.11,Oct-91,,COSM27307;COSM1317326;COSM5879249,,,,,,both,

2,somatic,WT1,11:32396372;AAG>A,ENST00000332351,c.1132_1133delCT;,frameshift_variant,- | -,,0.21,20/97,,COSM96861,,,,,,both,

3,somatic,ACVR2B,3:38482270;C>T,ENST00000352511,c.1147C>T;p.(Arg383Cys),missense_variant,- | -,,0.11,Oct-88,,COSM5449864;COSM5449863,,,,,,,

3,somatic,AHNAK,11:62522261;A>T,ENST00000378024,c.12156T>A;p.(His4052Gln),missense_variant;(SR)(PON), -  |  - ,,0.13,Dec-92,,COSM4411983;COSM4411984,,,,,,,

3,somatic,AHNAK2,14:104945249;A>G,ENST00000333244,c.10202T>C;p.(Met3401Thr),missense_variant;(SR)(PON),- | -,,0.1,Sep-93,,,,,,,,,

3,somatic,AHNAK2,14:104943909;T>C,ENST00000333244,c.11542A>G;p.(Thr3848Ala),missense_variant;(SR)(PON), -  | <0.00005,,0.1,Oct-97,,,,,,,,,

When this file is opened by Excel, each line represents one row and each value separated by comma are allocated in separated columns.

On the other hand, while CSV files are used in a very varied context,  VCF are specific format of text files used for storing gene sequence variation. Comparing CSV with VCF, this is completely new being developed to resolve data storage issues in large-scale genotyping projects such as the 1000 Genomes Project. 

A simple example of a VCF file is shown below (taken from Wikipedia).

##fileformat=VCFv4.3

##fileDate=20090805

##source=myImputationProgramV3.1

##reference=file:///seq/references/1000GenomesPilot-NCBI36.fasta

##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x>

##phasing=partial

##INFO=<ID=NS,Number=1,Type=Integer,Description="Number of Samples With Data">

##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Depth">

##INFO=<ID=AF,Number=A,Type=Float,Description="Allele Frequency">

##INFO=<ID=AA,Number=1,Type=String,Description="Ancestral Allele">

##INFO=<ID=DB,Number=0,Type=Flag,Description="dbSNP membership, build 129">

##INFO=<ID=H2,Number=0,Type=Flag,Description="HapMap2 membership">

##FILTER=<ID=q10,Description="Quality below 10">

##FILTER=<ID=s50,Description="Less than 50% of samples have data">

##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">

##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">

##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read Depth">

##FORMAT=<ID=HQ,Number=2,Type=Integer,Description="Haplotype Quality">

#CHROM POS      ID         REF   ALT    QUAL  FILTER   INFO                             FORMAT       NA00001         NA00002          NA00003

20     14370    rs6054257  G     A      29    PASS    NS=3;DP=14;AF=0.5;DB;H2           GT:GQ:DP:HQ  0|0:48:1:51,51  1|0:48:8:51,51   1/1:43:5:.,.

20     17330    .          T     A      3     q10     NS=3;DP=11;AF=0.017               GT:GQ:DP:HQ  0|0:49:3:58,50  0|1:3:5:65,3     0/0:41:3

20     1110696  rs6040355  A     G,T    67    PASS    NS=2;DP=10;AF=0.333,0.667;AA=T;DB GT:GQ:DP:HQ  1|2:21:6:23,27  2|1:2:0:18,2     2/2:35:4

20     1230237  .          T     .      47    PASS    NS=3;DP=13;AA=T                   GT:GQ:DP:HQ  0|0:54:7:56,60  0|0:48:4:51,51   0/0:61:2

20     1234567  microsat1  GTC   G,GTCT 50    PASS    NS=3;DP=9;AA=G                    GT:GQ:DP     0/1:35:4        0/2:17:2         1/1:40:3


To convert a CSV file into a VCF file, I first needed to understand the minimum structure and necessary requirements for creating these files. For this conversion, the VCFv4.3.tex version was consulted. A VCF file starts with meta-information data (lines prefixed with '##'), continues with a header line (prefixed with “#”), and concludes with data lines, each containing information about position in the genome and genotype information (text fields separated by tabs). Empty values are not allowed, so a dot ('.') is used instead. For specifying a new line, both LF (“\n”) and CR+LF (“\r\n”) are allowed.


Meta-information lines are key=value pairs, and 'fileformat' is always required. The rest of the meta-information lines are optional.


As can be seen in the example, the 8 header line names (CHROM, POS, ID, REF, ALT, QUAL, FILTER, and INFO) are mandatory. Additional columns for extra information are allowed, and the format for each column follows a strict code that can be read in the documentation. Now, I will explain how the conversion was carried out.


When I faced this project, the first consideration was what kind of program I wanted to create for the conversion. Did we want a program that would only convert the CSV provided by my supervisor, or did we want a more flexible code that could recognize the vast majority of CSVs that store gene sequence variation? This question is not trivial, as the complexity of the code increases significantly between the two.


If we were to write a script that only converted the provided CSV (or a very similar one), we would need to modify the script every time we were required to do a similar conversion in the future. Thinking practically and aiming for efficiency (and also following the Best Practices for Scientific Computing), the most appropriate approach would have been to develop a simple interface that allowed anyone without informatics knowledge to convert the document themselves. With my current skills in programming, this would have been a challenging but acceptable project.


Since this is a learning project, I decided to start by creating a simple script that only converts the provided CSV into a VCF file. This is the one I will explain in this post. A more advanced one has been proposed as a paired collaboration assessment in the Module Programming of the Master's in Genomic Medicine.


Reflective thoughts 

This has been my first experience dealing with file conversion. The code generated here is simple, but this activity made me consider real-world problems such as interoperability or incompatibility issues due to formatting. Most of my time working on this task was spent contemplating possible errors I might encounter if I used this script with different input files. As mentioned above, this code is not very flexible, so as soon as the CSV format varies, errors are likely to arise. Finding this conversion quite fascinating, I proposed (and it was accepted) an assessment for the Module Introduction to Programming. In this assessment, we are considering different approaches to create code that can recognize as many different types of input as possible. 

In this activity, we are contemplating the creation of a user-friendly interface that would allow staff without a strong background in bioinformatics to convert variant data stored in CSV files into VCF files themselves. A new post will be written in which I will describe the new program, along with some discussion about this experience.



Monday, 16 November 2020

Compression-Time vs. Output data size

    In computer science, data compression is the process of saving information using less space (bits) than the previous representation. 


In workplaces where massive information needs to be saved (e.g., Clinical Genomics lab), compression is useful because it reduces the resources required to store and transmit data; however, it consumes time when we compress and decompress the data. Conversely, if data are stored uncompressed, it takes more space, but we don't need to spend time and resources to compress it, and accessing this data takes less time.


The objective of compression is always to reduce the size of the information, trying not to affect the content. However, data compression can be performed either losslessly (without information loss) or lossy (with information loss).


Lossy compression algorithms can remove data to further decrease the size, thereby reducing quality. In lossy compression, the bit rate can be constant or variable. Once the compression has been carried out, the original signal cannot be obtained, although an approximation can be made, and its similarity to the original depends on the type of compression. One example of this happens in lossy image compression used by digital cameras. We are not interested in this kind of compression because, when working with personal data, we need to ensure that health and care data are always collected, analyzed, stored, and shared securely and legally.


On the other hand, in lossless compression, data before and after compressing is accurate. Lossless data compression algorithms usually exploit statistical redundancy to represent data without losing any information; in other words, this process is reversible. It is mainly used in text compression.


It's also important to mention that when data is compressed, there exist different grades or levels of compression. Data can be compressed at different levels, from less compressed to more compressed output data size. Naturally, the higher level we choose, the more time it takes. When time and storage capacity are limited factors, studying the relation between compression execution time and output compressed data size is convenient to find a good balance between these two variables.


During the discussion I had with my supervisor about this learning project, one question emerged: Will the relation between execution time vs. output compressed data size remain constant across different compression levels? Or, on the contrary, is there an interval in which this relation fails, increasing time much more than the compression ratio? Answering this question is what we propose in this blog post.


To achieve this, we are going to repeat the same compression process 10 times, modifying the compression level. Using the program Picard, we will perform a SAM/BAM conversion that enables us to compress the final output. Picard allows us to do this at 10 different levels of compression, from Compression_level = 0, or non-compression, to Compression_level = 9, the largest compression ratio (for a brief explanation of what 'compression ratio' means, see Figure 1). Simultaneously, we can measure the execution time of every compression process to finally analyze the relation between execution time and output data size. Details of how this analysis has been performed can be seen on this GitHub page.


Figure 1. Illustration taken from Wikipedia.  The compression rate is the relation between uncompressed data size and the size of this data after compression. In our analyses, compression_level 1 reduced the file from 265 Mb to 76 MB (compression ratio = 3.48). On the other extreme, compression_level 9 reduces the size from 265 MB to 64 (compression ratio = 4.14) 


In the next chart (Figure 1), we have illustrated the relationship between execution time (in seconds) on the X-axis and the final output data size on the Y-axis. The size of the file before compressing was 265 megabytes (MB). As expected, it can be seen that the more we compress the file, the more time it takes. From compression level 1 to compression level 5, this relationship is approximately constant, but from compression level 5 to compression level 9, this constant begins to decay. Execution time increases significantly, while the output data size doesn't show a marked difference. This effect starts to become noticeable from compression level 6 to 7 but is more pronounced from compression level 7 to 8 and to 9.





Figure 2. Execution Time vs. Output File Size. Analyses were carried out where we compared the different levels of compression and the execution time needed.


In general, it appears that compressing data beyond level 5 offers no significant benefit. Unless it is absolutely necessary, the compressed file size doesn't vary substantially beyond level 5. In fact, if the compression level is not specified, Picard defaults to compressing the file at level 5.

When working with a 265-MB file, compressing the document up to compression level 9 doesn't make a noticeable difference in time, as can be seen in Figure 2. However, when compressing files of many GBs (e.g., a whole genome), time becomes an essential factor. This effect is illustrated in the next chart (Figure 3), found on a forum discussion website, where the same program was used to compress a ~700 GB file at various compression levels. Figure 3 shows that from level 1 to level 5, the time increased by 115 minutes, and from level 5 to level 9, the time increased by an additional 135 minutes.




Figure 3. Execution Time vs. Output File Size chart found on an online scientific discussion website that shows similar analyses carried out for us. In this blog post, this has been a convenient graph for comparison purposes.


Certainly! Here's the revised version of the text:


Comparing our analyses with those found in the other study, I have noticed something intriguing that I would like to note. The relationship between the size compressed and the execution time also depends on the CPU used. The time required to compress data is influenced by the processing capacity of your computer. If I had compressed a 700-GB file, I would have needed more time than what is shown in Figure 3. My compression-to-execution-time ratio for level 9 was 9.41E-05 (resulting from 0.0064GBs divided by 68 seconds), while the ratio shown in Figure 3 for level 9 was 0.01027 (185 GBs divided by 18000 seconds). This indicates that the CPU used in the other study's compression was more powerful than my CPU. From this comparison, we can infer that when determining the necessary compression level, we must take into account the processing power of our computer, as it influences the relationship between execution time and output data size.


In conclusion, in this learning project, we executed the same compression 10 times, altering the degree of compression, and measured the execution time to analyze these variables using a brief script. Finally, we discussed our results and made a comparison with a similar process, enhancing our understanding of the complexities involved in data compression.

Intolerance vs.gene length

Why do some genes seem to withstand variants, and others do not?

In clinical genomics labs, determining which mutations are most likely to produce disease is a challenge. While some variants can occur in regions of the genome without any major consequences, others have a significant impact on gene protein structure and function, resulting in pathologies.


There are currently approaches that attempt to classify variants according to their probabilities of conferring risk of diseases. These methods include the measure of conservation at the phylogenetic level, the repercussion on protein structure depending on amino acid characteristics, or analysis of population allele frequency.


In 2013, Slave Petrovski and collaborators developed a gene-level assessment that ranked genes based on their likelihood to influence disease. This intolerance scoring system assesses which genes have more or less functional genetic variation than expected. The ranking of all genes can be obtained from this Excel table. This interesting paper can be summarized in the following figure (Figure 1). This study analyzed the existing variety of approximately 17,000 genes to compare the number of common variants that probably affect gene function against the total number of genetic variants per gene. According to their distribution, Petrovski and collaborators developed the Residual Variation Intolerance Score (RVIS). An RVIS less than 0 indicates that a gene has fewer common functional variants than expected; conversely, an RVIS greater than 0 indicates that a particular gene has fewer common functional variants that affect functions.



MUTATION TOLERANCE AND INTOLERANCE IN THE HUMAN GENOME

 A regression plot illustrating the relation between X, genes are plotted based on their number of all variants and Y, the number of common functionally relevant variants. Mutation-intolerant genes under the regression line and Mutation tolerant genes above the line.


After this study, a question emerged between my supervisor and me. Assuming, first, that there is a constant rate of mutation across all human genes, it follows that the longer a gene is, the more likely that this gene will become mutated. Second, most of the genes are functionally important, meaning they are intolerant to variants. So, should we expect that longer genes would have fewer functionally important regions?


To answer this question, we propose the following experiment: We will compare the intolerance scoring rank provided by Slave Petrovski and collaborators with the length of these genes. We explain our analysis step-by-step in this GitHub repository. The results obtained and discussion from this analysis are commented on in this blog post.


In the following scatter plot, we have represented the length of 15,742 genes against the RVIS. We have also used the bivariate correlation or Pearson correlation to measure the linear correlation between these two variables. The results provided (p=0.1629) tell us that there is no linear correlation between the length of a gene and the intolerance score.


 



Scatter diaphragm between gene length against RVIS. Notice both variables have been normalized. 



With the assumptions we made at the beginning, we expected to see a negative regression; longer genes should have a lower RVIS. However, the results obtained do not provide evidence for this theory. One reason could be because the assumptions we made are not completely correct. Mutation rate does vary across the genome. Another reason could be due to the methodology I followed to measure the length of the genes. I have taken the complete sequence of the genes, but a more accurate methodology would have been to measure the length of the total exons per gene, excluding introns. (I will conduct this analysis in the future and comment on the results in a separate blog post.)

The fact that we did not see any relation between gene length and the intolerance score could be explained because RVIS ranks genes in terms of their likelihood to influence disease, ignoring variants that, though disruptive, do not produce any phenotype. (This is to be discussed with Ben.) Another analysis we could perform is comparing the score provided by PolyPhen-2 or a similar tool for all human genes against the length of all human genes. This is, in fact, mentioned in the discussion of the paper.


Difficulties I have found during performing this analysis.




When two data frames are merged, we need to ensure that the common column between them has the same values (in this case, gene names). I have found two issues. First, the same genes can be referred to in more than one way. If this happens, these genes will be deleted in the merged table (the data frame that contains information from the two data frames we want to link). To avoid this problem, we can configure the algorithm to display the genes that did not match, and then consider the reason for this discrepancy. The second issue I have found is similar to the first: not only is the algorithm incapable of identifying genes with different names, but even if the difference is minimal, such as a capital letter or a white space, it is considered a mismatch. In other words, the values of the two common columns have to be absolutely identical; otherwise, the program will not identify the gene, and that gene will be eliminated. This is significant because it is a source of information loss.


Reflection: How can this analysis have a positive impact on patient's health?




Two elements can be considered in this short analysis. First, any tool or data that assists in the interpretation of personal genomes is welcome, and RVIS is a significant contribution in this regard. My brief analysis performed here does not have a direct impact, as it is for learning purposes; however, the skills developed during this activity can be applied in future practice. For example, scripts like those developed in this activity can be used in clinical trials when large data frames with patient information are combined to find unexpected relationships between variables. An example might be patients undergoing treatment for one pathology who end up discovering beneficial effects for other, seemingly unrelated, pathologies. These kinds of studies are now possible due to the development of new technologies that enable us to store massive amounts of information, as well as the creation of new algorithms such as machine learning or deep learning that enable us to find relationships between seemingly unrelated variables.

More information about what can be studied with personal health data can be found in the following link. As a consequence, this data is becoming increasingly valuable. To read about the value of personal health data, the following news article provides some insights.