Monday, 16 November 2020

Compression-Time vs. Output data size

    In computer science, data compression is the process of saving information using less space (bits) than the previous representation. 


In workplaces where massive information needs to be saved (e.g., Clinical Genomics lab), compression is useful because it reduces the resources required to store and transmit data; however, it consumes time when we compress and decompress the data. Conversely, if data are stored uncompressed, it takes more space, but we don't need to spend time and resources to compress it, and accessing this data takes less time.


The objective of compression is always to reduce the size of the information, trying not to affect the content. However, data compression can be performed either losslessly (without information loss) or lossy (with information loss).


Lossy compression algorithms can remove data to further decrease the size, thereby reducing quality. In lossy compression, the bit rate can be constant or variable. Once the compression has been carried out, the original signal cannot be obtained, although an approximation can be made, and its similarity to the original depends on the type of compression. One example of this happens in lossy image compression used by digital cameras. We are not interested in this kind of compression because, when working with personal data, we need to ensure that health and care data are always collected, analyzed, stored, and shared securely and legally.


On the other hand, in lossless compression, data before and after compressing is accurate. Lossless data compression algorithms usually exploit statistical redundancy to represent data without losing any information; in other words, this process is reversible. It is mainly used in text compression.


It's also important to mention that when data is compressed, there exist different grades or levels of compression. Data can be compressed at different levels, from less compressed to more compressed output data size. Naturally, the higher level we choose, the more time it takes. When time and storage capacity are limited factors, studying the relation between compression execution time and output compressed data size is convenient to find a good balance between these two variables.


During the discussion I had with my supervisor about this learning project, one question emerged: Will the relation between execution time vs. output compressed data size remain constant across different compression levels? Or, on the contrary, is there an interval in which this relation fails, increasing time much more than the compression ratio? Answering this question is what we propose in this blog post.


To achieve this, we are going to repeat the same compression process 10 times, modifying the compression level. Using the program Picard, we will perform a SAM/BAM conversion that enables us to compress the final output. Picard allows us to do this at 10 different levels of compression, from Compression_level = 0, or non-compression, to Compression_level = 9, the largest compression ratio (for a brief explanation of what 'compression ratio' means, see Figure 1). Simultaneously, we can measure the execution time of every compression process to finally analyze the relation between execution time and output data size. Details of how this analysis has been performed can be seen on this GitHub page.


Figure 1. Illustration taken from Wikipedia.  The compression rate is the relation between uncompressed data size and the size of this data after compression. In our analyses, compression_level 1 reduced the file from 265 Mb to 76 MB (compression ratio = 3.48). On the other extreme, compression_level 9 reduces the size from 265 MB to 64 (compression ratio = 4.14) 


In the next chart (Figure 1), we have illustrated the relationship between execution time (in seconds) on the X-axis and the final output data size on the Y-axis. The size of the file before compressing was 265 megabytes (MB). As expected, it can be seen that the more we compress the file, the more time it takes. From compression level 1 to compression level 5, this relationship is approximately constant, but from compression level 5 to compression level 9, this constant begins to decay. Execution time increases significantly, while the output data size doesn't show a marked difference. This effect starts to become noticeable from compression level 6 to 7 but is more pronounced from compression level 7 to 8 and to 9.





Figure 2. Execution Time vs. Output File Size. Analyses were carried out where we compared the different levels of compression and the execution time needed.


In general, it appears that compressing data beyond level 5 offers no significant benefit. Unless it is absolutely necessary, the compressed file size doesn't vary substantially beyond level 5. In fact, if the compression level is not specified, Picard defaults to compressing the file at level 5.

When working with a 265-MB file, compressing the document up to compression level 9 doesn't make a noticeable difference in time, as can be seen in Figure 2. However, when compressing files of many GBs (e.g., a whole genome), time becomes an essential factor. This effect is illustrated in the next chart (Figure 3), found on a forum discussion website, where the same program was used to compress a ~700 GB file at various compression levels. Figure 3 shows that from level 1 to level 5, the time increased by 115 minutes, and from level 5 to level 9, the time increased by an additional 135 minutes.




Figure 3. Execution Time vs. Output File Size chart found on an online scientific discussion website that shows similar analyses carried out for us. In this blog post, this has been a convenient graph for comparison purposes.


Certainly! Here's the revised version of the text:


Comparing our analyses with those found in the other study, I have noticed something intriguing that I would like to note. The relationship between the size compressed and the execution time also depends on the CPU used. The time required to compress data is influenced by the processing capacity of your computer. If I had compressed a 700-GB file, I would have needed more time than what is shown in Figure 3. My compression-to-execution-time ratio for level 9 was 9.41E-05 (resulting from 0.0064GBs divided by 68 seconds), while the ratio shown in Figure 3 for level 9 was 0.01027 (185 GBs divided by 18000 seconds). This indicates that the CPU used in the other study's compression was more powerful than my CPU. From this comparison, we can infer that when determining the necessary compression level, we must take into account the processing power of our computer, as it influences the relationship between execution time and output data size.


In conclusion, in this learning project, we executed the same compression 10 times, altering the degree of compression, and measured the execution time to analyze these variables using a brief script. Finally, we discussed our results and made a comparison with a similar process, enhancing our understanding of the complexities involved in data compression.

Intolerance vs.gene length

Why do some genes seem to withstand variants, and others do not?

In clinical genomics labs, determining which mutations are most likely to produce disease is a challenge. While some variants can occur in regions of the genome without any major consequences, others have a significant impact on gene protein structure and function, resulting in pathologies.


There are currently approaches that attempt to classify variants according to their probabilities of conferring risk of diseases. These methods include the measure of conservation at the phylogenetic level, the repercussion on protein structure depending on amino acid characteristics, or analysis of population allele frequency.


In 2013, Slave Petrovski and collaborators developed a gene-level assessment that ranked genes based on their likelihood to influence disease. This intolerance scoring system assesses which genes have more or less functional genetic variation than expected. The ranking of all genes can be obtained from this Excel table. This interesting paper can be summarized in the following figure (Figure 1). This study analyzed the existing variety of approximately 17,000 genes to compare the number of common variants that probably affect gene function against the total number of genetic variants per gene. According to their distribution, Petrovski and collaborators developed the Residual Variation Intolerance Score (RVIS). An RVIS less than 0 indicates that a gene has fewer common functional variants than expected; conversely, an RVIS greater than 0 indicates that a particular gene has fewer common functional variants that affect functions.



MUTATION TOLERANCE AND INTOLERANCE IN THE HUMAN GENOME

 A regression plot illustrating the relation between X, genes are plotted based on their number of all variants and Y, the number of common functionally relevant variants. Mutation-intolerant genes under the regression line and Mutation tolerant genes above the line.


After this study, a question emerged between my supervisor and me. Assuming, first, that there is a constant rate of mutation across all human genes, it follows that the longer a gene is, the more likely that this gene will become mutated. Second, most of the genes are functionally important, meaning they are intolerant to variants. So, should we expect that longer genes would have fewer functionally important regions?


To answer this question, we propose the following experiment: We will compare the intolerance scoring rank provided by Slave Petrovski and collaborators with the length of these genes. We explain our analysis step-by-step in this GitHub repository. The results obtained and discussion from this analysis are commented on in this blog post.


In the following scatter plot, we have represented the length of 15,742 genes against the RVIS. We have also used the bivariate correlation or Pearson correlation to measure the linear correlation between these two variables. The results provided (p=0.1629) tell us that there is no linear correlation between the length of a gene and the intolerance score.


 



Scatter diaphragm between gene length against RVIS. Notice both variables have been normalized. 



With the assumptions we made at the beginning, we expected to see a negative regression; longer genes should have a lower RVIS. However, the results obtained do not provide evidence for this theory. One reason could be because the assumptions we made are not completely correct. Mutation rate does vary across the genome. Another reason could be due to the methodology I followed to measure the length of the genes. I have taken the complete sequence of the genes, but a more accurate methodology would have been to measure the length of the total exons per gene, excluding introns. (I will conduct this analysis in the future and comment on the results in a separate blog post.)

The fact that we did not see any relation between gene length and the intolerance score could be explained because RVIS ranks genes in terms of their likelihood to influence disease, ignoring variants that, though disruptive, do not produce any phenotype. (This is to be discussed with Ben.) Another analysis we could perform is comparing the score provided by PolyPhen-2 or a similar tool for all human genes against the length of all human genes. This is, in fact, mentioned in the discussion of the paper.


Difficulties I have found during performing this analysis.




When two data frames are merged, we need to ensure that the common column between them has the same values (in this case, gene names). I have found two issues. First, the same genes can be referred to in more than one way. If this happens, these genes will be deleted in the merged table (the data frame that contains information from the two data frames we want to link). To avoid this problem, we can configure the algorithm to display the genes that did not match, and then consider the reason for this discrepancy. The second issue I have found is similar to the first: not only is the algorithm incapable of identifying genes with different names, but even if the difference is minimal, such as a capital letter or a white space, it is considered a mismatch. In other words, the values of the two common columns have to be absolutely identical; otherwise, the program will not identify the gene, and that gene will be eliminated. This is significant because it is a source of information loss.


Reflection: How can this analysis have a positive impact on patient's health?




Two elements can be considered in this short analysis. First, any tool or data that assists in the interpretation of personal genomes is welcome, and RVIS is a significant contribution in this regard. My brief analysis performed here does not have a direct impact, as it is for learning purposes; however, the skills developed during this activity can be applied in future practice. For example, scripts like those developed in this activity can be used in clinical trials when large data frames with patient information are combined to find unexpected relationships between variables. An example might be patients undergoing treatment for one pathology who end up discovering beneficial effects for other, seemingly unrelated, pathologies. These kinds of studies are now possible due to the development of new technologies that enable us to store massive amounts of information, as well as the creation of new algorithms such as machine learning or deep learning that enable us to find relationships between seemingly unrelated variables.

More information about what can be studied with personal health data can be found in the following link. As a consequence, this data is becoming increasingly valuable. To read about the value of personal health data, the following news article provides some insights.