In computer science, data compression is the process of saving information using less space (bits) than the previous representation.
In workplaces where massive information needs to be saved (e.g., Clinical Genomics lab), compression is useful because it reduces the resources required to store and transmit data; however, it consumes time when we compress and decompress the data. Conversely, if data are stored uncompressed, it takes more space, but we don't need to spend time and resources to compress it, and accessing this data takes less time.
The objective of compression is always to reduce the size of the information, trying not to affect the content. However, data compression can be performed either losslessly (without information loss) or lossy (with information loss).
Lossy compression algorithms can remove data to further decrease the size, thereby reducing quality. In lossy compression, the bit rate can be constant or variable. Once the compression has been carried out, the original signal cannot be obtained, although an approximation can be made, and its similarity to the original depends on the type of compression. One example of this happens in lossy image compression used by digital cameras. We are not interested in this kind of compression because, when working with personal data, we need to ensure that health and care data are always collected, analyzed, stored, and shared securely and legally.
On the other hand, in lossless compression, data before and after compressing is accurate. Lossless data compression algorithms usually exploit statistical redundancy to represent data without losing any information; in other words, this process is reversible. It is mainly used in text compression.
It's also important to mention that when data is compressed, there exist different grades or levels of compression. Data can be compressed at different levels, from less compressed to more compressed output data size. Naturally, the higher level we choose, the more time it takes. When time and storage capacity are limited factors, studying the relation between compression execution time and output compressed data size is convenient to find a good balance between these two variables.
During the discussion I had with my supervisor about this learning project, one question emerged: Will the relation between execution time vs. output compressed data size remain constant across different compression levels? Or, on the contrary, is there an interval in which this relation fails, increasing time much more than the compression ratio? Answering this question is what we propose in this blog post.
To achieve this, we are going to repeat the same compression process 10 times, modifying the compression level. Using the program Picard, we will perform a SAM/BAM conversion that enables us to compress the final output. Picard allows us to do this at 10 different levels of compression, from Compression_level = 0, or non-compression, to Compression_level = 9, the largest compression ratio (for a brief explanation of what 'compression ratio' means, see Figure 1). Simultaneously, we can measure the execution time of every compression process to finally analyze the relation between execution time and output data size. Details of how this analysis has been performed can be seen on this GitHub page.
Figure 1. Illustration taken from Wikipedia. The compression rate is the relation between uncompressed data size and the size of this data after compression. In our analyses, compression_level 1 reduced the file from 265 Mb to 76 MB (compression ratio = 3.48). On the other extreme, compression_level 9 reduces the size from 265 MB to 64 (compression ratio = 4.14)
In the next chart (Figure 1), we have illustrated the relationship between execution time (in seconds) on the X-axis and the final output data size on the Y-axis. The size of the file before compressing was 265 megabytes (MB). As expected, it can be seen that the more we compress the file, the more time it takes. From compression level 1 to compression level 5, this relationship is approximately constant, but from compression level 5 to compression level 9, this constant begins to decay. Execution time increases significantly, while the output data size doesn't show a marked difference. This effect starts to become noticeable from compression level 6 to 7 but is more pronounced from compression level 7 to 8 and to 9.
Figure 2. Execution Time vs. Output File Size. Analyses were carried out where we compared the different levels of compression and the execution time needed.
In general, it appears that compressing data beyond level 5 offers no significant benefit. Unless it is absolutely necessary, the compressed file size doesn't vary substantially beyond level 5. In fact, if the compression level is not specified, Picard defaults to compressing the file at level 5.
When working with a 265-MB file, compressing the document up to compression level 9 doesn't make a noticeable difference in time, as can be seen in Figure 2. However, when compressing files of many GBs (e.g., a whole genome), time becomes an essential factor. This effect is illustrated in the next chart (Figure 3), found on a forum discussion website, where the same program was used to compress a ~700 GB file at various compression levels. Figure 3 shows that from level 1 to level 5, the time increased by 115 minutes, and from level 5 to level 9, the time increased by an additional 135 minutes.
Certainly! Here's the revised version of the text:
Comparing our analyses with those found in the other study, I have noticed something intriguing that I would like to note. The relationship between the size compressed and the execution time also depends on the CPU used. The time required to compress data is influenced by the processing capacity of your computer. If I had compressed a 700-GB file, I would have needed more time than what is shown in Figure 3. My compression-to-execution-time ratio for level 9 was 9.41E-05 (resulting from 0.0064GBs divided by 68 seconds), while the ratio shown in Figure 3 for level 9 was 0.01027 (185 GBs divided by 18000 seconds). This indicates that the CPU used in the other study's compression was more powerful than my CPU. From this comparison, we can infer that when determining the necessary compression level, we must take into account the processing power of our computer, as it influences the relationship between execution time and output data size.
In conclusion, in this learning project, we executed the same compression 10 times, altering the degree of compression, and measured the execution time to analyze these variables using a brief script. Finally, we discussed our results and made a comparison with a similar process, enhancing our understanding of the complexities involved in data compression.




