Format file and compression parameters

Sagues · October 8, 2021, 8:31am

I’ve exported unstructured grids to xml files in ascii, binary and appended format. I’ve used LZ4 and LZMA compression with level 5 and 9.

All exported files in ascii (zlibcompression, default option) have a size of 691 Mb. If I exported in binary and appended (zlib compression) all files have a size of 216 Mb. If I used LZ4 or LZMA compression with 5 level, both formats (binary and appended) all files have a size of 180 Mb. And if I use the same compressions with 9 level all files have a size of 180 Mb too.

First of all I would like to know the diference among binary and appended, specially in relation to size files and speed of writing.

Moreover, I would like to know the diference among compressions LZ4 and LZMA.

Thank you very much

Sagues · December 14, 2021, 8:44am

The following link allows to see a benchmark to know the performance of all compressors

gist.github.com

https://gist.github.com/oldcai/7230548

compression_benchmark.py

import time
import requests
import zlib
# pip install lz4 pylzma
import lz4
import pylzma as lzma


def analysis_ratios(source):
    time_before_zip = time.time()

This file has been truncated. show original

output.txt

test a million repeated characters:
source len: 1000000
zipped len: 992
lz4ed  len: 3936
lzmaed len: 221
zip  compression ratio: 0.00099
lz4  compression ratio: 0.00394
lzma compression ratio: 0.00022
compression ratio zip/lz4/lzma: 1 / 0.25 / 4.488688
zip  time: 0.007863044738769531

This file has been truncated. show original