Timeseries object - Gorilla or TSXor Compression

[ewinnington]

I'm thinking of using Garnet to store a set of timeseries (power production curves) to store the state of a scheduling system.

Since these timeseries have super regular time points (1min, 5min, 15min, 1h) granularities and often multiple identical values one after another, I'm thinking of using a in-memory efficient compression. Generally, in Garnet, the compressed representation of the timeseries could be stored as a string / binary directly, but I'm thinking of a custom object which compresses the timeseries and returns it uncompressed in responses to Get/Set requests.

For the compression, I'm thinking of basing on the compression used in the paper https://www.vldb.org/pvldb/vol8/p1816-teller.pdf

Gorilla compresses data points within a time series with no additional compression used across time series. Each data point is a pair of 64-bit values representing the time stamp and value at that time. Timestamps and values are compressed separately using information about previous values.

I re-implemented the timeseries compression in cpp last time as a test (with no look at performance), but I'll start looking for a C# implementation or write my own and bench it. Also comparing with TSXor https://github.com/andybbruno/TSXor for performance and compression size.

Since with the super regular time series, I have a time delta of deltas returning zero, it will only cost 1 bit per timepoint for the timestamp, so I see a big improvement in storage cost of a (time, value) timeseries. Of course, I could store the start point and granularity as a base of the object, but Gorilla is generic and thus with little effort I can support non-regular timeseries. Most of the value have small jumps between values and long repeated sequences, which means a good compression ratio. (Side note: RLE might also be appropriate for some of the timeseries I have - potentially a pluggable choice of compression based on knowledge of the TS? - but most of my TS do have enough variations that RLE is not ideal)

Has someone else looked at this?

Do you have a similar use case?

Is this also something of interest to the Garnet team?

Have you found a good C# Gorilla Compression or TSXor implementation?

[AlgorithmsAreCool]

While this is quite interesting, is Garnet a good fit for Timeseries data? Garnet excells as a cache, supporting random access to records. I would expect a SQL or dedicated timeseries database would be a better fit for timeseries data since they excel at sequential reads and aggregations.

[vazois]

We haven't looked at time series management, but it might a good fit for Garnet on case-by-case basis.
Unless you want for some reason to specifically rely on SET/GET, there is an alternative command BITFIELD that might suit your needs.
BITFIELD supports SET/GET/INCR according to the Redis protocol but I don't see why we cannot extend it to support logical operators.
If you think it works for your use case and need additionally functionality, we welcome contributions.

[jaelys]

Does native expand the data type like redis time series ?