North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Time Series databases

  • From: michael.dillon
  • Date: Thu Feb 08 07:15:08 2007

> > Going back to this thread, deals in 
> financial transaction
> > databases where they store millions of ticks.  They appear to have a
> > transactional based language with a solution that appears 
> to be robust and
> > fail resistant.

> hmm, that is quite interesting. and apparently people out there _are_
> using it for things like counter values and what not - based on their
> FAQ. I'd absolutely love to know more about the algorithms and math
> behind something like kdb+

KX publish a bunch of information about their product. Their lineage
goes back to APL and the J language, both of which found most of their
users in financial services.

However, the general issue of time-series databases is more interesting.
Google will take you to lots of research using keywords like:

time-series database delta wavelet search indexing maxima

Of course, don't use them all at once. To give you a flavor of the stuff
that people have done, here is a slide presentation on compression and
indexing that does not use averages like RRD does:

In addition to Google, it is a good idea to search CiteSeer because it allows you to quickly track down
references to other papers so you can read them all as a set.

I don't think there are any full-blown open-source implementations that
you could integrate into your own systems. There is stuff like Metakit which stores data by column rather
than by row. And people who have thought about how to efficiently store
time-series probably cobbled together their own systems using bsddb or

If you are stuck in the SQL world, then check out these articles on star
and snowflake schemas. and follow up the references at
the bottom of the page.