Thursday, May 11, 2006

Oh Daytona, don't you spy on me


Historically, DaytonaTM databases have scaled up to the point where either the customer runs out of data to store or else runs out of the means to store the data. Our most recent best example of that is AT&T's 312 terabyte Hawkeye database which is implemented using Daytona as a federation of two HP Integrity Itanium Superdome partitions running HP-UX.

The 64 CPU (SMP) partition houses 6 largish Daytona tables of call detail data. As of Sept 14, 2005, the largest one of these tables contained 743 billion records whose average (compressed) length was 52.2 bytes. This 52.2 bytes uncompresses on average to a uncompressed format measuring 216 bytes, implying an expansion factor of 4:1 . (This uncompressed format is essentially a human-readable ASCII format.) This compressed table takes up 38.8 terabytes; clearly it is much better to buy 38.8 terabytes of disk than 159 terabytes. In all, this partition contains 1.026 trillion records. (FYI, all terabytes here are 1024 based.)

The second partition is qualitatively different: it consists of a low-level data store that in and of itself does not use Daytona and which stores all of the raw data that is distilled into the Daytona records in the first partition. Most of this raw data is in the so-called AMA format, which is a standard format long used by telecoms. There is a lot of information in the raw records that is not present in the corresponding distilled Daytona records and actually the distilled Daytona records contain some information that is not in the raw records. This raw data is stored using an exceedingly effective AT&T compression algorithm called pzip achieving compression ratios of 6:1 here. Special home-grown C code is used to parse unpzip'd AMA data into a human-readable form.

There is a web-interface that binds these two partitions into a federated database: the interface invokes precompiled web-parameterized Daytona queries on the first partition and displays their output. Then if the user would like to see a display of the corresponding raw records located on the second partition, they press a button and the web-interface performs a Daytona query on the first partition that uses the Daytona indices to produce information that informs the custom C code on the second partition where to find the corresponding raw records. This is essentially a nested-loops indexed join. The fact that this integration is so simple and was so easy to implement is a testament to Daytona's flexibility.

And I was happy when I bought a pair of 300GB SATA drives...


Post a Comment

<< Home