Monday, May 22, 2006

Global derivatives market expands to record US$298t

Insanity.

The global derivatives market expanded to a record US$298 trillion in the second half of 2005, led by a 34 per cent increase in contracts to insure debt payments, the Bank for International Settlements (BIS) said on Friday.

Credit-default swaps rose to US$13.7 trillion, the Basel, Switzerland-based bank said in its bi-annual derivatives survey.

Growth of the market for derivatives slowed to 5 per cent from 8 per cent six months earlier, based on those contracts traded outside of exchanges.

The credit derivatives market is expanding so quickly that banks have not kept up with the paperwork, leaving trades unconfirmed for days or weeks. Former US Federal Reserve chairman Alan Greenspan on Thursday said he found it 'appalling' that some people were recording the trades on 'scraps' of paper.

Thursday, May 11, 2006

Oh Daytona, don't you spy on me

Daytona:

Historically, DaytonaTM databases have scaled up to the point where either the customer runs out of data to store or else runs out of the means to store the data. Our most recent best example of that is AT&T's 312 terabyte Hawkeye database which is implemented using Daytona as a federation of two HP Integrity Itanium Superdome partitions running HP-UX.

The 64 CPU (SMP) partition houses 6 largish Daytona tables of call detail data. As of Sept 14, 2005, the largest one of these tables contained 743 billion records whose average (compressed) length was 52.2 bytes. This 52.2 bytes uncompresses on average to a uncompressed format measuring 216 bytes, implying an expansion factor of 4:1 . (This uncompressed format is essentially a human-readable ASCII format.) This compressed table takes up 38.8 terabytes; clearly it is much better to buy 38.8 terabytes of disk than 159 terabytes. In all, this partition contains 1.026 trillion records. (FYI, all terabytes here are 1024 based.)

The second partition is qualitatively different: it consists of a low-level data store that in and of itself does not use Daytona and which stores all of the raw data that is distilled into the Daytona records in the first partition. Most of this raw data is in the so-called AMA format, which is a standard format long used by telecoms. There is a lot of information in the raw records that is not present in the corresponding distilled Daytona records and actually the distilled Daytona records contain some information that is not in the raw records. This raw data is stored using an exceedingly effective AT&T compression algorithm called pzip achieving compression ratios of 6:1 here. Special home-grown C code is used to parse unpzip'd AMA data into a human-readable form.

There is a web-interface that binds these two partitions into a federated database: the interface invokes precompiled web-parameterized Daytona queries on the first partition and displays their output. Then if the user would like to see a display of the corresponding raw records located on the second partition, they press a button and the web-interface performs a Daytona query on the first partition that uses the Daytona indices to produce information that informs the custom C code on the second partition where to find the corresponding raw records. This is essentially a nested-loops indexed join. The fact that this integration is so simple and was so easy to implement is a testament to Daytona's flexibility.


And I was happy when I bought a pair of 300GB SATA drives...

What have I missed?

Gold @ 720, silver @ 15, and it looks like the NSA has been collecting info on domestic calls.