After a long wait, the official paper on BigTable is now available:

http://labs.google.com/papers/bigtable.html

Andrew

err.. how new is this? It's got the old labs logo and says ©2005 Google at the bottom.

To appear in:
OSDI'06: Seventh Symposium on Operating System Design and Implementation,
Seattle, WA, November, 2006.

I just finished reading the paper. There were some interesting tidbits in there and it also helped better my understand of BigTable.

"A BigTable is a sparse, distributed, persistent multi-dimensional sorted map"

The map stores data in the following format:
(row:string, column:string, time:int64) → string

They currently have 388 non-test Bigtable clusters running on about 24,500 machines (assuming they aren't lowballing, like Jeff Dean has hinted at).

We wondered why Analytics had some performance issues? Well, that Bigtable cluster stores 200 terabytes of raw click data. In the table, each row is a user session and is stored as a tuple of the website's name and the initial date of the session.

The one of the crawl tables is 800 terabytes and the other is 50. I'm not sure why they have two.

Personalized search provides one row for each user. It has 93 column families, because it is used for per-user settings by many different teams. Also, when searches and clicks are stored, they are stored in the table under the user's row with the timestamp representing the time of the search or click.

After reading this paper, I want to play around with BigTable even more :). Bigtable sounds like it is really easy to write software for (although, you have to get out of the relational database mindset) and it performs great.