Pagoda has been converted to use a Sqlite database internally rather than flat tab delimited files. This port leveraged the sqlite3 Ruby gem which worked well. This database is different to the database used on the website itself.

There are several benefits from this conversion. Actions like modifying columns, or adhoc queries, can be done using SQL statements. It flushed out issues like non-unique values in columns I hadn't realised were present. All the data is now in a single file which could in theory be distributed.