Developer Libraries » IOCipher

When sqlfs is under heavy load (like 3x fsx sessions), the SQLite WAL log grows without bounds until it fills all available space. I used the attached patch to run sqlite3_wal_checkpoint_v2() as regular intervals. I ran three instances of fsx:

1. ~/code/guardianproject/libsqlfs/tests/fsx -d -l 10485760 -o 1048576 /mnt/testfile-big
2. ~/code/guardianproject/libsqlfs/tests/fsx -d -l 10485760 -o 1048576 /mnt/testfile-big2
3. ~/code/guardianproject/libsqlfs/tests/fsx -d -c 25 /mnt/testfile-c25

• 1000, the WAL log grew to ~248MB in ~1 minute
• 100, the WAL log grew to ~600MB in ~1 minute
• 1, the WAL log grew to ~40MB in ~1 minute

Fixes suggested by sjlombardo:

1. is that we could do nothing, and leave as is. The WAL is pretty much working as documented, it's just never getting a chance to completely checkpoint because the database is never idle. if you don't think that it's likely for this level of continuous activity to occur in live usage, then most applications should never see this sort of behavior
2. we could introduce some sort of periodic blocking checkpoint, e.g. after every N commits, explicitly force a blocking checkpoint via sqlite3_wal_checkpoint_v2. that would balance out the performance impact of running the checkpoints
3. we could disable wal, and switch back to using the standard journal mode. The changes we made to obtain reserved locks using begin immediate and the addition of the busy handler should make the library stable under load even without wal, though we'd loose the performance boost on write and the improved read-write concurrency