Bug #308

Exremely long IOCipher write time when copying from SD card

Added by Anonymous over 5 years ago. Updated almost 5 years ago.

Status:ClosedStart date:09/20/2012
Priority:NormalDue date:
Assignee:abeluck% Done:


Target version:0.1


It takes forever (~2 minutes in reality) to copy a file from the SD card to the encrypted VFS. The same operation only takes about a second when I don't use IOCipher. After performing a trace it's clear that the bulk of the time, ~99%, is spent in the Posix.pwriteBytes() method of the IOCipher library. I've attached a sample Android app (VfsTest.zip) and the trace file (IOCipher.trace) to help illustrate the problem as well as a copy of the file I was transferring (it's a pic of Titanaboa from the Smithsonian).

I tested this on a Galaxy Nexus (Google Play store version) and an HTC Incredible 2 running 2.3.7.

IMG_20120915_151830.jpg - image to transfer (1.47 MB) Anonymous, 09/20/2012 08:45 pm

VfsTest.zip - sample app (466 KB) Anonymous, 09/20/2012 08:45 pm

IOCipher.trace - trace file (2.03 MB) Anonymous, 09/20/2012 08:45 pm


#1 Updated by abeluck over 5 years ago

  • Status changed from New to In Progress
  • Assignee set to abeluck

Reproduced.. checking this out.

#2 Updated by abeluck over 5 years ago

updating from email convo:

Hey Abel,

I saw you were the assignee on the Bug report so I thought I would update
you on a work around I'm trying out.

I noticed (somewhat obviously) that the bigger I made the buffer the faster
it would copy the file. I'm not overly familiar w/ the intricacies of I/O,
let alone encrypted I/O, but it made sense that if the performance
increases with fewer writes I'd try to just do one big write. So I tried
copying the file to a ByteArrayOutputStream as an intermediary and then
just writing to the encrypted VFS in one call
(FileOutputStream.write(baos.toByteArray()). This significantly improved
write time to almost as quick as the native file system.

This seems like a viable work around for me at the moment given the small
file sizes I'm dealing with. However, it's probably a bad idea for a
general work around.


#3 Updated by abeluck over 5 years ago

from email:

Hey Aaron,

Thanks for the bug report

Each write call to libsqlfs (done by that Posix method pwriteBytes) has
some significant SQL overhead. So, while I don't have the data to
support it yet, my hypothesis is that the SQL overhead is causing poor

Your workaround reduces the number of calls to libsqlfs, and hence the
SQL overhead. We've talked about buffering small writes in IOCipher,
then flushing them periodically in larger writes to libsqlfs, we'll see..

Another way to implement file copying is using FileChannels. The result
is the same, a single large write is performed rather than a bunch of
smaller writes.
Here's an example snippet https://gist.github.com/3763841

We're still in the process of profiling and optimizing IOCipher and its
underlying components. Your feedback is quite helpful, definitely keep
it coming.

Soon, I hope to write some docs for IOCipher describing how using it
differs from normal Java IO. Using larger buffers is a great example of
the types of tips I hope to include there.



#4 Updated by n8fr8 about 5 years ago

  • Target version set to 0.1

#5 Updated by hans almost 5 years ago

  • Status changed from In Progress to Resolved

I think we've resolved this with the latest SQLite WAL locking improvements. Abel, if you agree, please close this.

#6 Updated by abeluck almost 5 years ago

  • Status changed from Resolved to Closed

Absolutely. We aren't as fast as normal I/O but we're order of magnitudes faster now!

Also available in: Atom PDF