Feature #2598

downloader Service for F-Droid with support for multiple channel types

Added by hans about 4 years ago. Updated over 3 years ago.

Status:ClosedStart date:11/21/2013
Priority:ImmediateDue date:
Assignee:pserwylo% Done:

0%

Category:-
Target version:new unified Downloader infrastructure
Component:

Description

F-Droid currently handles downloading index.jar and APKs in the UI thread, and it only allows downloading via HTTP. Create a downloader Service that handles all file downloads in the background. It should also be based on an interface or abstract class so that it can handle downloading via multiple types of channels (HTTP, OTRDATA, bluetooth, etc).


Related issues

Related to Bazaar - Feature #2599: UI for showing status of downloads In Progress 11/21/2013
Related to Bazaar - Feature #2627: unit tests for standard downloader functions New 11/21/2013

Associated revisions

Revision c4b0eb9b
Added by Hans-Christoph Steiner over 3 years ago

.Downloader --> .ApkDownloader to distinguish from .net.Downloader

org.fdroid.froid.Downloader is only used for downloading APKs, so name it
appropriately.

refs #2598 https://dev.guardianproject.info/issues/2598

Revision d9f9b682
Added by Hans-Christoph Steiner over 3 years ago

simplify ProgressListener.Event creation and use

This aims to simplify the creation and use of Event objects among classes
that implement the ProgressListener Interface. This is also needed in
order to split out the downloading support from various places and put it
all into a centralized

refs #2598 https://dev.guardianproject.info/issues/2598
https://gitorious.org/f-droid/fdroidclient/merge_requests/29

Revision c1b5bf52
Added by Hans-Christoph Steiner over 3 years ago

implemented IconDownloader for UIL downloads with FDroid classes

This lets UniversalImageLoader (UIL) use FDroid's generic Downloader
infrastucture so that connection configuration all happens based on the URL
in DownloaderFactory.

refs #2598 https://dev.guardianproject.info/issues/2598
refs #2367 https://dev.guardianproject.info/issues/2367

History

#1 Updated by hans about 4 years ago

Some related FDroid tickets:

#2 Updated by hans about 4 years ago

#3 Updated by hans about 4 years ago

  • Priority changed from Normal to Immediate

#4 Updated by hans almost 4 years ago

These are the current relevant classes:
  • .ApkDetails.DownloaderHandler
  • .Downloader
  • .net.Downloader

.net.Downloader is a new effort from pserwylo to lay the groundwork for a single download process for APKs and index files. Icons are downloaded and cached using the UniversalImageLoader library. That means we'll have to figure out how to plug in FDroid's central download service into UniversalImageLoader. Looks like we do that by subclassing com/nostra13/universalimageloader/core/download/BaseImageDownloader and adding it to the com/nostra13/universalimageloader/core/ImageLoaderConfiguration

#5 Updated by pserwylo almost 4 years ago

G'day hans, I had a flip through the classes you mentioned. Indeed, I did write the .net.Downloader. Here is a quick writeup of some of the things I noticed.

org.fdroid.fdroid.net.Downloader

(Somewhat generic) HTTP File downloader

HTTP Specific features

  • etag checking (optional)
  • InputStream is from HttpURLConnection.openConnection()
  • HTTP status code

F-Droid specific stuff

  • ProgressListener

What it DOESN'T know/do

  • Know about what type of file it is downloading (nor should it) - just stores a reference to a java.io.File when done
  • Do any caching - rather it just tells us if we have seen the file before (caching is up to the other classes which use this class)
  • Support interrupting downloads (though it really should support this)
  • Not sure if it works well enough with multiple threads

org.fdroid.fdroid.Downloader

Apk file downloader

  • Checks hash of apk (from index) against downloaded file
  • A lot of the code is to do with enabling canceling of the download (which should be implemented in the HTTP downloader at some point)
  • Custom status codes (not HTTP status codes, rather "Error", "Corrupt", "Unknown")

org.fdroid.fdroid.AppDetails.DownloadHandler

This looks like an inner class which connects the Apk downloader described above and the UI in the AppDetails activity. It is kind of nice that there is this adapter class so that neither class has to understand the other, but would be nice if the updating of the UI was done through progress events.

Potential for refactoring

Given all of this, I had a quick go at refactoring the code base to make these things work a bit better. You can check it our here if you want, but here is a quick summary:

  • One class handles copying from input stream to output stream, and sending progress listeners. This is the .net.Downloader.
  • Any HTTP specific stuff is in a subclass called HTTPDownloader
  • The ApkDownloader instantiates a HTTPDownloader (to download the apk), as does the RepoUpdater (to download the index). *

However, the reason it is only a suggested refactor is because I ended up removing a key functionality (cancelling downloads) and also i didn't actually test the changes (just checked they compiled). But I was keen to see what sort of code could get mushed around find a more suitable place to live.

Also, there will probably need to be one more level of inheritence between .net.HTTPDownloader and .net.Downloader in order to facilitate Bluetooth and other protocols. This would be something like "CacheableDownloader", because the "etag" is pretty HTTP specific, but we'd still want to know if the index has changed or not. Perhaps your Baazar could roll your own cache check similar to e-tags, but slightly different implementation. Then both BluetoothDownloader and HttpDownloader would both extend CacheableDownloader.

Anyway, I'm rambling now. Let me know if any of it didn't make sense.

#6 Updated by pserwylo almost 4 years ago

Oh wow, those headers are really big.

#7 Updated by hans almost 4 years ago

sketch of relationship between .net.Downloader and UIL's Interface:
https://gist.github.com/pserwylo/f4b8d3a12a96ef375199

#8 Updated by hans almost 4 years ago

I think we can just assume that Downloader subclasses will implement caching, and for those that don't they can just always return false when asked whether something is cached. That should keep things readable.

Also, things will be a lot simpler if Downloader just implements the UIL ImageDownloader Interface. Its just a single method which Downloader needs to have anyway. So its just a matter of renaming inputStream() to getStream().

#9 Updated by pserwylo almost 4 years ago

https://gitorious.org/f-droid/pserwylo-fdroidclient/commit/7c31327d3e700c4e4e4b0f9fc451e8b09c8a7ad1

Commit message explains details.

I took your advice, and put an abstract isCached() method in the base class (and renamed eTag to cacheTag). The HttpDownloader returns "statusCode == 304" and until/if you decide to implement caching in other subclasses, you will just need to return false.

Note: Still doesn't cancel downloading. I'm happy to take a look at that at some point in a few days.

#10 Updated by hans almost 4 years ago

  • Target version changed from 0.2 - ChatSecure/Bluetooth to Integrate Kerplapp into FDroid

#11 Updated by pserwylo almost 4 years ago

Now that the Downloader stuff is looking a little better now, I'm going to sink my teeth into a Bluetooth client/server in F-Droid. My idea for the BluetoothDownloader (which extends Downloader) is to do a bit more than just opening up an InputStream on the client, OutputStream on the server, and shove the contents of a file down the pipe (although that would most likely work).

Instead, I'm thinking that during the call to inputStream(), do a little bit of a handshake with the server before returning the BluetoothInputStream, ready to be read to get the contents of the desired file. This handshake will look something like:

  • Client to server: "details:[filename]"
  • Server to client: "notFound" or "cacheTag:[file hash],fileSize:[size]"
  • Client to server: "download:[filename]"
  • Server to client: dump the file contents down the socket

Almost like a cheap, hacky version of HTTP, but specific for what we need here, because it will likely only be two F-Droid apks talking over this protocol.

Any thoughts on this?

#12 Updated by hans almost 4 years ago

@pserwylo this sounds very useful. As far as making it like HTTP, why not use HTTP command? i.e. GET, PUSH, etc? Then they are documented and well known.

#13 Updated by hans over 3 years ago

  • Target version changed from Integrate Kerplapp into FDroid to new unified Downloader infrastructure

#14 Updated by hans over 3 years ago

  • Subject changed from downloader service for F-Droid with support for multiple channel types to downloader Service for F-Droid with support for multiple channel types
  • Status changed from New to In Progress
  • Assignee set to pserwylo

Lots of related work here.
https://gitlab.com/fdroid/fdroidclient/merge_requests/15

pserwylo do you think we still need to use a Service for this?

#15 Updated by pserwylo over 3 years ago

Short answer: An Android "Service" is not necessary for the local repo - or even multiple downloader types.

Long answer: In the long term, it will be necessary for F-Droid in general, namely background downloading, so that a user can:

  • view the details of one app
  • ask to install it
  • then while it is downloading, visit the app details screen for another
  • ask to install that
  • also facilitates notifications that track download progress

Currently, the life of app downloads are tied to the AppDetails screen which created them. They will indeed persist beyond that screen, but if you view the app details for another app, it will probably not play nicely.

#16 Updated by hans over 3 years ago

all of this is complete, except for putting it all into a Service. I created a new issue for that here:
https://gitlab.com/fdroid/fdroidclient/issues/103

#17 Updated by hans over 3 years ago

  • Status changed from In Progress to Closed

Also available in: Atom PDF