Server Architecture v2¶

Overview
The Administration Server should support the following functions:¶

1. Verified Upload of media, form data, and various other electronic files.
2. Secure, encrypted bit storage of verified files.
3. Encrypt/decrypt of messages and uploads
4. Derivative creation of uploaded media and metadata
5. User interaction support for accessing uploads and the derivatives of uploads (e.g., add annotations, reports, watch video, full text search, etc.)
6. Controlled access to derivatives and original files
7. User/Device management
8. Messaging between server and registered devices

These functions should be modularized. Interaction between the modules should be REST based, to allow optimal flexibility and reuse of modules within different contexts. See API documentation for details on the REST calls.

The REST back-end will be python-based, with a Tornado web server/application, and a couchDB (?) database. There will be a hidden TOR service for file upload.

Use Case Details¶

Verified Upload of media, form data, and various other electronic files: Verification Module¶

• A registered user device submits an upload request, with corresponding metadata blob to the trusted destination's server via Tor-wrapped HTTPS; via a REST call (?).
• Server verifies requesting device has corresponding, valid SSL certificate registered with server
• If so, server logs request in its secure database, and generates authentication token based on timestamp and the submitted metadata blob and returns token to requesting user device
• Registered user device then submits upload file, with the authentication token, via To-wrapped HTTPS; via a REST call(?).
• If authentication token and device certificate check out, the server then generates submission record

This module could accept various media types and files (i.e., would not have to be overly tied to the J3M as the metadata used for authentication), so this module could be re-purposed within any system needing a verifiable chain of custody. For example, this system could accept form data (separate from a media file), or an epub file that contained multiple media files, html/xml docs etc.

Notes:
The current setup directly uses J3M as part of the decryption. When you pass media, J3m is passed in encrypted blob. 1st thing it does it determine file type and look for J3M. It then decrypts the J3M.

Already currently starts a file based on keywords. This is before goes into database. Don't necessarily need entire J3m object stored. More suitable for J3m parts to be stored in flat files.

Submission record is created after going through J3Mifier

Secure, encrypted bit storage of verified files: Secure File Storage Module¶

• This module will check for new submissions on a regularly defined interval/or Verification Module will call this module at time submission record is created
• When new submission is found, module requests submission blob, via a REST call (?)
• Module will then submit file to storage "vault", and record location in submission record
• Module will provide REST call to download original submission from "vault" upon request
• TBD: Module will perform regular checksum/bit corrosion checks
• TBD: Module will perform file format obsolescence watch

Encrypt/decrypt of messages and uploads: Encryption Module¶

• server will decrypt object encrypted with passed pgp keys. Function reached via REST call
• server will encrypt objects with passed pgp key. Function reached via REST call

Derivative creation of uploaded media and metadata: Derivatives module¶

• this module will check for new submissions/or Verification Module will call this module at time submission record is created
• Modules request submission blob via REST call
• calls to Encryption Module to decrypt blob
• If submission contains images/video, generates media file derivatives
• breaks submitted metadata into appropriate derivatives context

User interaction support for accessing uploads and the derivatives of uploads: User Interface Module¶

• this module will enable user interaction, through a web interface, to the derivatives and submissions, via REST calls
• the approach will be to provide a template-like responsive, web interface that supports necessary the use of the submitted files, media, and REST calls that can be repurposed. For example
• * Full text search
• * display of regions within a video or image
• * Saved searches
• * Geo search
• * Add annotations
• * Reports (e.g., new submissions, by submission status,
• * data feeds

This will use Sammy to create a state-like app. It will use custom js to achieve pixel specific (non-responsive) placement of image and video regions. And it will support REST calls that break down user interactions to smaller data object transfers to the nonSQL db; as well as support access outside of the template provided.

Use of back-end based templating/ties to a back-end technology (e.g., python, CometD, etc.), will be avoided. This framework should be highly adaptable and be able to repurposed in a variety of formats.

Controlled access to derivatives and original files
See Security below.¶

Messaging between server and registered devices
(Outside of REST responses, should we be tying this to something that could work with OSTel in some form??)¶

Security¶

For round one, security for the REST calls will rely on SSL certificates. The trusted destination will act as a certificate authority, and will grant "requesting" devices/systems a organization matching certificate. A requestor (device, system or browser) must have a corresponding browser certificate or server certificate for the REST call to be accepted.

In addition, a light-weight user login system will be developed with Python, to support user identification, for user interactions that are occurring. This should remain lightweight at the moment, and be easily replaced with institutional authentication systems if needed.

There is also an assumption that the trusted destination server will be hosted on an internal network, for this iteration of development. Special attention should be paid to the security of each of these modules. IPTables should be updated to drop connections from any known malicious IP addresses. Furthermore, certain modules, i.e. those accessible via hidden services, should only accept connections from Tor traffic, meaning all non-tor traffic should be dropped.

CouchDB views could also have individual admins, that are view and privilege specific.

Future iterations will consider and implement additional approaches to authentication and authorizations as needed, such as implementing OAuth for a public API, SE Linux for tightly controlled permissions between modules.

H2. Design Assumptions