The new app update 3.0.5 (which will roll out slowly over the next days), there is a very first implementation of the end-to-end encryption.
Opt-in & beta
The end-to-end encryption feature is opt-in at the moment, not mentioned in the app and it is very possible, that I will change the implementation again before publicly releasing it (which would mean that you have to re-do the steps setting up the sync as described below)!
To opt-in to/unlock the end-to-end encryption, you need to open any entry, put the text ENABLE E2EE into the heading field and save the entry. A small info toast should show up, informing you have successfully unlocked the feature. This does not yet mean that the sync is now encrypting all your data, it only makes the end-to-end encryption available when freshly setting up the sync (more on that below).
Similarly, if you want to opt-out again, do the same with the text DISABLE E2EE
New "v5 sync"
Currently, when you look in the Diarium folder on your Cloud storage (unless you are using iCloud, in which case the folder is not visible to you), you see that Diarium stores files & folders that are called "v4_database", "v4_mediaindex" and "v4_Media". In order to have a clear separation between the old sync and the new sync, the new files created with the new syncing implementation will use the "v5" prefix instead of the "v4" prefix.
"v5 sync" does not only support the new end-to-end encryption for which a user has to provide a password, it also aims to build a foundation to make future changes to sync easier to facilitate - e.g. when potentially introducing the "multiple journals" feature request or when adding new encryption algorithms.
The default sync mode changes as well - instead of using a hard-coded password that encrypts the data, the data will not be encrypted at all, which can lead to faster sync speeds.
Migration from "v4 sync" to "v5 sync"
In the long run, the goal is to migrate all users to the new "v5 sync", however there are difficulties:
- Users could already have large databases uploaded to their cloud, potentially multiple gigabytes. Migrating to v5 would mean to re-upload (& re-encrypt) all that data.
- Users upgrade the apps on their devices at different times
Updating apps should be as seamless as possible, and for users who are happy with the sync, nothing should change (for now). Therefore, the new "v5 sync" will only be offered to users when the app does not find an existing "v4_database" file on the Cloud storage. At some point in the future, if features like "multiple journals" require it, upgrading to the "v5 sync" might be required and I might have to introduce a "migration assistant UI" into the app, that guides the user through the process and also deletes the unneeded "v4" files from the Cloud after the migration was successful. Until then, this flow chart shows how the app should decide if to use "v4 sync" or "v5 sync":
Testing the "v5 sync"
Before starting, please create a backup of your database in the app settings!
In order to test the "v5 sync" with E2EE you need to go through the following steps:
- Update the app to min. 3.0.2 on all your devices
- Save an entry with
ENABLE E2EE as the header on all your devices
- Delete all files from your Diarium folder in your Cloud
- Start sync on one device -> The app should display the dialog allowing you to choose between "default" mode and "end-to-end encryption"
- Start sync on other devices -> App should download the v5_database and ask for the password if needed (no password needed if E2EE is not used)
Technical details
For both "default" and "e2ee" mode, the first 2 bytes of the "v5_database" file contain meta information:
- The first byte is the "sync version". At the moment, "0" is the only valid value for it. In the future, it might be incremented to support additional features like "multiple journals" for which the rest of the file data might have a different structure
- The second byte is the "encryption mode". The value "0" stands for "no encryption", the value "1" stands for "e2ee" which is described in more detail below. Other values are currently not supported, but may be added in the future.
- For the "no encryption" mode, the rest of the file data is the ZIP-compressed SQLite database, not encrypted.
- For the "e2ee" mode, the first 16 bytes after the 2 "version bytes" are the PBKDF salt (16 bytes), the HMAC (32 bytes), the encryption IV (16 bytes) and the encrypted master key (48 bytes). This data is used to decrypt the master key using the user password, if needed. These 114 bytes are the "database header" and store on the user device. The same header is prepended for each database upload. For more details on how these salts/IVs/HMACs are generated, see below. After the "database header", there is another HMAC (32 bytes) and IV (16 bytes) used for decryption of the database.
- The media files aren't compressed. When using the "e2ee" mode, they are encrypted with the master key and contain the 32 byte HMAC and 16 IV for the AES-CBC-HMAC decryption before the payload
- The mediaindex file is compressed, but never encrypted (as it doesn't contain confidential information)
When the user enters a password, the app uses 600.000 rounds of PBKDF2-HMAC-SHA256 and a 16 byte salt (600.000 rounds is a recommendation of OWASP - https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#pbkdf2) to derive an 32 byte AES key. This AES key is used to encrypt the master key (which is generated in the next step) - the encrypted master key is part of the "database header" which is prepended to the "v5_database" before upload.
The 32 byte master key is created using a cryptographically strong random number generator - this is the master key used to encrypt/decrypt all the journal data. This master key is stored in a secure location on the user device, together with the "database header" (e.g. the Keychain on iOS/macOS).
AES265-CBC is used as encryption algorithm, in an "Encrypt-then-MAC" mode. Before the app attempts to decrypt the master key, the database or a media file, it first checks if the corresponding HMAC of the IV + ciphertext is correct. If not, the sync errors out right away. This prevents oracle attacks, even though they wouldn't be likely in the usage scenario of Diarium anyways. In order to not use the same key for both HMAC-SHA256 and the AES256, the HKDF algorithm is used with SHA256 in order to derive two individual keys - The encryption key (for AES256) is derived using the "info" parameter of the UTF8 representation of the string "e" while the authentication key (for HMAC-SHA256) is derived using the "info" parameter of the UTF8-representation of the string "a".
One more thing worth mentioning: If a user would encrypt & upload files (incl. media), then delete the "v5_database", then set up the sync again with a different password & master key, the previously uploaded media files would be incompatible (not decryptable) with the new master key. Diarium wouldn't immediately detect that, as it cannot download and check all the files on every sync. Diarium would simply see that there are media files already present and would not re-upload and encrypt with the correct, new key. In order to prevent this problem from happening, Diarium will upload the "v5_mediaindex" and the "v5_Media" folder with a suffix that is generated from the master key. More specifically, the suffix is the first 8 characters of the SHA256 hash of the master key. If there is no master key (when using the "non-e2ee" sync), then the string "6e340b9c" is used as suffix. This suffix ensures that Diarium doesn't "see" those files when querying available media files and instead re-uploads them, encrypted with the right key.
Your feedback is important
Please let me know if you have any questions or suggestions on how to improve the feature. This is perhaps the biggest change I have done in the sync ever so it's important that I will find and fix all bugs before releasing it publicly. Also, if you think that some description in the app is too technical, please let me know. Many of Diarium's users have no technical background, e.g. don't know what "end-to-end encryption" means. I don't want to confuse or scare away those users.
