Set password for cloud sync encryption

timo

I wish it was so trivial as you make it sound. I can't answer all your points, I'll pick just a few:

Eddie It is concerning that a developer of a DIARY/JOURNAL app takes security so lightly

The app started as a hobby project a few years ago, mainly for my girlfriend when I was still in uni. Back then and probably still now I am no security expert. I just wanted to get the app done as quickly as possible for others to be used and I never expected the app to be big enough to have a community forum where I have to excuse myself for the bad security decisions I made.
It's impossible to make everything right on the first try. And now switching from the existing sync solution to a new one comes with many obstacles that you only see when you dive into it.

Eddie Also, all you need to do is warn users that if they lose their password, they will lose their data.

I wish it was that easy... You expect users to actually read such a warning. And then the 1 star reviews start pouring in because users forget their password and the app has no "recover/reset password" button.
People have already lost access to their journals because they didn't use the Cloud sync and then just uninstalled the app, deleting all their data. I feel bad for them and I want to minimize these cases to happen. Having a Cloud sync feature with lots of warnings and options to click through doesn't exactly lower the bar for people to use it and backup their journal that way.

Eddie By the way, I have been in IT for 20 years and have been a programmer for 10+

I can tell. But not everyone has that sense of privacy and for the majority of people, other features are just more important. And in order to have a functioning business, I need to be aware of that. According to you, this sync problem is an absolute deal-breaker of the app. According to another user, not being able to put stickers on the journal entry is the deal-breaker and AAALL the other journal apps support it. And to the next user, not being able to checkbox-lists in the rich text editor is the deal-breaker. The list goes on and on, just read through the other feature requests here in the forum. Now, who do I listen to?

Eddie because you see it as low priority

I think the fact that is the single feature request in the forum (among the literally hundreds) that is marked as "planned" should show that I take this seriously.
I work on many apps on multiple platforms and do my best to make every user happy. I already work much more than 40 hours a week. I wish I would have already implemented this feature, but it just wasn't possible.

And as someone in IT for 20 years, you can set up a personal WebDAV server that you secure appropriately (maybe even running in your private home network and not allowing internet access) and use that as a sync option.

I appreciate that you participate in this forum and I hear you that you want to see this improved. Just please also understand that this cannot be done in a day or two. I care a lot about the users of this app (hell, I even created a community forum where I provide support and everyone can submit and vote for feature requests) and it is a bit frustrating to hear when the opposite is claimed.

Eddie

timo timo Sorry, I didn't mean to imply that you don't have a lot on your plate that is pulling you in many different directions. I admit that I may be a bit biased when it comes to security, not because I am paranoid, but because there are MANY examples of nearly daily data breeches, ransomware attacks, and identity theft, not to mention possible extortion attempts. It might sound like a stretch but it has happened.

I also know the struggles and frustrations of dealing with users who think THEIR feature is the most critical one. It is just something I feel is important for such sensitive information, but also understand your dilemma. It is a business at the end of the day and the most in demand features are going to give you the biggest return. There is nothing wrong with that logic and prioritization strategy.

I see that you do think it is something that is lacking and would be useful, and therefore it is on your To-Do list. It does not seem like it is that complex, but I am thinking of it as a data encryption implementation problem and not a cloud syncing integration one as well. And I’ll admit, I do not know how much that part changes the amount of effort required to add such a feature.

The WebDAV approach sounds like it could be an option. I am a little rusty and have not dealt with it before, but I guess it could be a project and learning experience for me to get back into a little coding.

P.S. We have to be careful with those "hobby" projects. That is what Linux started out as but quickly expanded into the complex ubiquitous Operating System we know it as today. But then again, Linus Torvalds had volunteers to help him stay on top of the constantly expanding list of bugs and new features. ;-)

Thanks for the quick reply!

timo

Thanks again for your comments, really appreciate it that you care so much about having this topic of the app improved. And I am determined to take care of it. I wish I could give a specific timeline on when it's gonna happen, but if I did, I would probably miss it anyways.
Currently there's a lot to do and improve "behind the scenes" that doesn't express itself in new features, it's rather getting the app smaller, faster and more reliable. That's why there aren't so many "new features" announced here, but that doesn't mean I stopped working on the app.
When changing the sync I will have to do a lot preparing, implementing and testing to make sure this feature will cause as little trouble as possible, so it won't be ready next week (and also not next month). But it's the next "big thing" I will do on the app (no stickers or checkbox lists for now 😉).

Alex

I've just recently signed up and would like to say thanks for the awesome app. Really happy with it.

I would like to echo how important it is to me that there is an option to set up a a custom passphrase for encryption.

I had thought I would back up to my synology NAS but tbh I'm not comfortable leaving webDAV enabled on the NAS just for this one reason. So I've turned it off an am syncing with an old Dropbox account which is used for nothing else. Knowing the data stored there is encrypted with a custom passphrase would be a fantastic peace of mind for me.

This was rather a lengthy way of saying "+1"! 😂

Thanks again for a lovely app!

Sir

Hi our dear timo, I really prefer the custom password and full data encryption idea. As the internet speed and cpu capacity increases, it's privacy that matters much more than encryption speed or internet traffic to people who do write diary on app!

enjoy-life

Local and Cloud encryption (should be the same, as most of the sync methods just mirror the folder to the cloud) is the only major feature that's missing for me from Diarium.

AI think @timo mentioned in another thread that local encryption is not needed, because if an attacker is already in your filesystem, it's too late. That's kinda true, but you could make the same argument about a password manager, yet it makes a lot of sense to have the password database stored encrypted.
And to be honest my diary is about as private and important as my passwords.

EAPTCB

There's some good news for iOS users backing up Diarium to iCloud - with the release of ios 16.3 there is now an option to enable end to end encryption for all icloud data. Here's some information.

username

While I somewhat admire Timo's optimistic view of how data is handled in the cloud by companies who are known to profit from analysing, profiling and allegedly anonymising data for study and sale, the point I would like to reiterate is that non of the optimism or conspiracy matters when considering core of this issue that demands this feature.

All that matters is that I have have full control over the data I enter to this journal via established, peer reviewed and widely adopted encryption options that give me full control over my personal information. This is the only thing that determines the value of the features. My control of the data.

At this point in time we live in a world where it is simply naïve, at best, to think that companies like Google, Apple and Microsoft do not have relationships with intelligence and government agencies that they are not legally allowed to disclose. Handing over your data to warrants that you are not allowed to be informed about. Legal mass surveillance on a scale never known. A world where journalists are under the most threat of danger in declining democratic nations for having the wrong opinion of governments and companies.

There is simply no room for debate on this issue of whether the right to privacy is a fair or valid argument and a simple question would answer the necessity for this feature regardless of what you may or may not think about how much these companies care about your data.

Is the end user in full control of the incredibly sensitive and personal data they are entering or are there clear pathways to access that information should the user's database be acquired/accessed?

It's fundamentally that simple. It's irrelevant what, who or when an entity might become the bad actor. All that matters is that they can be compelled to do so and it could harm the end user because of the lack a solid E2E encryption solution.

We should really stop talking like "they don't care about your data" because we care and that's the only point that matters.

You could also let us sync to this https://cryptomator.org as being able to sync via cryptomator would at least allow the option to control data on normal cloud services.

timo

Just a quick heads-up: I have not forgotten about this feature request.
With the recent updates and the next few upcoming updates I will work on making the sync faster and more stable before adding this additional layer of complexity to the syncing logic.
As I said before, if it's not an option for you to wait any longer for this feature request to be implemented, I recommend setting up a WebDAV server which is completely under your control and sync your data to that server.

EAPTCB

timo Just a quick heads-up: I have not forgotten about this feature request.
With the recent updates and the next few upcoming updates I will work on making the sync faster and more stable before adding this additional layer of complexity to the syncing logic.

Hi timo, I totally understand, and also appreciate that you have other apps to update too - but can you give us a rough estimate of time? I mean is it weeks, months or years?

alegoje

Strong end-to-end encryption with my custom passphrase is the only thing preventing me from adopting your app, which I liked best. It's a pity but I must pass for now.

kaykay

A workaround I found was to disable the cloud encryption, manually create backup files and manually encrypt them (I use Zip & Rar for this AppStore link.

One huge issue I found is that when you set a password to unlock the diary it informs you that if the password is lost then the diary can not be accessed unless it is reset - which is completely untrue. All I had to do was delete the app and install it again, and it pulled the backup from iCloud.

timo

kaykay One huge issue I found is that when you set a password to unlock the diary it informs you that if the password is lost then the diary can not be accessed unless it is reset - which is completely untrue. All I had to do was delete the app and install it again, and it pulled the backup from iCloud.

You are right - I would probably need to rephrase that to make it more clear. Longer text (which this would result in) however also means that less people read through it...

EAPTCB

kaykay A workaround I found was to disable the cloud encryption, manually create backup files and manually encrypt them (I use Zip & Rar for this AppStore link.

This workaround doesn't sync across devices though? if I were to add an entry on Windows how would I see it on my iPhone?

timo

I cannot promise anything, but let's put it this way: If I won't implement this until the end of the year, I would be disappointed by myself.

timo

I am finally able to share some first details about this new feature. I will try to explain the concept I have in mind and would appreciate your feedback on it.

The "E2EE" / end-to-end-encryption Cloud Sync will allow users to provide a custom password that is used to encrypt the files that are being uploaded to the Cloud when using the Cloud Sync
The "E2EE" mode will be optional, users will be able to continue to use the existing sync like before
The "E2EE" mode will operate on the same Cloud folder, but the files will have slightly different names, in order to prevent conflicts between them:
- The "v4_database" file (which is the main journal database) will be named "v4e_database" in the E2EE mode
- The "v4_Media" folder (containing all the entry attachments) will be named "v4e_Media" (note: in the iCloud sync, this folder does not exist as the iCloud Drive is not folder/file based. I will have to use a different way to keep the "normal" and "e2ee" media files separated)
- The "v4_mediaindex" file will be named "v4e_mediaindex". This file is not encrypted as it doesn't contain any secret information. The file contains a list of what files are located in the "v4_Media" folder. This is for optimizing performance, as querying the contents of a folder with potentially thousands of files through the Cloud APIs can be very slow
The "v4e_database" and all the media files in the "v4e_Media" folder will be encrypted using a 256 bit master key and the AES265-CBC algorithm. The AES algorithm also uses a 128 bit IV (initialization vector) which will be different for each file and the IV will be prepended to the encrypted payload data before uploading to the Cloud
The encryption key is automatically generated when first setting up the E2EE encryption Cloud Sync. The encryption key itself is encrypted (also using AES265-CBC) using an encryption key that is derived (using PBKDF2 key derivation) from a user-provided password. The encrypted encryption key is then appended to the "v4e_database" file before uploading, together with the salt for . The advantage of not using a key derived from the user password to encrypt all files is that the user could in the future change his password, without the app having to re-encrypt & re-upload all journal files. Only the "main" encryption key would then have to be re-encrypted using the new key derived from the new password. I don't plan to add such a "change password" feature for now, but it's good to have the option if I decide to do so.
So in the end, the "v4e_database" will have this structure:
- Bytes 1-32: main encryption key that has been encrypted using a different key which has been derived by a user-provided password
- Bytes 33-48: salt for PBKDF2 / IV for decrypting main encryption key
- Bytes 48-64: IV for decrypting the v4e_database file
- Bytes 65-end: Encrypted database file
The user will not have to enter the password every time the sync runs. The app keeps a copy of the encryption key locally in a safe place that is not accessible for other apps (e.g. on iOS/macOS this would be in the "Apple Keychain"). In case this key would be deleted (e.g. user manually deletes it in the Keychain Access app), the app would download the "v4e_database" file and prompt the user for the password -> app derives decryption key -> app decrypts and stores the main encryption key. The same thing would happen if a new device connects to the sync, which does not yet have the main encryption key stored locally. One thing I need to make sure here is to verify that the user in fact has entered the correct password - because decrypting data with the wrong key does not necessarily cause an error, but could simply output a wrong result (so not the correct "main" encryption key).

Now into the less technical aspects on how this feature surfaces to the user.
I plan that if the app is already using the "normal" sync, Diarium will keep using this mode, even when new devices are connected. The app will only offer the new "e2ee" feature if the user has not used the Cloud Sync before (= if the app has not detected a v4_database or v4e_database file in the selected cloud storage). So if existing users would want to use the E2EE sync, they would have to delete the "Diarium" folder from their Cloud Storage.

If there is no database file in the Cloud Storage, Diarium will show a selection between the "normal" sync and the E2EE sync (including a warning that warns users from potential data loss if they forget their password). The dialog should probably also link to a page here in the forums that explains the difference between the two syncing modes.

So if the user picks the "E2EE" syncing mode, the user is prompted to enter a password (probably with a "confirm password" field). The app then encrypts and prepends the encryption key like described above and uploads the first "v4e_database" file.

So much for now. I will try to implement this concept and probably come back with problems I haven't considered yet. What do you think? Do you see any flaws in this approach? Anything important that I missed? What could be improved (both technical and from a user-perspective)?

EAPTCB

timo I like the sound of this so far, you have done a good job with this idea and can't wait for when it's rolled out! I don't know how I missed this post! and can't believe I'm the first to reply since last month - there are so many of us wanting this feature I'm surprised at the lack of feedback for this post.

timo So if existing users would want to use the E2EE sync, they would have to delete the "Diarium" folder from their Cloud Storage.

Really? that's strange, I remember when I was using Day One I turned on E2EE once I upgraded and the app automatically converted my existing data into encrypted files.

timo The "v4e_database" and all the media files in the "v4e_Media" folder will be encrypted using a 256 bit master key and the AES265-CBC algorithm.

I have a question - what about futureproofing? Apple users are able to login to websites and create logins WITHOUT passwords. From what I understand a similar feature may become a standard in the future across other platforms and devices, In fact already all these websites and services support passwordless login. Could Diarium take advantage of passwordless E2EE by using device registration and biometrics in place of a password/code Would you need to prepare for this now or is it something which is difficult/irrelevant to E2EE?

Max

timo

Thank you very much for considering the feature, much appreciated. From my perspective a one time delete of the cloud storage content e.g. iCloud is acceptable, at least as long it is possible to migrate the original data and files with a manual export/import.

Looking forward to see the feature surfacing. 👍️

timo

leonix Just to clarify: "Default" means no encryption at all?

Yes.

leonix With enabled E2EE encryption: Would it be possible to change the encryption password later on?

I talked about this in the thread above

timo The encryption key is automatically generated when first setting up the E2EE encryption Cloud Sync. The encryption key itself is encrypted (also using AES265-CBC) using an encryption key that is derived (using PBKDF2 key derivation) from a user-provided password. The encrypted encryption key is then appended to the "v4e_database" file before uploading, together with the salt for . The advantage of not using a key derived from the user password to encrypt all files is that the user could in the future change his password, without the app having to re-encrypt & re-upload all journal files. Only the "main" encryption key would then have to be re-encrypted using the new key derived from the new password. I don't plan to add such a "change password" feature for now, but it's good to have the option if I decide to do so.

leonix I case you switch to the new V5 with individual passwords you will get many request "how can I get my diary back" and many frustrated users.

I already get these now, because users don't use the sync at all, even though the app shows warning dialogs every 30 days.

leonix I would recommend to set the option of a "single key for all users" as the default option (as in v4) - instead of no encyption at all.

Can you elaborate why you think this would be better?

timo

Thanks for your feedback.

EAPTCB Really? that's strange, I remember when I was using Day One I turned on E2EE once I upgraded and the app automatically converted my existing data into encrypted files.

I could technically also implement this, but it would cause a series of problems and additional difficulties. I could still add such an "auto-migration" in the future but for now I try to keep it simple.

EAPTCB Apple users are able to login to websites

I haven't read the article in its entirety but I think this feature would only work on Apple-devices, which makes it unsuitable for Diarium.

argentum2f

New prospective user here. I've been looking for a decent journal app with this feature, so I'll stay tuned. I actually have two priorities:

Data is, at least on principle, recoverable without the app (e.g. use a standard format, or at least a publicly described format that wouldn't be hard for someone to build simple decryption and conversion tool for). The general idea is I want assurance that I don't end up years down the line with unreadable files because the app got sold to a company that pulls it offline, or the dev loses interest, or dies or something and the app stops getting updated and is incompatible with any of my devices.
Data is private. This means that it's not possible for the app author, or cloud storage provider, to access my entries - at least to a commonly accepted degree of security.

The proposed solution looks good, for the 2nd part - but I worry about the first point. It's good to have an export feature - but what if I don't do that because I already have backups and think I'm secure, but then when I realize (years later) that I should've, I no longer can?

timo

argentum2f The proposed solution looks good, for the 2nd part - but I worry about the first point. It's good to have an export feature - but what if I don't do that because I already have backups and think I'm secure, but then when I realize (years later) that I should've, I no longer can?

What would be your suggestion? I plan to publicly post the implementation details of the encryption/decryption algorithms once they are final so that it would be possible to recreate the decryption in another program/script.

VojtaKolacek Data somewhat readable/exportable without the app - currently nope

Currently the sync uses a hard-coded key to encrypt all data. I am not planning to publish it once the other sync encryption has been implemented, but I might add an option to not do any encryption at all, if that is a popular demand from users. I think Diarium does already offer a good variety of export/backup options (the .diary file created using the backup is also just a SQLite database) and chances are slim that the app disappears from one day to another with only the Cloud sync data left, preventing the use of the export/backup features

VojtaKolacek

Hi, a new user here. Similar to @argentum2f, my main criteria for a journal app are

Custom cloud backup (WebDAV - check)
Data somewhat readable/exportable without the app - currently nope. Your proposed solution would work for me personally, as writing a script to extract v4e_database to a regular sqlite.db is trivial. Adding an option not to encrypt the cloud data works even better for me, as the WebDAV server is under my control anyway.

« Previous Page Next Page »