Entry deletion limit and forced database merge

Antyrox

Recently i was cleaning up my diary. I sat down and combed through years of my diary and deleted a whole bunch of unnecessary notes and attachments. After multiple hours of work i deleted a significant amount of junk records and cleaned up my diary. Afterwards i decided to sync my cleaned up diary to my phone. When i synced my local diary from PC to the cloud, everything was fine. However, when i tried to sync my diary from the cloud to the phone i suddenly got the error message stating that the cloud has fewer entries than local database. The error stated that because the cloud has 5+ fewer entries then local the databases will be merged. Afterwards, instead of deleting all the junk entries that i spent hours cleaning, the phone just re-uploaded all of the local deleted entries to the cloud.

I simply cannot overstate how incredibly frustrated and annoyed i am after so much wasted effort. I wasted hours of cleaning the diary for nothing and now i have to redo all the work again from scratch. Except now i have to sync my PC and my phone like every 5 minutes or so to make sure i did not deleted more then 5 entries at once.

I understand that sync was designed to be this extremely paranoid to prevent the data loss, but why the hell wasn't I warned about it? The Diarium knows the last sync date and it knows how many entries were deleted since last synchronization. If only up to 5 entries are allowed to be deleted per sync, then why in the world is there no warning when i am deleting the 6th entry or more? Just a simple warning message could have saved me hours of work and frustration!

The current limit of 5 deleted entries per sync and mandatory database merge is ridiculous. I have only 2 devices that i need to sync and it is already a massive problem for me. I cannot even imagine how it is for people who have like 4+ different devices that they need to sync every time they deleted 5 entries. The current paranoid approach is fine for normal day to day use when entries are mostly just added. But there has to be a solution for cases like mine when a bunch of junk entries need to be deleted at once.

And also why wasn't i even asked by the Android app if i wanted to merge local and remote databases or not? Instead of just deciding to merge, the app could have told me that the remote database has X fewer entries and asked me if i wanted to delete those local entries, or if i wanted to merge. To make sure that the user will not press something without reading, just lock the choice buttons temporarily and give a 30 seconds countdown to force the user to actually read the message. And if user does actually decide to remove the local entries instead of merge, just force the user to create a backup of the local database just in case.

Diarium version: 3.1.42
OS: Windows and Android

timo

Sorry about this and the time loss you experienced. A few years back there was such a warning prompt during the sync that allowed the user to choose between choosing local or remote database or merge if there's a sync conflict - unfortunately it caused a lot of confusion and after changing it to "merge always" massively reduced the support requests I received. Also, I think it's rather uncommon to delete extreme amounts of entries at once - I personally never deleted more than one at the same time. But I understand that usage patterns are different.
I also try to avoid any user interaction during sync because in some situations it's just not possible to show the prompt, e.g. when the sync happens in the background, with no UI visible.
I will think again in more depth about this and how the logic can be improved.

In the meantime, when you do delete a massive amount of entries and dont want to sync every 5 entries, you can delete all the entries first, then create a database backup and then import that backup on every device involved instead of using the sync

Antyrox

timo

If a user prompt during the synchronization is impossible then i would like to propose an alternative solution.

The biggest problem right now is the fact that there is no way to find which entries were restored during the sync. I think that a simple tag can alleviate this issue significantly. During the synchronization the Diarium can tag all of the restored entries with a special tag. Something like: "Diarium - Merge restored", or something similar. That way the user at least doesn't have to waste any time searching which deleted entries were restored. That way the user can simply search the entries by this tag and manually delete them again. If the user does not need this tag after the merge then he can simply delete the tag. This is a pretty simple and unintrusive solution in my opinion.

Also, i still think that some sort of a warning is required for the entry deletion. If the user is not supposed to delete more than 5 entries per sync, then tell it to the user explicitly. Just some sort of a message window when he tries to delete the 6-th entry.

roblen

Antyrox

timo

Perhaps the limit could be extended again. I have a feature request that might be relevant here. I think a recycle bin would be very useful. It's easy to accidentally delete the wrong entry. Or you might realise shortly afterwards that you still like the entry after all.
This way, entries could first end up in the recycle bin, which you can empty at any time (or the respective entry is automatically deleted after X days). And you can check again to make sure you deleted the right entry.

Antyrox

roblen
Yeah, i also thought about a recycle bin. This is definitely a feature that the Diarium desperately needs. While i was cleaning up my diary there were multiple times when i accidentally deleted a wrong entry and had to do awkward gymnastics to restore it from my other device. So yes, i agree that a recycle bin is both necessary on its own and also can be a possible solution to the sync problem. I thought about a recycle bin as well, but i proposed a tag solution instead simply due to the simplicity. I just thought that the simplicity of such a solution would make it much more likely that at least something is done about this problem.

timo

With the next version I will make some adjustments in an attempt to resolve and/or better understand this problem:

I will raise the "5 entries" threshold to 10 entries that need to be missing before a merge occurs
There no longer will be a merge when the minimum and maximum entry date of both local and downloaded database are equal
There no longer will be a merge if less than 10% of the entries in local database are missing compared to the remote database AND there is an entry in both databases that is around "today" (max 2 days in the past or future)
When the sync/merge happens in the background, the app will show a system notification to increase likelyhood that the user is notified about this merge and understand what is going on
I will add some logging for me to better understand the characteristics of those large-data deletions (e.g. what percentage of the local database is missing in the remote/downloaded database)

What do you think? Would that resolve this issue @Antyrox?

Antyrox

timo

Thank you very much for addressing this issue. I really appreciate those changes and I think that for the most part they actually do solve my problem. However, I have a little bit of mixed feelings about such a solution.

First of all, I really like that there will be no forced merge if the minimum and maximum entry date are the same. It seems like a good metric to decide if the merge is needed or not. With that rule alone I should be able to reliably purge the diary as long as I keep the min and max entry dates consistent. It's great that now I have an actual option for mass cleaning the diary.

I think that the above changes do solve my problem, but i dont like the fact that the solution is implemented through obscure automatic synchronisation rules. As long as I am aware of those rules, it should be fine, but I still have some reservations about Diarium deciding whether the merge on sync is needed or not. There is still a very real possibility for the Diarium to automatically screw up hours of work.

Maybe a new option could be added to the Diarium? A new option can be called "Synchronization policy" or something similar. Such an option could have two optional modes: “automatic” or “paranoid”.

When the Diarium encounters too many deleted entries in the "automatic" mode it will automatically decide in the background if the merge is needed or not. To decide if merge is needed it will use the rules that you have described above.

In the "paranoid" mode, if too many entries are missing, the Diarium could simply wait for the user input and ask the user if the database should be merged or not.

Previously you mentioned that you try to avoid any user interaction during sync because it is impossible when the sync happens in the background. With this approach the background sync without any user interactions is still possible. Sync policy could be set to "automatic" by default for all users and the Diarium will sync with no interactions in the background. However, the optional "paranoid" mode can give the user a choice to disable automatic background merge in exchange for more control.

I think that an option like this could be great. I could keep my sync policy "automatic" during my regular day to day use, but switch it to "paranoid" when i do a diary clean up. That way I can be 100% sure that there definitely will not be any automatic merge on sync.

timo

Thanks for the time to write your feedback an your suggestion.

I wrote above why I see waiting for user input or a dedicated setting as problematic. I am sure it would cause more confusion and probably increase support load because of it. I just read a review today in which a user highlighted the simplicity of the app - if I would add settings for niche things like that, the app would be unusable in no time and significantly increase maintenance cost on my side..

If you are regularly putting in hours of work to delete entries, I am not sure if Diairum is truly the right app for you. In the end it's a journal app and everything is built around that core functionality. Personally, in over 9 years of journaling with Diarium I never deleted more than one entry at a time, but I understand that usage patterns might differ. Also, you can always make a backup copy of your database that you can re-import at any time if you are concerned about a merge happeing.

I will carefully monitor the new logging and see for how many people such a merge is happening and in how many cases my new rules will prevent the merge. I will also carefully listen to user reports of either lost data or uninteded merges.

Antyrox

timo

Well, the solution with the automatic merge is not the solution that I hoped for, but it is still a solution. I still have some reservations about automatic merge due to my prior experience and my prior loss of data, but I do understand that my suggested option goes against the core philosophy of simplicity and I won't try to argue any further. Now that I am acutely aware that mass entry deletion can cause serious problems I definitely can take proactive steps to mitigate this issue with backups. And the new, soon to be released, merging rules will definitely help me a lot. Thank you for addressing this issue. The merging rules that you have described is an acceptable solution to my problem. My ticket can be closed as completed.