Your comments
I plan to use Java Mission Control.
But don't bother yet. Once I have done my tests, I'll come back to this thread with what I found (or didn't find). Then I might be able to tell you what I am looking for (or I'll have found and fixed bugs).
I think you should start by trying to run Ubooquity manually and see how it goes.
Just put the jar file in a dedicated (writable) directory and run:
java -jar Ubooquity.jar -headless
Once you have it running and you have successfully accessed a few shared files, then you can try setup something more permanent.
Interesting.
I guess you are using the GUI (not the web admin page).
So the web server part would be broken whereas the rest of the application is still running...
I'll double check that all critical threads are covered by the exception catching mechanism. But it will be a few weeks/months before I do it, I want to finish and release the online epub reader first.
As for the memory management mechanism, the JVM memory management is consistent, it's just a matter of default strategy. Java's default strategy is to free memory only when you come close to the allowed limit (this is not entirely true for all GC strategies, but still...). This makes sense as freeing memory more frequently would consume resources (and induce latency or add micro freezes) without providing much benefit (as long as you consider memory as a non critical resource).
In any case, a small application like Ubooquity should not require any specific memory configuration, nor should it require more than a few hundred megabytes of memory. I have to investigate and create some test case with massive numbers of files.
I don't have 2 TB of comics, only around 50-100 GB. :)
So you did not see any heap errors (or any other kind of memory related error) when Ubooquity becomes unresponsive ?
And when it does, is the java process still alive or did Ubooquity completely stopped working ?
Monitoring memory of Java application without specific tools is not easy as Java will use as much memory as possible inside the limits you gave it (the principle being that there is no point in spending cycles freeing memory if you are not close to the limit).
If you define a 1 GB limit, even the smallest of Java programs might consume a few GB while it could easily live with a 200 MB limit.
So you won't be able to guess much by just looking at memory numbers in your task manager unfortunately.
(and if you had no heap errors, the problem is probably not related to memory at all)
The log file grows up to 1MB, then it is zipped and a new log file is created.
5 zip files are kept, when a 6th one is created, the oldest one is replaced.
This behavior can be overridden by your own configuration.
This tutorial explains how to use an external configuration file.
The settings you'll want to modify are "maxIndex" (max number of kept zip files) in the "rollingPolicy" section and "maxFileSize" (max size of a log files before it is zipped) in the "triggeringPolicy" section.
The full documentation of the logger configuration is available here, but it's not very user friendly. Ask me if you need more info.
As for the logging of read comics, I'll add it.
Agreed, 4 GB was already an insanely high amount of RAM for what Ubooquity does.
Like I said, I have to do some profiling, but it will take time.
One last thing to check on your side: does the first occurrence of the memory error happen on the same file each time ?
I'd like to rule out a problem caused by a specific file (already happened in the past with broken PDF files).
Very, very nice !
I was interested in using Let's Encrypt with Ubooquity but was kind of put off by the relative complexity of the procedure.
This will help a lot, thanks !
If you don't mind, I might include this tutorial in the documentation sometime.
I only had a quick glance at your logs, but, like Elouan, I think your problems are (at least partially) coming from the memory shortage.
Once a memory error has happened, all bets are off. The application can crash right after the error or much later, or even continue running like nothing happened.
This kind of error is critical, you can't expect Ubooquity to run properly as long as they happen.
To prevent them, you can increase the allocated memory using a command line option:
On my side I'll have to do some profiling to look for potential memory leak (or file handles leak), but I don't know when I'll find the time to do it.
Just for my information, how many comics were listed in the database when it crashed (I don't need exact numbers, just an idea) ? And what was the size of the database file ?
Customer support service by UserEcho
Well, I don't know much about Python myself. :)