Too many files open creates a crash

Thanks for the precise characterization of the problem, it helps a lot to know exactly where to look.

I'll do my best to fix this in the next release.

Guess I'll have to investigate on Linux though, can't reproduce the problem on Windows...

Thanks a lot for looking at this problem. I'm pretty sure you're not closing the files when we navigate from page to page.

I've found an other way (in linux) to investigate exactly the problem:

• find pid of ubboquity
• look in the folder /proc/PID/fd

this folder contains all the file descriptors used by the pid; it's the best way to investigate the problem

I've just encountered the bug again, so I counted the files in that folder, (command line = ls -l fd | wc -l) , it tells me 1023 ! remenber: I'm only allowed 1024 in linux; so I'm pretty sure that's what causing the bug

• After restart : ls -l fd | wc -l) = 48
• After login : ls -l fd | wc -l) = 75
• Now when I load new pages, here what I get

admin@DiskStation:/proc/9202$ sudo ls -l fd | wc -l
99
admin@DiskStation:/proc/9202$ sudo ls -l fd | wc -l
119
admin@DiskStation:/proc/9202$ sudo ls -l fd | wc -l
139
admin@DiskStation:/proc/9202$ sudo ls -l fd | wc -l
159

So, It's pretty clear that I load 20 new fd for each page (corresponding to the 20 icons that I've defined in the admin settings)... until I bump into the 1024 limit

Agreed, the problem is pretty clear.

Now I have to find the exact point of the leak, which should not be too complex, as soon as I get the time to do it.

In the meantime, if you have root access, you should be able to raise the max number of opened file handler (far from ideal, I know).

Hello,

I wanted to get some news about this issue; do you think you will correct it for the next release ? I'm asking because when I open the server to a new user, he starts navigating a lot to see what's there and this keeps happening... that's quite annoying.

I've tried increasing the max number but that doesn't help: it just delays the crash a little, but the crash happens anyway

if the cause is unknown, I propose a workaround that would allow me to keep the server running: catch that exception, and end ubooquity process with an error code (I believe exit(9) is the correct exit code in that case?) so the OS can detect the crash and launch a procedure to restart ubooquity from scratch (equivalent to kill 'ubooquity' followed by start ubooquity.

I didn't look into it yet, although I know exactly how to reproduce it, thanks to your tests.

It's on the list, perhaps not for 2.0 but definitely for 2.1 (see here).

Hi,

I've downloaded the latest release and installed it on my system. Globally, it's working as before: I like the new look and feel of the admin interface, scanning of my (rather large) collection went without a single issue; so I have to give a thumb up for the new release

on the down side, I didn't see much improvment on this particular subject (nor on other features I was hopping to see in release 2.0 -like a way for users to modify their password- why would you not open-source ubooquity? you've built a cool piece of software, it could gain momentum with 3rd party apps and reader coming in, but 1 year between 2 releases, it's way too slow: you should let us contribute)

Anyway, back to the topic: I've tested today, and the file leak is still there, everthing goes smooth but once I reach 1024 file handles, it breaks down

I still can't reproduce the problem on Windows (working with Ubooquity 2.0.2), so I tried on my Raspbery Pi (on Raspbian).

I followed your protocol, and the number of opened file descriptor stays quite stable, around 70, even when browsing a lot of pages with hundreds of covers.

So I don't understand why you still have the problem unfortunately.

Dear Tom

The memory leak problem is still there on Linux Synology NAS DSM 6.1 with Ubooquity 1.10.1

Consequently the server crashes although the process is not killed but the server remains unusable

Therefore I wonder if the leak is really Windows specific.

Maybe a temporary approach could be within the ubooquity code to force the java [Unix/Windows] FileSystemProvider to close all files that were parsed when loading a library page each time a new library page is loaded ?

Could you please workaround the leak and release a fix in the version 1.10.1 so that we have a fully stabilized version in the meantime you release 2.1 ?

...20170501 21:03:42 [pool-1-thread-6] ERROR com.ubooquity.f.d - Could not list files in folder:
java.nio.file.FileSystemException:

/volume1/distr/Books/Comics/Library/FR ± Soleil/Arleston [S] # Opale
(02-001-2013) [Codex] ¤ (FAN,HER,TEE): Too many open files

at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[na:1.8.0_102]
   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:1.8.0_102]
   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[na:1.8.0_102]
   at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427) ~[na:1.8.0_102]
... etc

Thank you very much in advance

All the best

Regards

...

hi Tom,

how many pages do you have on your library? maybe there is only 2 different pages to load? my settings are: 24 comics per page, and i have more than 50 pages

also can you try something? on your raspberry, can you reduce the number of file allowed to something below 1024, and then browse your library for many pages.... 

Hi Tom

i got the same problem. again today.;.. it had been a long time without issue, probably because I upgraded to a more recent synology model, but the issue is not solved

this is the message i got in the log:

Error while reading log file: java.io.FileNotFoundException - /volume1/homes/Ubooquity/Ubooquity/logs/ubooquity.log (Too many open files)

to reproduce the problem, maybe you can install dsl on your pc using https://xpenology.com/

also, i re-iterate my wish to participate in the development of ubooquity: it’s a very nice piece of softawre but updates are too scarce and we could make this even better