0
Under review

Ubooquity stops periodically on FreeBSD

Tom Wilkinson 9 years ago updated 7 years ago 34
I am running Ubooquity on FreeNAS 9.2. It is inside a jail and if I start the script Ubooquity will eventually stop being visible on the usual port (I'm not sure if Ubooquity itself is stopping or just the GUI goes).

Currently I have a fix for this that is less than ideal. If there is a constant connection (e.g. leave Chrome pointing to the page at home) I can access Ubooquity indefinitely. As soon as this connection is broken however this issue returns (normally overnight is enough to cause it to go down).
This happen to me the other day as well, canyouseeme.org said the port was opened and server answered, but I couldn't use the URL locally or over the internet. I'm using Windows 8.1. Restarted the server and it worked again,it was unusual.
Under review
Unless Chrome has some sort of auto-refresh mechanism, keeping a tab open on Ubooquity won't do anything as the communication only happens when you are downloading something (the page, a file, an image...).
Once the page is displayed, the server (Ubooquity) does not know anything about the state of your browser.

I'd be interested in seeing the log files of Ubooquity for the period containing the shutdown (as a lot af things are logged when Ubooquity starts and stops).
You can send them to: tom 'at' vaemendis 'dot' net

I know that this is pretty old, but I have a very similar issue. Here is the error I can find in the logs:


20161024 08:30:29 [main] ERROR com.ubooquity.Ubooquity - Exiting application because of exception
java.io.IOException: Bad file descriptor
   at java.io.FileInputStream.available(Native Method) ~[na:1.8.0_102]
   at java.io.BufferedInputStream.available(BufferedInputStream.java:410) ~[na:1.8.0_102]
   at com.ubooquity.Ubooquity.main(SourceFile:283) ~[Ubooquity.jar:1.10.1]
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_102]
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_102]
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_102]
   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
   at com.ubooquity.e.c.a(SourceFile:823) [Ubooquity.jar:1.10.1]
   at com.ubooquity.Launcher.main(SourceFile:10) [Ubooquity.jar:1.10.1]



I'm sure this is of my own doing but I'm not sure where to take it from here.

+1

Thanks for the log.

That's an error I can (and will) investigate.

I realize that it's been a while. I would like to add that I have found a way to prevent this error.


To explain, I am not taking credit for this just telling you how it happened. A friend and co-worker of mine liked my idea and began work on this. He was able to pull a string off of the main files here and tweaked it in some way and made it work. In a addition to this there was an issue with installing the jdk and doing the necessary mounts. We found that it needs to be done outside the jail and mounted within it.


Here is the string:

nohup java -Xmx512M -Xms512M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=2 -XX:+AggressiveOpts -jar Ubooquity.jar -port 2202 -webadmin 2 > 1 &


The catch to the string is that it needs to be one line. The reliable way to run it is to save it in a SH file as one line then run the file.


Here is the example of preforming the mount:

What the install tells you to do (and most guides you will find):

mount -t fdescfs fdesc /dev/fd

What you need to do from outside the jail:

mount -t fdescfs fdesc /mnt/Test/jails/Ubooquity/dev/fd


P.S. Also found a way to make it auto start as a service within the jail, but that's not what this post is about. I'll post if asked. That took me some research and reverse engineering (changing some words really) of someone else's start up for another project I was doing.

I seem to have a similar problem. Running 1.7.6 on a pi b with raspmc, Ubooquity stops after a few hours of scanning my library of about 30 gigs of comics. The log doesn't seem to say much, but I'll mail it.
Just in case it might help other people: Bo has solved the problem by switching from Raspbmc to Raspbian.

I've been having this issue as well. First on ubuntu server, then windows 10. At first I assumed that it was a server issue and reinstalled my operating system multiple times before switching it over to my windows 10 system and the problem persisted. I then thought it could be an issue with the size of my comics library since I have over 100,000 comics and it's size is hovering just over 2TB. Since I switched my library over to windows 10 I tested by not adding all of my books but the problem was still there. Now I'm not so certain. I constantly have to restart the server to get ubooquity to respond again, and even then it doesn't always work. It's rather frustrating that I cannot find the cause. If you'd like to take a look at my current logs you can grab them here.

just a thought: does the problem occur during the scanning process? looking at some of your logs, it seems you're running out of heap space during scanning. This would be the root problem forcing you to restart Ubooquity: some books in your library are causing problem

No, it does not happen during scans I have automatic scanning turned off. It hangs after a while just sitting idle.

I only had a quick glance at your logs, but, like Elouan, I think your problems are (at least partially) coming from the memory shortage.

Once a memory error has happened, all bets are off. The application can crash right after the error or much later, or even continue running like nothing happened.

This kind of error is critical, you can't expect Ubooquity to run properly as long as they happen.


To prevent them, you can increase the allocated memory using a command line option:

http://vaemendis.net/ubooquity/doc/faq.html#i-have-java-lang-outofmemoryerror-java-heap-space-messages-in-the-logs


On my side I'll have to do some profiling to look for potential memory leak (or file handles leak), but I don't know when I'll find the time to do it.


Just for my information, how many comics were listed in the database when it crashed (I don't need exact numbers, just an idea) ? And what was the size of the database file ?

I usually increased the memory allocation to 4gb, but it still seems to happen. It did on both of my ubuntu servers with 4gb allocated as well as my windows 10 machine (which I did forget to set the memory on). The server never seems to hang during the scans. They always complete fine. As for the number of comics, I have 110,597 comics. And my database file is 180mb.

I have allocated 8Gb ram to java using -Xmx8000m and it still runs out of heap space. There has got to be a memory leak somewhere.

Agreed, 4 GB was already an insanely high amount of RAM for what Ubooquity does.

Like I said, I have to do some profiling, but it will take time.


One last thing to check on your side: does the first occurrence of the memory error happen on the same file each time ?

I'd like to rule out a problem caused by a specific file (already happened in the past with broken PDF files).

I dont believe so. I have disabled startup and automatic scanning and it still seems to happen. I've restarted it several times so no scans have taken place. I have however noticed better performance using the 64 bit version of Java, but some of my users run into trouble using 32 bit browsers.

One interesting thing I have noticed is that when I launch the server with no automatic scanning, it only seems to use 160mb and climbs to about 4-500mb before holding steady. The moment a scan is ran it jumps to 1.5 Gb and even after the scan completes the memory usage never decreases. This is on 64 bit java btw. I have not tested that on 32 bit. When I do, I'll clear my log file so as to get clean info.

So I'm having a similar problem on a Windows Server 2012 machine with 64-bit java 1.8.0_71. On initial startup, ubooquity idles at 145.5MB ram usage. I currently have my scan interval set to 15 minutes as I've been updating my database frequently and have it running headless. Watching the ram usage, if I manually launch a new library scan the usage jumps and then settles higher than it was before the scan. The difference doesn't seem to have any kind of consistency to it. I saw values between 600K and 2MB. That being said, my end result is the same where the page just stops responding.


Looking at my ram usage for ubooquity the last time it stopped responding had it at 800MB used. I can understand the potential for high ram usage with Tom as he has 2TB of comics, but I only have a little less than 1GB in 112 comics right now, so I feel like that wouldn't be the issue. Also, my logs say everything is running just fine. I also have the same temp fix where if I leave a connection to ubooqity open it has no issues staying connected. But if I disconnect, one night is enough to get it to not respond again.


I don't have 2 TB of comics, only around 50-100 GB. :)


So you did not see any heap errors (or any other kind of memory related error) when Ubooquity becomes unresponsive ?

And when it does, is the java process still alive or did Ubooquity completely stopped working ?


Monitoring memory of Java application without specific tools is not easy as Java will use as much memory as possible inside the limits you gave it (the principle being that there is no point in spending cycles freeing memory if you are not close to the limit).

If you define a 1 GB limit, even the smallest of Java programs might consume a few GB while it could easily live with a 200 MB limit.


So you won't be able to guess much by just looking at memory numbers in your task manager unfortunately.

(and if you had no heap errors, the problem is probably not related to memory at all)




Ah, in my case I was talking about both your and Justin's problems and got you guys mixed up. And no, I didn't see any heap or other errors unless they would have been sent to err on a cmd window rather than the log file, which I can check if that's the case. And interesting regarding the memory issues. I'm used to programming in c/c++/python where memory usage is mostly to entirely consistent and/or predictable. And ubooquity on the server side is still responsive when this happens. I can change settings, scan for new comics, and close the program. More interestingly however, hitting the save and restart button doesn't throw any errors in the log file, and the log shows that its restarted, but its still not accessible. Only if I close the entire running instance and restart it will I get access to it again.

Interesting.
I guess you are using the GUI (not the web admin page).
So the web server part would be broken whereas the rest of the application is still running...
I'll double check that all critical threads are covered by the exception catching mechanism. But it will be a few weeks/months before I do it, I want to finish and release the online epub reader first.


As for the memory management mechanism, the JVM memory management is consistent, it's just a matter of default strategy. Java's default strategy is to free memory only when you come close to the allowed limit (this is not entirely true for all GC strategies, but still...). This makes sense as freeing memory more frequently would consume resources (and induce latency or add micro freezes) without providing much benefit (as long as you consider memory as a non critical resource).


In any case, a small application like Ubooquity should not require any specific memory configuration, nor should it require more than a few hundred megabytes of memory. I have to investigate and create some test case with massive numbers of files.

If you have any suggestions on monitoring tools, I'd be willing to lend a hand with testing. I have a massive collection (2.06 TB), and multiple platforms I can test on. I'm primarily a Linux user, but I do have OSX and Windows machines.

I plan to use Java Mission Control.

But don't bother yet. Once I have done my tests, I'll come back to this thread with what I found (or didn't find). Then I might be able to tell you what I am looking for (or I'll have found and fixed bugs).

Excellent. Ill keep my eyes open for a reply. :)

Ah yes, that would be important to know. I've been using the GUI as I remote into my server frequently working on projects, and it's easy enough to open when it's right there. And sounds good regarding that taking a while as epub would be nice. And consistent isn't the idea I was meaning. I probably shouldn't be writing posts in bed at 1am. XD My point was I'm used to languages with aggressive garbage collection and/or I take care of it aggressively myself. Makes sense though now. I've made a point of avoiding learning java, so am fairly ignorant when it comes to things like that with it.


I might try moving ubooquity over to my linux VM for now and see if that helps things if it keeps not responding.

Personally, I've noticed better performance on the Linux side of things. As long as it's the 64 bit oracle version and not openjdk.

I'll definitely give that a shot then.

Well, I don't know much about Python myself. :)

You dont need to know much about python to use Linux. ;)

Every time I close SSH I get the following:


20170213 23:01:00 [main] ERROR com.ubooquity.Ubooquity - Exiting application because of exception
java.io.IOException: Bad file descriptor
   at java.io.FileInputStream.available(Native Method) ~[na:1.8.0_121]
   at java.io.BufferedInputStream.available(BufferedInputStream.java:410) ~[na:1.8.0_121]
   at com.ubooquity.Ubooquity.main(SourceFile:283) ~[Ubooquity.jar:1.10.1]
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
   at com.ubooquity.e.c.a(SourceFile:823) [Ubooquity.jar:1.10.1]
   at com.ubooquity.Launcher.main(SourceFile:10) [Ubooquity.jar:1.10.1]

Can you share your startup script?

Just to clarify, are you asking the OP or me? I ask because I did mention an auto start script.

OP here. I stopped using Ubooquity after this kept happening and no longer have a copy of my startup script.

Do you use something other than Ubooquity?

Not currently. I haven't had as much time for reading recently and so it fell by the wayside. I do intend to try and install the latest version of Ubooquity soon.