0
Planned

java issues...crashing on scanning large PDF's

Anthony Reinking 7 years ago updated 7 years ago 3

Converting these is not acceptable as I have over 500, and all of what I download is PDF format.

What can I do to avoid this crash? I am using the latest JAVA on Windows 2012R2, this was working fine up until one of the latest updates of Ubooquity?


Here is the error message, as soon as I start an 'admin' scan of my magazines.


20161218 10:32:12 [Scanner thread] ERROR com.ubooquity.Ubooquity - Uncaught exception on thread: Scanner thread
java.lang.OutOfMemoryError: Java heap space
   at java.awt.image.DataBufferByte.(Unknown Source) ~[na:1.8.0_111]
   at java.awt.image.ComponentSampleModel.createDataBuffer(Unknown Source) ~[na:1.8.0_111]
   at sun.awt.image.ByteInterleavedRaster.(Unknown Source) ~[na:1.8.0_111]
   at sun.awt.image.ByteInterleavedRaster.createCompatibleWritableRaster(Unknown Source) ~[na:1.8.0_111]
   at sun.awt.image.ByteInterleavedRaster.createCompatibleWritableRaster(Unknown Source) ~[na:1.8.0_111]
   at org.apache.pdfbox.filter.DCTFilter.fromBGRtoRGB(DCTFilter.java:245) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.filter.DCTFilter.decode(DCTFilter.java:170) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.cos.COSInputStream.create(COSInputStream.java:69) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.cos.COSStream.createInputStream(COSStream.java:163) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.pdmodel.common.PDStream.createInputStream(PDStream.java:235) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject.createInputStream(PDImageXObject.java:565) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.pdmodel.graphics.image.SampledImageReader.from8bit(SampledImageReader.java:233) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.pdmodel.graphics.image.SampledImageReader.getRGBImage(SampledImageReader.java:138) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject.getImage(PDImageXObject.java:340) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.rendering.PageDrawer.drawImage(PageDrawer.java:781) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.contentstream.operator.graphics.DrawObject.process(DrawObject.java:62) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:815) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:472) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:446) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.contentstream.PDFStreamEngine.processPage(PDFStreamEngine.java:149) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.rendering.PageDrawer.drawPage(PageDrawer.java:189) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.rendering.PDFRenderer.renderPage(PDFRenderer.java:208) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.rendering.PDFRenderer.renderImage(PDFRenderer.java:139) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at org.apache.pdfbox.rendering.PDFRenderer.renderImageWithDPI(PDFRenderer.java:94) ~[pdfbox-2.0.0.jar.1080217776487141892.tmp:2.0.0]
   at com.ubooquity.fileformat.pdf.b.a(SourceFile:71) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.fileformat.pdf.b.a(SourceFile:44) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.a.b(SourceFile:382) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.a.a(SourceFile:212) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.a.b(SourceFile:316) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.a.a(SourceFile:35) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.a$1.run(SourceFile:116) ~[Ubooquity.jar:1.10.1]

   at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_111]

0
Planned

Would love to simply change comics to magazines. Can this be done?

jmcloughlin 7 years ago updated by RockstarTom 2 years ago 2

Would love to be able to add a magazine repository ..


Just a suggestion ..

0
Answered

Synology Package

Rob Espinosa 7 years ago updated by Tom 7 years ago 2

Are there any plans to create a Synology Package? I see that it's possible to get Ubooquity running on a Synology but I just don't feel comfortable installing it manually.


I really love what you are doing with Ubooquity, I've longed for a Plex like solution to reading comics for a very long time and very much look forward to being able to use your product for both eBooks and Comics!

0
Answered

Syslog configuration

Elouan 7 years ago updated by Tom 7 years ago 1

Hi,


I'd like to integrate ubooquity logs with the built-in synology logs. This way I'd benefit from notifications (important to know that I have an error requiring restarting ubooquity) and a better log management (search and archiving)

I tried to understand how logs are working in ubooquity, but couldn't figure out how to do that


The simplest solution I could think of is to configure ubooquity for syslog protocol: is it somehting you can do? can I define an IP/port for ubooquity to send the logs? that woudl definitely be a nice improvement, and I'm pretty sure there are light open-source libraries for that

0
Planned

Java error during scaning

Elouan 7 years ago updated by Anthony Reinking 7 years ago 7

Every now and then, during scanning, generally when my NAS is otherwise busy, I get a "java heap space" exception. The error is displayed in the log like the one at the bottom of this post.

Once the error has happened, ubooquity hangs: it's not killed (it still shows in the process list), but it's not responsive either. That's an issue in my view to hae a runnung process that really is dead

When that happens all I have to do, is stop and then restart ubooquity and the scan can generally continue without problem (it was a temporary issue; I also suspect memory leaks in the code since this only happens when scanning a large number of comics; it only ever happened when the number of comics to scan was above a few 1000s)

To avoid this situation,I propose to add some improvement to the scanning process:

  • catch error when scanning a book; and dump a more relevant message to the user (for example, giving the name and the file and the item that was being scanned at the time of the error (cover image, title, metadata...)
  • when a memory error occurs, shut down ubooquity process; In my case, since I've added a "nohup" commands in the startup script, upstart will restart ubooquity and I would have a functioning server instead of a zombie process (I've also added to the "nohup" commands a limit to the number of restart so that if ubooquity crashes 3 times in a row, then upstart will not restart it anymore)
  • Optional: add the incriminated file to a "black list" so ubooquity doesn't try to rescan it next time... and of course allow the admin to reset the black list

ERROR com.ubooquity.Ubooquity - Uncaught exception on thread: Scanner thread
java.lang.OutOfMemoryError: Java heap space
   at java.awt.image.DataBufferByte.(DataBufferByte.java:92) ~[na:1.8.0_101]
   at java.awt.image.ComponentSampleModel.createDataBuffer(ComponentSampleModel.java:445) ~[na:1.8.0_101]
   at java.awt.image.Raster.createWritableRaster(Raster.java:941) ~[na:1.8.0_101]
   at javax.imageio.ImageTypeSpecifier.createBufferedImage(ImageTypeSpecifier.java:1074) ~[na:1.8.0_101]
   at javax.imageio.ImageReader.getDestination(ImageReader.java:2892) ~[na:1.8.0_101]
   at com.sun.imageio.plugins.jpeg.JPEGImageReader.readInternal(JPEGImageReader.java:1071) ~[na:1.8.0_101]
   at com.sun.imageio.plugins.jpeg.JPEGImageReader.read(JPEGImageReader.java:1039) ~[na:1.8.0_101]
   at com.twelvemonkeys.imageio.plugins.jpeg.JPEGImageReader.read(Unknown Source) ~[imageio-jpeg-3.1.0.jar.2469300273615590271.tmp:3.1.0]
   at javax.imageio.ImageIO.read(ImageIO.java:1448) ~[na:1.8.0_101]
   at javax.imageio.ImageIO.read(ImageIO.java:1352) ~[na:1.8.0_101]
   at com.ubooquity.fileformat.cbz.a.a(SourceFile:87) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.f.a.a(SourceFile:41) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.b.a(SourceFile:63) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.a.b(SourceFile:531) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.a.c(SourceFile:470) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.a.b(SourceFile:35) ~[Ubooquity.jar:1.10.1]
   at com.ubooquity.data.feeder.a$1.run(SourceFile:123) ~[Ubooquity.jar:1.10.1]
   at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]

0
Fixed

(admin page) Missing login expiration

Elouan 7 years ago updated by Tom 7 years ago 7

It seems that the admin page is not very secured: once the admin is logged in, the page .../admin stays open for an infinite amount of time until a user clicks on "log administrator out"

  • Even after a full night without activity, [and in fact, without even any connection to the page] in the morning, I can still access the admin page without beeing requested to put in the password
  • one the admin is logged in, there is no IP check. I've moved from home (local network) to office (external network), and ubooquity doesn't detect there has been a change and doesn't request to put in the password

This is worrisome because it means that once an admin is logged-in, anybody can access the admin page; and I mean anybody: it's not even limiting access to registered users, anybody trying that page will gain access to the admin page

This is a bit strange because it seems to me [although I haven't really tested that] that the main pages for consulting the library does not suffer from this security breach

I suggest to improve

  • add an "inactivity timer" in the admin page, set to 5 minutes by default [you may add this parameter in the admin settings];
  • detect the IP address of the client connection so that whenever it's a new IP, ubooquity requests a new login.
  • add something at the bottom of the page displaying the number of clients connected to the admin page => this could help detect intrusion
0
Under review

Broken pipe issue

Elouan 7 years ago updated by Tom 7 years ago 1

Here is an issue I've found that I don't understand

When I set the number of items per page to 20 and read the comics from internet, everything is going fine: browsing is fast, and I don't see any problems

But, when I set the number of items to 24, then every time I load a new page some icons takes a long time to load (more thant 0sec), and I see 4 warnings in the logs

20161114 14:07:50 [pool-1-thread-20]  WARN com.ubooquity.e.e - Interrupted while sending response (perhaps the client stopped the download) - java.net.SocketException - Broken pipe
20161114 14:07:50 [pool-1-thread-13]  WARN com.ubooquity.e.e - Interrupted while sending response (perhaps theclient stopped the download) - java.net.SocketException - Broken pipe
20161114 14:07:50 [pool-1-thread-15]  WARN com.ubooquity.e.e - Interrupted while sending response (perhaps the client stopped the download) - java.net.SocketException - Broken pipe
20161114 14:39:59 [pool-1-thread-15]  WARN com.ubooquity.e.e - Interrupted while sending response (perhaps the client stopped the download) - java.net.SocketException - Broken pipe

I don't understand this issue nor do I know where to search for a cause; I don't believe it's a performance issue on my Synology NAS: a difference of 4 items on a page doesn't seem enough to create a problem, and I have a fast connection to internet, so I don't believe the problem is there either...


If I set the number of items to 26, I see that 6 items take a very long time to load... but I dont see any warning in the logs

If I set the number of items to 21, I see that 1 item take a very long time to load... but I don't see any warning in the logs either


What could be the cause of this ? why is 20 a magic number ?

0
Not a bug

Not pulling favicon/touch icon images from custom theme

EaF 7 years ago updated 7 years ago 3

Hi Tom,


I'm using a custom theme (the dark theme made by Hidea). I've added all the necessary faviocn .png images to the custom theme folder (with the same filenames used by your default Ubooquity theme) but Ubooquity or the browser isn't using them, it just loads the default Ubooquity favicon images


I've tried on the following browsers:

Chrome desktop, mobile (ios & android)

Safari mobile

Microsoft Edge


Any ideas what's going on?

0
Answered

Cannot start Ubooquity from command line

Black Six 7 years ago updated 7 years ago 2

I'm running Ubooquity on a Mac mini. Generally it works fine but periodically it will crash requiring me to go and restart it. I'm trying to set up a cron job to check on it every so often and restart if need be but am running into a problem.


If I double-click Ubooquity.jar from the OS X GUI it launches fine. If I try to start it via Terminal though, I get this:


Computer:Ubooquity user$ java -jar Ubooquity.jar -webadmin
Exception in thread "main" java.lang.UnsupportedClassVersionError: com/ubooquity/Launcher : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)


Any ideas?

0
Planned

Feature: add Ratings to comics

Elouan 7 years ago updated 7 years ago 2

Since you plan to add a read status to comics, I believe this would be fairly easy to implement

I'm assuming you are going to have a table keeping track of the reading status of each comics for each user.

You could add in the same table, a column keeping track of comic's rating by user. This way we could have new benefits:

1: when opening the comics, provide ratings in the details window(and if the user has not put any rating, ubooquity could compute an average for all users)

2: folder (or any grouping), could display an average of the ratings for all comics within the folder/group below the image (again: display average for the connected user or the average over all users if no user ratings are available)

3: (in a future) a new sorting criteria for comics