OPDS via Chunky Comic Reader is slow

Do you have a lot of comics in your library? Chunky has to fetch them in small batches which slows things down if there are a lot - you might improve matters if you organise them into folders and browse by folder.

I do have a lot of comics, but they are all organized into folders. I am browsing by folder, but that's not the issue--it's fairly snappy. The issue is with loading the actual images in a comic--that's what takes ~20 seconds to load per page/image.

Aaaaah gotcha.
I'm out of my own area of expertise, but out of interest, are the comics CBZs or CBRs?

I have a mixture of both, but probably more CBRs than CBZs.

On a related note, it would be great if there were a way for Chunky to pre-cache a few pages. It doesn't appear to be doing this.

It actually does try to cache up to 7 pages ahead and 3 behind your current page. But if each page is taking 20 seconds to load, it's not gonna get very far with that...

For the speed issue, you should be aware that a badly-compressed CBR can be death for performance - if 'solid' mode is enabled, the whole thing is compressed as a single blob, which means that if you want to get page 100, you first have to decompress pages 1-99 to get to it. You can normally spot this when reading gets slower and slower the further you get through the comic. Chunky deals with these by unzipping and rezipping them when you import them, which is why some comics take longer to import than others.

Dunno if that's the issue though.

Thanks for the info. I'll test some of both and see if there's a difference.

So I found some weird behavior after testing this with both CBZ and CBR. First of all, Chunky won't stream most of my CBZs at all. Meaning that when browsing comics in my folders, if I tap on a CBZ comic, Chunky does nothing. It just remains on the browsing window.

I then tried tapping the download icon, and it acted like it was going to download but then stopped. I updated the permissions on the files in this particular folder to 777, and then it was able to download the comics via the download icon. However, tapping the comic name (to stream) still does nothing. The odd thing is, all the CBRs in that directory were still able to stream and download with the original, more restrictive permissions.

I then managed to find some CBZs in a different folder that actually would both stream and download from Chunky, and even stranger, they had the same restrictive permissions that the others did that wouldn't work. I checked the actual folder permissions for both and they were both 777. I haven't been able to find a single CBR yet that won't stream in this manner, so the issue seems to associated with CBZs.

Finally, to my original issue, I couldn't tell any difference in streaming performance between the CBZs and CBRs. They are both very slow to load.

I'm a bit late to the party, but I'll try to provide some info.

First, regarding the comics streaming performance, the easiest way to know if the problem comes from Ubooquity is to try to read a comic directly in a browser instead of using Chunky, as the mechanism that provides the images is the same (except for preloading, Chunky preloads more pages). So if the online reading is slow, Ubooquity's the problem.

That being said, I'm pretty sure the performance problem comes from Ubooquity. The way image extraction is currently done is pretty naive. The archive is reopened for every image.
Also I don't know how the Java libraries I use for extraction perform against native tools, but I suppose they might be slower (I'll have to do some tests).

I have thought about two main solutions for now:

• Full comic extraction when a page is opened. The whole archive would be extracted in one pass. There are a lot of side effects to take into account though (mostly cache management and concurrent access) and the performance gain is not guaranteed. I need to test that too.
• Use native tools. I could have Ubooquity call native "unzip" and "unrar" commands (at least on Linux). That way I would be assured to make the best of the available hardware. Could be used in conjunction with the first solution.

Let me know if you have other ideas.

As for the file permissions, Ubooquity only need read access to the comics files. The only folder where it needs writing permission is the one it is run from, to be able to write the database, the thumbnails etc. I don't know how writing access on comics file could have an impact on anything Ubooquity does. Could you send me your log file (in the "logs" directory), if there are errors inside, it could give me alead.
My address: tom "at" vaemendis.net

Hey Tom,

I did some further testing between the web browser (mobile Safari) and Chunky, and the results were somewhat inconsistent, so I'm not sure how helpful they are. When I tested yesterday, basically what I found is loading in the browser, while not being incredibly fast, was still roughly twice as fast as Chunky. I used both CBRs and CBZs, and tried both smaller and larger files. On the whole, CBZs seemed to load slightly faster that CBRs. Files with smaller images loaded quite a bit faster (obviously).

My comics that have what I would say medium-sized images took about ~4 seconds to load in the browser, and ~7-8 seconds to load in Chunky. Small image files took about ~2 seconds in browser, ~4 in Chunky. Larger images took around ~10-12 seconds in browser, and anywhere from 20-30 seconds in Chunky. Those ratios were mostly consistent when I was testing the first time, but I decided to try again this morning with one of the same medium-sized comics, and now when I tested the results were close to the same for Chunky as before, however now, the browser speeds were slower, more closely matching Chunky's speed, which aligns more with what you said.

I think extracting the archive completely at the beginning is preferable to extracting each page individually. Really the only downside is it will take a little more time initially, but the benefit of having each image loading faster is worth it. I liken it to streaming movies--they take longer initially to buffer, but I would much rather have that longer wait at the beginning than watch a choppy movie that pauses every few seconds to re-buffer. I think that should be the goal of any streaming service, is to give the illusion that whatever media is being streamed is running locally. Of course the streaming devices can't control sub-par network conditions, but if conditions are good, stutters and pauses should never be caused by the streaming device itself. I do like how Chunky attempts to "buffer" images to try and provide a more seamless experience, which is what I would expect. If I have to wait, really even just a few seconds, I would rather just download the file locally, because when I "flip" a page, I expect the next to be there just as with a physical comic. Having to wait takes me out of the experience.

I also think using native tools is a good idea. I wrote a script that automatically unrars or unzips (depending on archive type) all my comics when they are downloaded and converts the images to WEBP and re-archives them as CBZs (to try and help loading times). The unarchive process using those native tools is quite fast, even with larger images. But I'm also not a Java programmer, so they may not be much if any faster than the Java libraries equivalent.

Anyway, thanks for listening. Hope that helps some.

We agree on what a good streaming experience is, I just have to take the time to improve it. ;)

What I have in mind is not to extract the whole archive before serving the first page, but rather serve it as soon as it is available. This solution involves some problems like how to know when the image file is actually ready (not in the middle of extraction), how to manage concurrent request on the same comic archive, how to manage extracted images eviction (they can use a lot of disk space), etc.

As for the speed of the extraction itself, from I have read so far, zip file extraction should be almost as fast as a native tool (didn't do the test yet). Rar extraction is done using an old Java library that is not maintained anymore, so I wouldn't be surprised if the performances were not that good (I use CBR for testing only as I think this is an almost completely useless file format for comics).

Last question: the CBZ and CBR files you tested, did they contain WEBP files ?

Just to chime in here: If you're gonna pre-extract images, you might consider preprocessing them at the same time (resizing, transcoding) and caching the result. Then when a request came in you could stream the raw file to the web-server instead of having to read the entire thing, process it, encode it, and *then* start transferring it.

I guess you could also just skip image processing completely and pipe the unzip output straight to the web-server (if the API allows it) - it'd be cleaner and simpler and I'm guessing just as fast, although it would also mean clients would potentially get sent images in weird formats or large sizes. But web-browsers and comic-readers are built to deal with those in any case so IMHO it's not a deal-breaker.

The problem with resizing images upon extraction is that the resulting cached comic will specific to a screen size. Size which can easily change when resizing the browser window for instance.
And since I have to do the resizing on the fly, I might as well do the transcoding since resizing the image requires transcoding it anyway.
Or I could decide that a width of 1200 or 1500px for a comic image is goos enough for everybody (the Apple way of doing things ;)) and do the resizing/transcoding as soon as I extract the image.
Not my favorite solution but it might be the most efficient one in the end.

The problem is, if I allow usage of native commands to do the extraction, the extraction/resizing phase will have to be done in two passes. More complexity, more concurrency problems, more possibility for failures.
Nothing impossible but longer to implement.

As for serving the original file, it has already been requested for WebP files (somewhere in this forum) and declined as I don't think of WebP as a viable format (at least until it gets support outside of Google products).
Although the performance increase is a much better reason for doing it, it would still be difficult as it would require Ubooquity to know the type of each image in the comic archive when it generates the HTML for the online reader and the OPDS XML (the type is explicitely defined in the OPDS link).
I don't know how to do that efficiently yet.
Still not easy but I have a few ideas...

Some of them were WebP, not all. I didn't really notice much of a difference with them, which I guess now after reading these latest comments makes sense as I had just naively assumed it was serving the original images.