Ubooquity V3, beta version

No worries man, no one was expecting this for September 21st anyway. 
I'm just glad Ubooquity isn't actually dead (as I'm sure many others are). Hope you can sort out your problems soon!

Any hints on what new features we can expect?

The most visible change is the display of the reading progress on books (by books I mean comics as well). 

But to do it I completely rewrote the way themes are managed. Until now you could only change the style and icons of Ubooquity with a custom theme. With V3, the whole HTML structure can be customized. This means a lot more freedom in the way collections can be displayed.

I also merged the "books" and "comics" parts in the code and the database (having two "chains" meant twice the work each I had to modify something), changed the language I use to develop Ubooquity, and dozens of other minor changes. I will write a full change log for the release. :)

What? Seriously? I thought this project was over. That's great news! I still use it. I really haven't found a better book library for PDFs than this one, especially if the PDF is lacking any internal or external metadata. Having a book library based on file structure has been key for my use-case. I'm waiting with bated breath. :)

With the theme rewrite do you have a theme template you could share that we could start working on ahead of the release?

That's the point of the beta version. :)

Binary's ready, I just have to finish the doc which explains how the templating system works.

I'll post it here as soon as it's ready (and write a release post later).

Here you go: https://vaemendis.net/ubooquity/downloads/Ubooquity-3.0.0.zip

Java 17 (or more recent) is required.

[EDIT]: you'll find the latest version here: http://vaemendis.net/ubooquity/downloads/

The documentation on the new theme system: https://vaemendis.github.io/ubooquity-doc/pages/theme-creation.html

And a description of the data (variables) made available to the theme, by page: https://vaemendis.github.io/ubooquity-doc/pages/theme-variables-reference.html

The variables are very unlikely to change before the release of the beta.
The default theme will (HTML and definitely CSS), as it is an awful mess in its current state.

Let me know if you have any question !

I'm giving it a try and it seems to work fine.

Did you change something to the way folder pictures are chosen? It used to be the cover of the first issue, alphabetically (provided folder.png is absent) but now it seems to chose a random issue? Or am I missing something.

Yes, the folder pictures were found using a very inefficient way that required several database requests. Now everything is done in a single requests, which completely ignores the user settings regarding sorting. 🤦‍♂️

I'll try to fix that !

Should be fixed in the official beta:

https://vaemendis.net/ubooquity/article23/ubooquity-3-beta

OK I'll try it. You might want to correct your link so that it's httpS, chrome didn't want to load it and firefox gave me a warning

Indeed, thanks !

I've installed it but it is still scanning my library. The initial scan seems to be faster than before at 149,000 and still going but not struggling on the odd files as it has done in the past. Looks like it will be done in about 14hrs instead of at least 24hrs

Hi, very happy to see this rising again from ashes ... !

Hope it will ultimatly accept rar5 .cbr.

In the meantime, it crashes (on my x64 Win 11 with the latest java RE) at start with "A java Exception has occurred" error.

Could you give me the exact version of Java you use ?

Also, the few log lines after the "A java Exception has occurred" would help too.

As for rar5, support depends on the RAR library used by Ubooquity (Junrar). I'm not sure it will be ever implemented: https://github.com/junrar/junrar/issues/23

Given that RAR files have no advantage over ZIP files for comics, I don't understand why this format is still in use though.

Was using jre-8u381-windows-x64 when first reporting the crash. I upgraded today to jre-8u391-windows-x64 and still crashes. Tested on another computer and same problem.
I must admit I don't know where to find the crash log ...
Only thing I can tell is that the crash message comes from "Java Virtual Machine Launcher", that the previous Ubooquity works great on theses computer, only the 3 beta crashes, and that on the other computer there is another message precising that the problems seems to come from "JNI" initialization system.

You are using Java 8, the new version of Ubooquity requires at least Java 17, hence the crash.

I understand that, however that version is the only one oracle is pointing to when looking for the JRE. I Have found nbo way to download somthing like a Java 17 JRE. See what happens when one is looking for the customers java download : https://www.java.com/en/ (clik on the Download link, it will redirect you to JRE 8.xxx.
I must be missing something. Has the JRE been replaced by something else ? 

Here is a link to JDK 17, though I used 21 as that was the first link I found and it is working.

https://www.oracle.com/java/technologies/javase/jdk17-archive-downloads.html

Sorry, I am an old school java end-user, and was always told that JDK was for developers and JRE (runtime) for end user. Didn't noticed that Oracle ended JRE at version 8 ann I thought that there last JRE8u392 was teh corresponding runtime to their 21 JDK ... Thanks for pointing me. Maybe a link on the Ubooquity website would help some other dumb JRE users like me.
By the way, I discovered that Eclipse still offers an up-to-date (based on OpenJDK 21) JRE.

Ok it works with Java 21 SE. Nice update. Still wishing for RAR 5. Keep the good work. Thanks

I got the CBR's with no cover from my NAS, unpacked it and repacked it with zip .. 
no issue there. I am more happy that I do not have rar5 than that I have such issues. So little investment in my own time here .. 

Hi,

Had some issues with what seem to be rar5 (cbr files) , so if you install 7 zip and add it to your PATH in windows (or hardlink to 7z.exe) the following will convert them from a CMD window:

for %f in ("*.cbr") do ( 7z e "%~nf.cbr" -otempdir && 7z a -tzip "%~nf.cbz" .\tempdir\* -sdel && rmdir /S /Q tempdir && del "%~nf.cbr" )

i leave no warranty, ;)

Installed, scan complete: no problems. Nice quick and clean interface. Nice to be able to put content on read and not read, and the progressbar is appreciated too…!

Thanks for the feedback everyone ! :)

Great .. working in a test environment .. 

But see during scan 1 comic with broken image .. 

in the old version I have a front-cover.

Looks like that 'diacritical signs' are not well handled in the new version

Aéropostale -- Legendarische Piloten - 01 - Guillaumet

in linux in shell 'A'$'\303\251''ropostale -- Legendarische Piloten - 01 - Guillaumet.cbr'

I opened an issue on the Linuxserver/Ubooquity Github, they will update their Docker image when there is a stable release!

Also: Awesome work Tom!!!!!!!

They updated their docker image to include java 17 already, so if you put the jar file on your docker host you can bind mount it to "/app/ubooquity/Ubooquity.jar" and it will launch v3, :) 

Can confirm it works!

Hello Tom,

I'm very happy to see that the work hasn't been stopped on ubooquity. I've run the beta it on my PC, and it works fine. I haven't seen any issues

Good work :-)

Its been a long time waiting, just popping over to look at the beta.

Many thanks :-)

Im liking Ubooquity 3, so far.  The only thing Ive noticed is that the "uparrow" does not return to the main page.  It works fine on the "recent additions" page, but not from "comics", "books" or "magazines"

I think this problem occurs only, if you have "Bypass single root folder" activated.

Ah ha.  That was it.  Thanks!

Hello,

Thank you for all the effort and work on this new version.

The userName variable is only available on the login and home page. This does not allow you to validate if the user has a session into the library and other sections to add a logout button and other options, to close the session you must return to the home.

Greetings.

Hello Luis,

The logout button does not need the userName variable, it just redirects to the /?logout=true URL, which logs the user out.

Also, Ubooquity will not serve any page other than the login page if the user is not logged in. So I'm not sure I understand the use case for making the userName variable available on other pages.

Apart from displaying the currently logged user name on every page, what would you use it for ?

Have a  nice day !

Finally managed to get my Fantastic Fiction mod working in V3. Still some cludgy workarounds to sort out, but overall working as expected. Built on top of Northernlights for now. Works like FF in that all text and images bring up the book details.

I look forward to this being released, I would love to see my collection of books organized like that.

I would like to release, but my database (Calibre) is formatted in a specific format that I have been doing from day one. 10,000 in db and a few TB to add.

I name my books as <series> <number> - <Title>  eg Commander Grimes 01 - Rendezvous on a Lost World

Also I have the author as <firstname> <lastname> in author sort.

Then I export as folder structure <author> <series> <title> and single titles under author dir.

My Javascript expects this format. 

I am willing to share and anyone can pick apart the logic and adapt to themselves if needed. Basically my JS finds a series and then looks at the link for the actual books under the series. (As my folder structure is in series, Ubooquity adds the folder series as separate links).

Nice !

About the workarounds, if you think about modifications that would make this kind of theme easier to make, let me know.

I can't make any promises, but if some easy to make modification to Ubooquity's code can help, I might as well throw it in.

Only major issue I had was the Progress bar / reading state was causing issues and had to be turned off (remove from page-library). This will cause not to display on comics etc. 

Tried to remove after page load with removeclass, but no go. I know the remove class works as I use it to remove the original cellcontainer classes so Ubooquity's books do not show. Only my classes.

I do not know if there is any way this could be turned off on selected categories?.

So, I'm a bit of a novice/idiot when it comes to coding, but I had figured out the way in the older version to get a "studio logo" as a folder image, but I'm drawing a bit of a blank with the new setup.

I understand the nature of how the images are called has changed to make it better, but if anyone could point me to something to assist I would greatly appreciate it.

That said, really liking the overall nature of the v3 update.

I simply have a file named folder.jpg in the folder for the studio and some of the comics as well. My images are 312x312 and work well with the new update. My folder structure is D:\<comics><image><Kill or be Killed>kill or be killed 01.cbz

On the studio page I also have a header image on some of the studios, that is driven by this code in a html & css file.

The header.jpg is 2200x512  hope this help. obviously there is nothing to check for viewport size but it scales nicely regardless for me. as you can see the mix of comics with and without their own folder.jpg, lots of work to fill in all the missing ones. I hope this helps.

Oh, that is awesome, gives me something to focus on to sort it out.
I'll dig in tonight and see what's what.

Thanks for the response.

update --
Ok looked at this and I did have a similar setup from previous versions with similar sized header & footers, but for some reason the new system is not picking up those css & html's and implementing them, so i guess I'm missing something.

I did edit the base template theme to more match plex colours and add the button to be able to go home from the top nav bar on any page, but didn't change anything to references those 2 files in the studio folder or the jpg. I fact I had forgotten they were even there and was surprised when the pages loaded. Maybe try creating a new theme, simply name it on the settings page and see if that makes a difference?

Well ... I'm an idiot.

so, for anyone else who has my mental block, I had neglected when I installed clean the new version to make sure that I had checked the box in the Advanced Settings tab, to "Enable folder metadata display" so ... no wonder it wasn't reading the files that I had all nicely curated.
I may do more overall color tweaks like you said you did, but now that I have all but the newest (the Skybound/Energonverse) with their logo's that I spend time making, I'm happy for now.

Got this running side by side with my 2.x Ubooquity installs and its working great now that I'm calling the right java version to run each.  

Was wondering if there will be any way to port the 2.x database info to 3.x.  I have a lot of reading history that I'd like to be able to carry over to version 3.  

If not, can someone confirm what database this is using?  I thought I saw a reference to H2 somewhere, but want to be sure.  I'll look into creating an ETL process myself if data transfer is not going to be part of the release.  

I haven't used H2 before, but I used to do data warehousing, business intelligence, and ETL for a living, mostly with Oracle and SQL Server, so I'm willing to take a crack at it. If anyone else is interested, let me know and I'll try to stick to platform agnostic tools and will document the process (if it works).  

Hi, I can confirm that this is, indeed, using H2

As Bob said, the database is H2. But there is no plan for an automatic database content migration.

More specifically, the last Ubooquity 2 version uses H2 1.4.193 while Ubooquity 3 currently uses H2 2.2.222.
Any JDBC compatible client will be able to open a Ubooquity database (use "H2 embedded" mode). I personally like to use DBeaver.

If your goal is to migrate the bookmark table, you don't necessarily need to migrate everything.

You might try something like :

• import all your files in Ubooquity 3
• create a mapping between the v3 document id and the v2 document id + type (as there are 2 different tables in v2: books and comics, whereas there is only 1 in v3)
• then you can copy the bookmarks from the old table to the new, applying some conversion function in the process, as the lastupdate field was renamed to updateDate, and the boolean isFinished was replaced by an enumeration with the following values :
  • UNREAD (which you don't need, as the bookmark does not exists in this case)
  • IN_PROGRESS
  • FINISHED

This is just from the top of my head, there might be a better way.

(e.g: once you have the mapping, you might use the bookmark REST API to insert bookmarks, the advantage being that when using the API you don't need to provide the reading status, is is automatically computed by the server)

Let me know if you need any more info.

Thanks Tom, that gives me a great place to start.  Based on the consolidation you mentioned around books/comics, I had a feeling there were some table changes around that.  

I'll look into the options around mapping the bookmarks vs a full data migration.  

Appreciate the info around the field/data changes made as well.  If I come up with anything that might be useful, I'll report back here.  

Got the data migration to work tonight.  Per Tom's suggestion, I just migrated over the bookmarks.  I can confirm that the finished and in progress bars are showing up on the comics.  

The whole process is just two SQL scripts, one of which is wrapped up in a data export task, which also defines the landing table.  I used DBeaver Community Edition, also per Tom's suggestion as its free.  Just an FYI if anyone else does this, I had to add the JDBC driver for the version of H2 being used in Ubooquity 3.  I was playing around with DBSchema, which had the updated driver, so I just pulled it from there. I'm running DBeaver 24.1.3, so a future version may have the more recent JDBC driver.

In order to do this as simply as possible, I do export the data from the Ubooquity v2 database to a new table in the Ubooquity v3 database via the data export functionality in DBeaver.  After running the insert script, the new table can be dropped.  

Again, if anyone is interested, please let me know and I'll create a post with the SQL code and screenshots from DBeaver for the data export settings, along with instructions.  

yes please share your script and a tutorial :-)

Due to a harddrive issue, I've lost my Ubooquity 3 Beta install.  I get the 404 page if I try to get the download as linked.  Any chance of getting pointed to it?

It's ok, I was able to go directly to the download itself, and found 3.0.2.  The link pointed to an earlier version that is no longer there.

But you're right, I edited the message with a link to the download folder so that the latest version can always be found.

Hi Tom!

I've been playing around with writing a theme for Ubooquity v3 and I've noticed a few things that I thought I should bring to your attention:

1.  If your numbering system follows 1 thru 10 and 10 thru 100 then the sorting system lists 1, 10 thru 19, and then starts at 100.  Of course, someone could rename of their collections 001 thru 100 and the problem goes away.  But it is a bug that been around since version 2.

2. I would really like to be able to access {{#userName}} mustache tag while using the files in the /library folder because, as it is, you can't logout anywhere except the page-home.htm (especially the page-library and page-library-category-root-dir.html files).

3. Would it be too much to ask for a "items per page" for each used category so we have the option to load more or less depending on those categories?  Similarly, can we have a way to turn off the items per page when displaying the page-library-category-root-dir.html?  Maybe a mustache tag override? :)

4. I'm assuming this is a scripting error that I'll find and fix eventually, but the arrow up that goes to the {{parentUrl}} in the toppagebar <div> in the default theme does not go back any farther that the page-library-category-root-dirs,html page.  In other words, there is no way to get back to the page-home.html page once you've entered the page-library-category-root-dirs.html page. The easy workaround was to add and <a href="{{rootPath}}"> tag around the Ubooquity banner.

Also, I would like to thank you so much for continuing Ubooquity.  I am pretty happy with v2 using my heavily edited comixology 2 theme.  I like the sliders on the homepage and the added ability to listen to my audio books. I even used some javascript to remember which header image I was displaying in the parent folders so I didn't have to have one in every sub-folder (I thought that was pretty slick). ;)

Anyway, I plan on doing something a bit more original with v3.  I've already integrated a dual color scheme that seems to be fully functional on every page (which was a challenge because lots of things disappeared until I realized you were using svgs all over the place that I had to use filter on to get to display).  After I finish integrating the color schemes into the popups I'm going to really focus on a snazzy looking homepage and some audiobook functionality.  If it turns out decent I'll submit it for your approval and share. 

Peace out \/

G

Couple of questions for those in the know:

1.  Are there any plans on changing the current lexicographical sorting method or, perhaps, adding the feature of numeric sorting in Ubooquity?  I ask because I have over 50,000 comics/books/magazines that are sorted numerically.  And while it's no big deal to use a batch script to rename them so they'll sort correctly with a lexicographical sort, I'd rather hold off if there's something in the works.

2.  I saw someone else reported that {{parentUrl}} will not go back to the home page when "bypass root folder" is on.  For anyone intersted, here is a small javascript fix to add to your inc-library-topbar.html:

document.addEventListener('DOMContentLoaded', function() { const currentUrl = window.location.href;
if (!(currentUrl.includes('latest=true') && currentUrl.includes('random=true') && currentUrl.includes('search=simple'))) { const parentUrl = document.querySelector('#arrowup').getAttribute('href'); const rootPath = '{{rootPath}}'; const category = '{{category}}'; const targetUrl = `${rootPath}/${category}/`;
if (parentUrl === targetUrl) { document.querySelector('#arrowup').setAttribute('href', rootPath); } } });

3. Has anyone else noticed that {{parentUrl}} becomes {{rootPath}} when closing the reader app after reading a book/comic/magazine displayed during a search?  Likewise, when you enter a folder that's displayed during a search, {{parentUrl}} becomes the parent of that particular folder instead of going back to the search results page. Fortunately, I have been able to fix most of this with javascript but I was wondering if it's going to stay that way or if there are plans to address this in the future?

4.  My final question is probably more directed at Tom and it's regarding the "items to display" setting.  There are times when I would really like to change the number of cellcontainers displayed in a page.  If I set "items to display" at a higher value, I am able to truncate the results.  However, it seems I can't prefetch or get information from the {{nextPageUrl}} without loading it.  Am I missing something or is what I'm trying to do a futile attempt?

Thanks,

G

Hello Gary,

sorry for having missed your message two weeks ago.

1. The sorting in Ubooquity is done at the database level, and unfortunately the database I use (H2 Database) does not offer any way to plug a custom sorting algorithm. Changing the underlying database is not possible as it would make the program a lot lesse portable (unless I find another full Java embeddable database). So no, nothing in the works.

2. Indeed there is a bug, I'll fix it.

3. I will look into it.

4. I don't understand what you are trying to do. The "items per page" setting is just that: the number of books or comics contained by each displayed page. Each page is plain HTML returned by the server, with a number of books corresponding to the number you chose. The {{nextPageUrl}} is just the URL to another page with the next X books (X being the "items per page").

(I see there are other questions in your previous post, I'll look at them too)

Hi Tom.

The sorting algorithm is not really a problem.  Like I said, it's nothing to write a batch script to rename my books.  I just didn't want to do it and then change it back. ;)

As for what I'm trying to accomplish with "items per page"...

There are times, like when I'm displaying the {{rootPath}}/comics or {{rootPath}}/books section that I don't want to limit display to an items per page.  I want all the folders displayed.  Then, like when displaying the latest or random comics/books, I'd like to customize the number of items to display.  What would really be interesting is creating a mustache tag to do it.  Maybe something like {{#itemsperpage=50}} this block would only load 50 items on the page {{/#itemsperpage}} or {{itemsperpage=all}} this block would display everything {{/itemsperpage}}

BTW:  I've been making good progress on my theme design.  I added a button to the popupmenu for bookmarks so that I can create story arcs and book series' with JSON files (like the comixology theme) except the JSON exporter allows you to remove single issue as well as fill out the metadata fields to use for your bookseries/story arcs pages.

Here is a taste:

Just to give you an idea of what it displays like in story arcs or book series:

BTW:  Did you notice that the user name appears in the banner of my page-library?  session storage is a wonderful thing. :)

Good evening, I thought I would once again see if anyone else had run into this odd thing.

I got a new PC, so had to re-install/setup my stuff, and while I am 99% back up and running (and enjoying the current v3 beta with no real issues) I did run into an odd thing that I can't recall if I had happen on my old setup, so not sure if its a glitch in my setup or something else.
I'm seeing a decent number of folders/comics that for some reason don't pull up with cover images, for instance below in my Dynamite Comics section, Bob's Burgers overall shows as just a folder, because the first comic doesn't seem to want to populate the image. It is a bog standard .cbz file similar to all the rest and when I look with my local viewer (ComicRack) it seems fine.
It doesn't affect reading it at all from what I can tell, just makes it a little less cohesive for quick browsing/viewing.

Any thoughts? did I break something or configure something odd when I set it up this time?

Hi Seth,

Check your logs.  See if you're getting an error when it tries to import the comic. 

If I remember correctly, I had the same problem with a few comics when I first loaded them into Ubooquity 2.12.  What I found was that the offending .cbz/.cbr contained a single non-image file (usually a .cvs that someone added for navigation purposes) that interfered with loading the first image of the comic into Ubooquity.  To fix it, I had to open the comic with comic rack, delete the file from the archive, and then re-load it into Ubooquity.

Hope that helps. But definitely check your logs for more information.  They should tell you if Ubooquity is having a problem loading the image from a comic or book.  That's how I ended up tracking it down.

Regards,

G

Yeah, not 100% sure what was the cause, but some of the cbr's seemed to be ... not correct, so just reconverting them back to that format again seems to have fixed it.

I do know that sometimes people rename CBRs or CBZs without actually converting them... and that can cause problems as well.  There is a little utility out there called ZENCBR.  It's a little difficult to navigate until you get the hang of it, but it checks a folder (and all sub-folders) for CBRs and CBZs, tests them to find out if they're CBR or CBZ, and coverts archives that are improperly named to whichever you choose.  I believe it went through all 50,000 of my comics in about 2 minutes.  Then, when I reimported them into Ubooquity, a bunch of books showed up that I could not get to import earlier.  Imagine that?  Heh!  

Good Luck! 
G

Maybe I'm doing this wrong, but when I go to the link to download the v3 beta I just get a 404 error. Is there somewhere else I need to go to install the v3 Beta?

sounds like the link is pointing incorrectly.  If can, check the text of the URL.  You may have to manually go to the overall download page and click the appropriate version folder there

https://vaemendis.net/ubooquity/downloads/

In the downloads folder, I only see version 3.0.2 from 2023.  The beta directory is a completely blank screen.  No directory listing or anything.

That's the one. AFAIK, 3.0.2 is the latest beta (and, yeah, it's already pretty old)

https://vaemendis.net/ubooquity/downloads/Ubooquity-3.0.2.zip