0

Huge collection corrupts database all times

Moises Belda 2 months ago updated by Carsten Albrecht 1 month ago 6

I'm trying to add a huge collection of ebooks (80k more or less) to my Ubooquity installation based on docker. I've added some extra memory with parameter MAXMEM=2048 in order to avoid memory problems during the long indexation needed

The problem is that when it finishes to index (without any error in logs), the system becomes unstable, I cannot access to the web UI correctly. For this reason I need  to restart the docker container. When I restart it, in the startup log, the "Loading preferences"step takes about 10 minutes and it launches and error "Could not get number of existing tables".  I attached the error below. It seems like the database became corrupted, also I've checked the database generated (ubooquity-5.mv.db) and its size is 7GB. I have repeated the process several times, indexing the full library, or even indexing the library in 3 stages, but at the end, the last stage fails in the same way.

Is there any limitation in the size of the database which at the end makes the database corrupted?

Thx in advance. 

Error launched when I restart after indexing all the elements:

20201113 12:08:59 [main] INFO  com.ubooquity.Ubooquity - Loading preferences
20201113 12:11:08 [main] ERROR com.ubooquity.data.database.d - Cound not get number of existing tables
org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-193]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.message.DbException.get(DbException.java:168) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.message.DbException.convert(DbException.java:295) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.openDatabase(Database.java:304) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.<init>(Database.java:267) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.openSession(Engine.java:64) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.openSession(Engine.java:176) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSessionAndValidate(Engine.java:154) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSession(Engine.java:137) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSession(Engine.java:27) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:349) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:115) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:99) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:188) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:351) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:383) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:225) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:197) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at com.ubooquity.data.database.d.h(SourceFile:132) [Ubooquity.jar:2.1.2]
    at com.ubooquity.data.database.d.<init>(SourceFile:52) [Ubooquity.jar:2.1.2]
    at com.ubooquity.Ubooquity.main(SourceFile:211) [Ubooquity.jar:2.1.2]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
    at com.ubooquity.e.a.a(SourceFile:823) [Ubooquity.jar:2.1.2]
    at com.ubooquity.Launcher.main(SourceFile:10) [Ubooquity.jar:2.1.2]
Caused by: java.lang.NullPointerException: null
    at org.h2.mvstore.db.ValueDataType.compare(ValueDataType.java:102) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.compare(MVMap.java:708) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.Page.binarySearch(Page.java:334) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.binarySearch(MVMap.java:461) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.get(MVMap.java:450) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.remove(MVMap.java:538) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.TransactionStore.rollbackTo(TransactionStore.java:502) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.TransactionStore$Transaction.rollback(TransactionStore.java:799) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.openDatabase(Database.java:273) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    ... 24 common frames omitted
20201113 12:13:03 [main] ERROR com.ubooquity.data.database.d - Could not configure collation strength
org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-193]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.<init>(Database.java:267) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.openSession(Engine.java:64) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.openSession(Engine.java:176) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSessionAndValidate(Engine.java:154) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSession(Engine.java:27) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:349) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:115) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:99) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:188) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:351) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:383) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:225) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:197) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at com.ubooquity.data.database.d.i(SourceFile:155) [Ubooquity.jar:2.1.2]
    at com.ubooquity.data.database.d.<init>(SourceFile:53) [Ubooquity.jar:2.1.2]
    at com.ubooquity.Ubooquity.main(SourceFile:211) [Ubooquity.jar:2.1.2]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
    at com.ubooquity.e.a.a(SourceFile:823) [Ubooquity.jar:2.1.2]
    at com.ubooquity.Launcher.main(SourceFile:10) [Ubooquity.jar:2.1.2]
Caused by: java.lang.NullPointerException: null
    at org.h2.mvstore.db.ValueDataType.compare(ValueDataType.java:102) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.compare(MVMap.java:708) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.Page.binarySearch(Page.java:334) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.binarySearch(MVMap.java:461) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.get(MVMap.java:450) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.remove(MVMap.java:538) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.TransactionStore.rollbackTo(TransactionStore.java:502) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.TransactionStore$Transaction.rollback(TransactionStore.java:799) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.MVTableEngine$Store.initTransactions(MVTableEngine.java:255) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.open(Database.java:765) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.openDatabase(Database.java:273) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    ... 24 common frames omitted</init></init></init></init></init></init></init></init>

Nobody suffered same problem?

there are better programs for ebooks IMO.Comics work great of course

any suggestion for large collections of books?

calibre with Calibre Web

It does not matter whether it's a huge load of comics or books. The cause for this and similar issues is a leak on open file handles. It can be analyzed in the logs. In my case, ubooquite was running fine for approx. one month, and then I seem to have hit some sort of item limit. I have installed the server on my Synology DS916+ with latest DSM release 6.2.3 and JAVA8.

Yesterday suddenly icons of books and comics disappeared. This evening, when I added a couple of epub books, instead of adding them from one moment to the next one ALL comics vanished from the database. I even wasn't able to log on to the admin interface any longer, due to a JAVA exception "too many open files".

Since ubooquity is running with lots of other services on my DS, I guess it depends on those other services running at the same time at which point the OS (DSM is linux-driven) declines to give further file handles to ubooquity.

And at that point it is more or less random where the crash occurs, be it deleting entries from the database or corrupting the whole database at once.

It is too sad that the server hasn't been maintained/updated for mor than two years now, and it is even more sad that it is not open source - otherwise I'd been able to fix all the file handles and memory leaks on my own.