1. General
  2. Bugs
0

Huge collection corrupts database all times

Moises Belda 3 years ago • updated by F688 2 years ago • 10

I'm trying to add a huge collection of ebooks (80k more or less) to my Ubooquity installation based on docker. I've added some extra memory with parameter MAXMEM=2048 in order to avoid memory problems during the long indexation needed

The problem is that when it finishes to index (without any error in logs), the system becomes unstable, I cannot access to the web UI correctly. For this reason I need  to restart the docker container. When I restart it, in the startup log, the "Loading preferences"step takes about 10 minutes and it launches and error "Could not get number of existing tables".  I attached the error below. It seems like the database became corrupted, also I've checked the database generated (ubooquity-5.mv.db) and its size is 7GB. I have repeated the process several times, indexing the full library, or even indexing the library in 3 stages, but at the end, the last stage fails in the same way.

Is there any limitation in the size of the database which at the end makes the database corrupted?

Thx in advance. 

Error launched when I restart after indexing all the elements:

20201113 12:08:59 [main] INFO  com.ubooquity.Ubooquity - Loading preferences
20201113 12:11:08 [main] ERROR com.ubooquity.data.database.d - Cound not get number of existing tables
org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-193]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.message.DbException.get(DbException.java:168) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.message.DbException.convert(DbException.java:295) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.openDatabase(Database.java:304) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.<init>(Database.java:267) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.openSession(Engine.java:64) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.openSession(Engine.java:176) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSessionAndValidate(Engine.java:154) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSession(Engine.java:137) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSession(Engine.java:27) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:349) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:115) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:99) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:188) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:351) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:383) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:225) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:197) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at com.ubooquity.data.database.d.h(SourceFile:132) [Ubooquity.jar:2.1.2]
    at com.ubooquity.data.database.d.<init>(SourceFile:52) [Ubooquity.jar:2.1.2]
    at com.ubooquity.Ubooquity.main(SourceFile:211) [Ubooquity.jar:2.1.2]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
    at com.ubooquity.e.a.a(SourceFile:823) [Ubooquity.jar:2.1.2]
    at com.ubooquity.Launcher.main(SourceFile:10) [Ubooquity.jar:2.1.2]
Caused by: java.lang.NullPointerException: null
    at org.h2.mvstore.db.ValueDataType.compare(ValueDataType.java:102) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.compare(MVMap.java:708) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.Page.binarySearch(Page.java:334) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.binarySearch(MVMap.java:461) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.get(MVMap.java:450) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.remove(MVMap.java:538) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.TransactionStore.rollbackTo(TransactionStore.java:502) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.TransactionStore$Transaction.rollback(TransactionStore.java:799) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.openDatabase(Database.java:273) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    ... 24 common frames omitted
20201113 12:13:03 [main] ERROR com.ubooquity.data.database.d - Could not configure collation strength
org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-193]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.<init>(Database.java:267) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.openSession(Engine.java:64) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.openSession(Engine.java:176) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSessionAndValidate(Engine.java:154) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Engine.createSession(Engine.java:27) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:349) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:115) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:99) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:188) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:351) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:383) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:225) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:197) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at com.ubooquity.data.database.d.i(SourceFile:155) [Ubooquity.jar:2.1.2]
    at com.ubooquity.data.database.d.<init>(SourceFile:53) [Ubooquity.jar:2.1.2]
    at com.ubooquity.Ubooquity.main(SourceFile:211) [Ubooquity.jar:2.1.2]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
    at com.ubooquity.e.a.a(SourceFile:823) [Ubooquity.jar:2.1.2]
    at com.ubooquity.Launcher.main(SourceFile:10) [Ubooquity.jar:2.1.2]
Caused by: java.lang.NullPointerException: null
    at org.h2.mvstore.db.ValueDataType.compare(ValueDataType.java:102) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.compare(MVMap.java:708) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.Page.binarySearch(Page.java:334) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.binarySearch(MVMap.java:461) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.get(MVMap.java:450) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.MVMap.remove(MVMap.java:538) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.TransactionStore.rollbackTo(TransactionStore.java:502) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.TransactionStore$Transaction.rollback(TransactionStore.java:799) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.mvstore.db.MVTableEngine$Store.initTransactions(MVTableEngine.java:255) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.open(Database.java:765) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    at org.h2.engine.Database.openDatabase(Database.java:273) ~[h2-1.4.193.jar.8884071570020133214.tmp:1.4.193]
    ... 24 common frames omitted</init></init></init></init></init></init></init></init>

Moises Belda 3 years ago

Nobody suffered same problem?

martin harrington 3 years ago

there are better programs for ebooks IMO.Comics work great of course

Moises Belda 3 years ago

any suggestion for large collections of books?

martin harrington 3 years ago

calibre with Calibre Web

Bobby Hill 3 years ago

Ubooquity documentation (vaemendis.github.io)


This this?

Carsten Albrecht 3 years ago

It does not matter whether it's a huge load of comics or books. The cause for this and similar issues is a leak on open file handles. It can be analyzed in the logs. In my case, ubooquite was running fine for approx. one month, and then I seem to have hit some sort of item limit. I have installed the server on my Synology DS916+ with latest DSM release 6.2.3 and JAVA8.

Yesterday suddenly icons of books and comics disappeared. This evening, when I added a couple of epub books, instead of adding them from one moment to the next one ALL comics vanished from the database. I even wasn't able to log on to the admin interface any longer, due to a JAVA exception "too many open files".

Since ubooquity is running with lots of other services on my DS, I guess it depends on those other services running at the same time at which point the OS (DSM is linux-driven) declines to give further file handles to ubooquity.

And at that point it is more or less random where the crash occurs, be it deleting entries from the database or corrupting the whole database at once.

It is too sad that the server hasn't been maintained/updated for mor than two years now, and it is even more sad that it is not open source - otherwise I'd been able to fix all the file handles and memory leaks on my own.

F688 2 years ago

Has this issue ever been resolved? 

Edward Yagi 2 years ago

I had this issue for at the beginning, i used to stop the scan, save the db and restart, so i could continue if it crashed/corrupted. It got tedious. I finally updated my java, but also, I specified a larger memory allocation when i started the jar file:   java -jar -Xmx4g Ubooquity.jar and it hasnt crashed since. Im at 40k books/comics.

-1
RiC 2 years ago

Just for metrics: my collection (mostly epub) counts around 60k files with a database around 845mb. Performace is good when no scanning oparations are active. No loss of data in the last years. Using docker on a Synology 21+ with plenty of RAM running from SSD, books are on HDD.

F688 2 years ago

Thanks but a solution would be to have a database backup feature or fix the leak but without any updates nothing can be resolved.

Customer support service by UserEcho