0
Under review

is there a size limit on database?

Adi Spivak 2 years ago updated 1 year ago 9

is there a limit to the database size?

during initial scan, some folder/books are added to the database, and at some point an error is shown and from that point nothing is added and all the rest show the same error (and while the indexer shows 16000+ books, 15000 are missing) :

20180730 14:26:12 [Scanner thread] ERROR com.ubooquity.data.database.c - Error while creating/updating comic in database
org.h2.jdbc.JdbcSQLException:
General error: "java.lang.IllegalStateException: Reading from
nio:/volume1/homes/Ubooquity/.Ubooquity2/ubooquity-5.mv.db failed; file
length -1 read length 1536 at 24132435 [1.4.193/1]"; SQL statement:
merge
into comic(parentid, rootid, isdirectory, title, norm_title, filedate,
filepath, filename, norm_filename,
lastupdatedate,filesize,description,language,norm_language,seriesname,norm_seriesname,seriesindex,altseriesname,norm_altseriesname,altseriesindex,volume,publisher,publicationdate,writers,norm_writers,pencillers,norm_pencillers,genre,norm_genre,pagecount)
key(filepath)values(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [50000-193]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.message.DbException.get(DbException.java:168) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.message.DbException.convert(DbException.java:295) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.Command.executeUpdate(Command.java:266) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]

at
org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:201)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at com.ubooquity.data.database.c.a(SourceFile:649) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.database.c.a(SourceFile:68) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.feeder.a.b(SourceFile:539) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.feeder.a.b(SourceFile:459) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.feeder.a.b(SourceFile:112) [Ubooquity.jar:2.1.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
Caused
by: java.lang.IllegalStateException: Reading from
nio:/volume1/homes/Ubooquity/.Ubooquity2/ubooquity-5.mv.db failed; file
length -1 read length 1536 at 24132435 [1.4.193/1]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:435) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.FileStore.readFully(FileStore.java:98) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Page.read(Page.java:190) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.MVStore.readPage(MVStore.java:1954) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.MVMap.readPage(MVMap.java:736) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Page.getChildPage(Page.java:217) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Cursor.min(Cursor.java:129) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Cursor.hasNext(Cursor.java:36) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]

at
org.h2.mvstore.db.TransactionStore$TransactionMap$1.fetchNext(TransactionStore.java:1388)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.db.TransactionStore$TransactionMap$1.(TransactionStore.java:1384) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]

at
org.h2.mvstore.db.TransactionStore$TransactionMap.keyIterator(TransactionStore.java:1379)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at
org.h2.mvstore.db.TransactionStore$TransactionMap.keyIterator(TransactionStore.java:1367)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.db.MVSecondaryIndex.find(MVSecondaryIndex.java:309) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.db.MVSecondaryIndex.find(MVSecondaryIndex.java:254) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.index.BaseIndex.find(BaseIndex.java:128) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.index.IndexCursor.find(IndexCursor.java:169) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.table.TableFilter.next(TableFilter.java:467) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.dml.Update.update(Update.java:102) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.dml.Merge.merge(Merge.java:154) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.dml.Merge.update(Merge.java:106) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.CommandContainer.update(CommandContainer.java:98) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.Command.executeUpdate(Command.java:258) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
... 7 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) ~[na:1.8.0_161]
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:721) ~[na:1.8.0_161]
at org.h2.store.fs.FileNio.read(FilePathNio.java:74) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:421) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]


sorry, forgot to say, running on synology 918+ with os version 6.2

Maby not much of help but i have indexed 30000+ on Windows 10 with java 8.

Running fine wih +104000 books.

so how do i get in touch with the developers to investigate the database corruption i am experiencing?


thanks.

adi

Under review

Did this problem happen several times ?

i tried this morning to click on clear books database and restart the scan, somewhere in the middle the errors have started again so, yes it does happen several times:


20180731 10:01:39 [Scanner thread] INFO
com.ubooquity.fileformat.pdf.b - Reading metadata of pdf file:
/volume1/Books/2.7K IT Books/Model-Based Development_ Applications.pdf
20180731 10:01:40 [Scanner thread] INFO
com.ubooquity.data.database.a - Inserting/updating book entry in
database : /volume1/Books/2.7K IT Books/Model-Based Development_
Applications.pdf
20180731 10:01:40 [Scanner thread] ERROR com.ubooquity.data.database.a - Error while creating/updating book in database
org.h2.jdbc.JdbcSQLException:
General error: "java.lang.IllegalStateException: Reading from
nio:/volume1/homes/Ubooquity/.Ubooquity2/ubooquity-5.mv.db failed; file
length -1 read length 768 at 4085534 [1.4.193/1]"; SQL statement:
merge
into book(parentid, rootid, isdirectory, title, norm_title, authors,
norm_authors, description, language, norm_language, rating, seriesname,
norm_seriesname, seriesindex, filedate, filepath, filename,
norm_filename, identifier, publisher,
lastupdatedate,filesize,publicationdate,pagecount)
key(filepath)values(?, ?, ? ,? , ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?) [50000-193]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.message.DbException.get(DbException.java:168) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.message.DbException.convert(DbException.java:295) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.Command.executeUpdate(Command.java:266) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]

at
org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:201)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at com.ubooquity.data.database.a.a(SourceFile:987) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.database.a.a(SourceFile:59) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.feeder.a.a(SourceFile:221) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.feeder.a.a(SourceFile:305) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.feeder.a.b(SourceFile:105) [Ubooquity.jar:2.1.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
Caused
by: java.lang.IllegalStateException: Reading from
nio:/volume1/homes/Ubooquity/.Ubooquity2/ubooquity-5.mv.db failed; file
length -1 read length 768 at 4085534 [1.4.193/1]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:435) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.FileStore.readFully(FileStore.java:98) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Page.read(Page.java:190) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.MVStore.readPage(MVStore.java:1954) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.MVMap.readPage(MVMap.java:736) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Page.getChildPage(Page.java:217) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Cursor.min(Cursor.java:129) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Cursor.hasNext(Cursor.java:36) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]

at
org.h2.mvstore.db.TransactionStore$TransactionMap$1.fetchNext(TransactionStore.java:1388)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.db.TransactionStore$TransactionMap$1.(TransactionStore.java:1384) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]

at
org.h2.mvstore.db.TransactionStore$TransactionMap.keyIterator(TransactionStore.java:1379)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at
org.h2.mvstore.db.TransactionStore$TransactionMap.keyIterator(TransactionStore.java:1367)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.db.MVSecondaryIndex.find(MVSecondaryIndex.java:309) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.db.MVSecondaryIndex.find(MVSecondaryIndex.java:254) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.index.BaseIndex.find(BaseIndex.java:128) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.index.IndexCursor.find(IndexCursor.java:169) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.table.TableFilter.next(TableFilter.java:467) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.dml.Update.update(Update.java:102) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.dml.Merge.merge(Merge.java:154) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.dml.Merge.update(Merge.java:106) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.CommandContainer.update(CommandContainer.java:98) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.Command.executeUpdate(Command.java:258) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
... 7 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) ~[na:1.8.0_161]
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:721) ~[na:1.8.0_161]
at org.h2.store.fs.FileNio.read(FilePathNio.java:74) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:421) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
... 28 common frames omitted
20180731 10:01:40 [Scanner thread] INFO com.ubooquity.fileformat.pdf.b - Reading metadata of pdf file: /volume1/Books/2.7K IT Books/model-based_visual_tracking.pdf
20180731 10:01:40 [Scanner thread] INFO
com.ubooquity.data.database.a - Inserting/updating book entry in
database : /volume1/Books/2.7K IT Books/model-based_visual_tracking.pdf
20180731 10:01:40 [Scanner thread] ERROR com.ubooquity.data.database.a - Error while creating/updating book in database
org.h2.jdbc.JdbcSQLException:
General error: "java.lang.IllegalStateException: Reading from
nio:/volume1/homes/Ubooquity/.Ubooquity2/ubooquity-5.mv.db failed; file
length -1 read length 768 at 4085534 [1.4.193/1]"; SQL statement:
merge
into book(parentid, rootid, isdirectory, title, norm_title, authors,
norm_authors, description, language, norm_language, rating, seriesname,
norm_seriesname, seriesindex, filedate, filepath, filename,
norm_filename, identifier, publisher,
lastupdatedate,filesize,publicationdate,pagecount)
key(filepath)values(?, ?, ? ,? , ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?) [50000-193]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.message.DbException.get(DbException.java:168) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.message.DbException.convert(DbException.java:295) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.Command.executeUpdate(Command.java:266) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]

at
org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:201)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at com.ubooquity.data.database.a.a(SourceFile:987) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.database.a.a(SourceFile:59) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.feeder.a.a(SourceFile:221) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.feeder.a.a(SourceFile:305) [Ubooquity.jar:2.1.1]
at com.ubooquity.data.feeder.a.b(SourceFile:105) [Ubooquity.jar:2.1.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
Caused
by: java.lang.IllegalStateException: Reading from
nio:/volume1/homes/Ubooquity/.Ubooquity2/ubooquity-5.mv.db failed; file
length -1 read length 768 at 4085534 [1.4.193/1]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:435) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.FileStore.readFully(FileStore.java:98) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Page.read(Page.java:190) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.MVStore.readPage(MVStore.java:1954) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.MVMap.readPage(MVMap.java:736) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Page.getChildPage(Page.java:217) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Cursor.min(Cursor.java:129) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.Cursor.hasNext(Cursor.java:36) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]

at
org.h2.mvstore.db.TransactionStore$TransactionMap$1.fetchNext(TransactionStore.java:1388)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.db.TransactionStore$TransactionMap$1.(TransactionStore.java:1384) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]

at
org.h2.mvstore.db.TransactionStore$TransactionMap.keyIterator(TransactionStore.java:1379)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at
org.h2.mvstore.db.TransactionStore$TransactionMap.keyIterator(TransactionStore.java:1367)
~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.db.MVSecondaryIndex.find(MVSecondaryIndex.java:309) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.db.MVSecondaryIndex.find(MVSecondaryIndex.java:254) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.index.BaseIndex.find(BaseIndex.java:128) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.index.IndexCursor.find(IndexCursor.java:169) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.table.TableFilter.next(TableFilter.java:467) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.dml.Update.update(Update.java:102) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.dml.Merge.merge(Merge.java:154) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.dml.Merge.update(Merge.java:106) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.CommandContainer.update(CommandContainer.java:98) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.command.Command.executeUpdate(Command.java:258) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
... 7 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) ~[na:1.8.0_161]
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:721) ~[na:1.8.0_161]
at org.h2.store.fs.FileNio.read(FilePathNio.java:74) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:421) ~[h2-1.4.193.jar.4067930195476961885.tmp:1.4.193]
... 28 common frames omitted
20180731 10:01:40 [Scanner thread]


I think I found the issue.

After some research, what I did is the following:

Stop ubooquity

Delete both db file and the DB.old

Change the configuration file memory option from 512 to 4096

Start ubooquity


After a long scan I now have 30,000 books and comics available and without issues.


Thank you for trying to help me.

Adi

I'm glad you could solve your issue.


Out of curiosity, did you have some OutOfMemoryError messages in your logs when the problem was happening ?

not that i could see, however, if it appeared just once, most likely i missed it.