0
Fixed

V3.0.1 beta scans not showing complete on dashboard but logs show complete

Matthew Bourns 6 months ago updated by Tom 4 months ago 6

When a scan is started it adds all the new items found and the user can view/ download etc but on the admin dashboard it does not show as complete the Total line is not populated. The log file though indicates that it is complete or at least it does from what I can tell. When I try to start another scan I get the message that one is already running. At the moment I simply stop and restart the server to resolve this. When I restart it then shows the total line populated from the previous scan 


Here are the last lines from my log file after I stopped and restarted the server:

20231105 08:36:25 [Scanner thread] INFO com.ubooquity.a.a.a.d - Inserting/updating opus entry in database : L:\Magazines\Crafts
20231105 08:36:26 [Scanner thread] INFO com.ubooquity.a.a.f - Found 14800 empty folders
20231105 08:36:28 [Scanner thread] INFO com.ubooquity.a.a.f - Deleted empty folders from database. Looking for new ones...
20231105 08:36:28 [Scanner thread] INFO com.ubooquity.a.a.f - Found 144 empty folders
20231105 08:36:29 [Scanner thread] INFO com.ubooquity.a.a.f - Deleted empty folders from database. Looking for new ones...
20231105 08:36:30 [Scanner thread] INFO com.ubooquity.a.b.a - Finished scanning directories. Populating caches...
20231105 08:36:35 [Scanner thread] INFO com.ubooquity.a.b.a - Caches populated
20231105 08:36:35 [Scanner thread] INFO com.ubooquity.a.b.a - Statistics: [total: 148999, added or updated: 8, removed: 0]
20231105 13:55:28 [AWT-EventQueue-0] INFO com.ubooquity.c.f - Applying preferences modifications and restarting server
20231105 13:55:28 [AWT-EventQueue-0] INFO com.ubooquity.a - Stopping internal server...
20231105 13:55:28 [AWT-EventQueue-0] INFO com.ubooquity.a.b.a - Aborting content scan...
20231105 13:55:28 [AWT-EventQueue-0] INFO com.ubooquity.a.b.a - Waiting for scanner thread to stop
20231105 13:55:33 [AWT-EventQueue-0] INFO com.ubooquity.a.b.a - Interrupting stuck scanner thread
20231105 13:55:33 [AWT-EventQueue-0] INFO com.ubooquity.a.b.a - Scanner thread stopped
20231105 13:55:33 [AWT-EventQueue-0] INFO org.eclipse.jetty.server.Server - Stopped Server@9a20cbd{STOPPING}[11.0.15,sto=0]
20231105 13:55:33 [AWT-EventQueue-0] INFO o.e.jetty.server.AbstractConnector - Stopped library@7126e26{HTTP/1.1, (http/1.1)}{0.0.0.0:2202}
20231105 13:55:33 [AWT-EventQueue-0] INFO o.e.jetty.server.AbstractConnector - Stopped admin@35adf623{HTTP/1.1, (http/1.1)}{0.0.0.0:2203}
20231105 13:55:33 [AWT-EventQueue-0] INFO o.e.j.server.handler.ContextHandler - Stopped o.e.j.s.h.ContextHandler@3bb9ca38{/,null,STOPPED,@library}
20231105 13:55:33 [AWT-EventQueue-0] INFO o.e.j.server.handler.ContextHandler - Stopped o.e.j.s.h.ContextHandler@5934153e{/,null,STOPPED,@admin}
20231105 13:55:33 [AWT-EventQueue-0] INFO com.ubooquity.a - Internal server stopped
20231105 13:55:33 [AWT-EventQueue-0] INFO c.u.pref.PreferencesRepository - Saving preferences...
20231105 13:55:33 [AWT-EventQueue-0] INFO com.ubooquity.a - Starting internal server...
20231105 13:55:33 [AWT-EventQueue-0] INFO com.ubooquity.a.b.a - Ignoring scan request as another scan is already running.
20231105 13:55:33 [Scanner thread] ERROR com.ubooquity.Ubooquity - Uncaught exception on thread: Scanner thread
org.h2.jdbc.JdbcSQLNonTransientException: General error: "org.h2.mvstore.MVStoreException: Reading from file sun.nio.ch.FileChannelImpl@9f60d0c failed at 189418556 (length -1), read 0, remaining 1536 [2.2.222/1]"; SQL statement:

select o.filepath from opus o
join (select filename, filesize, count(*) from opus group by filename, filesize having count(*) > 1 ) b
on o.filename = b.filename and o.filesize = b.filesize
order by o.filename [50000-222]
   at org.h2.message.DbException.getJdbcSQLException(DbException.java:566)
   at org.h2.message.DbException.getJdbcSQLException(DbException.java:489)
   at org.h2.message.DbException.get(DbException.java:212)
   at org.h2.message.DbException.convert(DbException.java:407)
   at org.h2.command.Command.executeQuery(Command.java:213)
   at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:131)
   at com.ubooquity.a.a.a.a.c(SourceFile:596)
   at com.ubooquity.a.a.e.f(SourceFile:156)
   at com.ubooquity.a.b.a.h(SourceFile:116)
   at com.ubooquity.a.b.a.b(SourceFile:102)
   at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.h2.mvstore.MVStoreException: Reading from file sun.nio.ch.FileChannelImpl@9f60d0c failed at 189418556 (length -1), read 0, remaining 1536 [2.2.222/1]
   at org.h2.mvstore.DataUtils.newMVStoreException(DataUtils.java:996)
   at org.h2.mvstore.DataUtils.readFully(DataUtils.java:455)
   at org.h2.mvstore.FileStore.readFully(FileStore.java:719)
   at org.h2.mvstore.SingleFileStore.readFully(SingleFileStore.java:60)
   at org.h2.mvstore.SingleFileStore.readFully(SingleFileStore.java:28)
   at org.h2.mvstore.SFChunk.readFully(SFChunk.java:35)
   at org.h2.mvstore.Chunk.readBufferForPage(Chunk.java:465)
   at org.h2.mvstore.FileStore.readPage(FileStore.java:1960)
   at org.h2.mvstore.MVStore.readPage(MVStore.java:1021)
   at org.h2.mvstore.MVMap.readPage(MVMap.java:632)
   at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1117)
   at org.h2.mvstore.Cursor.hasNext(Cursor.java:64)
   at org.h2.mvstore.tx.TransactionMap$CommittedIterator.fetchNext(TransactionMap.java:960)
   at org.h2.mvstore.db.MVPrimaryIndex$MVStoreCursor.next(MVPrimaryIndex.java:440)
   at org.h2.index.IndexCursor.next(IndexCursor.java:292)
   at org.h2.table.TableFilter.next(TableFilter.java:425)
   at org.h2.command.query.Select.gatherGroup(Select.java:517)
   at org.h2.command.query.Select.queryGroup(Select.java:488)
   at org.h2.command.query.Select.queryWithoutCache(Select.java:828)
   at org.h2.command.query.Query.queryWithoutCacheLazyCheck(Query.java:197)
   at org.h2.command.query.Query.query(Query.java:520)
   at org.h2.command.query.Query.query(Query.java:483)
   at org.h2.index.QueryExpressionIndex.find(QueryExpressionIndex.java:269)
   at org.h2.index.QueryExpressionIndex.find(QueryExpressionIndex.java:152)
   at org.h2.index.IndexCursor.find(IndexCursor.java:166)
   at org.h2.table.TableFilter.next(TableFilter.java:394)
   at org.h2.table.TableFilter.next(TableFilter.java:464)
   at org.h2.command.query.Select$LazyResultQueryFlat.fetchNextRow(Select.java:1843)
   at org.h2.result.LazyResult.hasNext(LazyResult.java:78)
   at org.h2.result.FetchedResult.next(FetchedResult.java:34)
   at org.h2.command.query.Select.queryFlat(Select.java:728)
   at org.h2.command.query.Select.queryWithoutCache(Select.java:833)
   at org.h2.command.query.Query.queryWithoutCacheLazyCheck(Query.java:197)
   at org.h2.command.query.Query.query(Query.java:520)
   at org.h2.command.query.Query.query(Query.java:483)
   at org.h2.command.CommandContainer.query(CommandContainer.java:252)
   at org.h2.command.Command.executeQuery(Command.java:192)
... 6 common frames omitted
Caused by: java.nio.channels.ClosedByInterruptException: null
   at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200)
   at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172)
   at java.base/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:992)
   at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:964)
   at org.h2.mvstore.DataUtils.readFully(DataUtils.java:441)
... 41 common frames omitted
20231105 13:55:33 [AWT-EventQueue-0] INFO com.ubooquity.a - Reverse proxy prefix: collection

Here is my system info:

Ubooquity version:3.0.1 built on 2023-10-15 at 17:52
Java version:21.0.1
Java vendor:Oracle Corporation
Java VM name:Java HotSpot(TM) 64-Bit Server VM
OS name:Windows 11
OS version:10.0
OS architecture:amd64
Number of processors:16
Max memory:8176 MB
Free memory:216 MB
Total memory:708 MB

Let me know if there is anything else you need or changes that I should make to my setup to get it working. 

Under review

This one's harder. Working on it...

Thanks let me know if you need anymore details or anything you would like me to try

Since 3.0.0, Ubooquity looks for duplicate files after scanning collections.

This can be a veeeeery long operation on big collections and cause issues like the one you encountered, so I deactivated it by default and added an option (in advanced settings) to re-enable it.

Here is the fixed version (3.0.2) : https://vaemendis.net/ubooquity/downloads/Ubooquity-3.0.2.zip

Let me know how it goes.

My scans show as complete now and I can launch another scan as well. On my library previously it never showed complete and after anywhere between 24-48 hrs after I started a scan the app would crash or my pc would lock up if I didn't stop the server and start again. Did you make any changes to the duplicate detection that you would like me to try out and see if there are any improvements apart from the new flag?

+1

Did you make any changes to the duplicate detection that you would like

me to try out and see if there are any improvements apart from the new

flag?

Thanks, for now I just made the feature optional.

I'll look into optimizations after the release of 3.1.0 (which will be the official Ubooquity 3, not in beta anymore).