spark-lucenerdd icon indicating copy to clipboard operation
spark-lucenerdd copied to clipboard

Published 20 hours ago • verified publisher icon zouzias

Improve logging

Open yeikel opened this issue 4 years ago • 2 comments

Is your feature request related to a problem? Please describe. I would be nice to have more logs to understand what is going on with the application. Sometimes it seems stuck.

Even with the log level set to DEBUG there is not a lot of information to trace the progress. For example, consider the following logging configuration :

log4j.logger.org.zouzias=DEBUG, console

And the log messages :

20/01/10 17:03:23 INFO LuceneRDD: Instance is created...
20/01/10 17:03:23 INFO LuceneRDD: Instance is created...
20/01/10 17:03:23 INFO LuceneRDD: Linkage requested
20/01/10 17:03:23 INFO LuceneRDD: Linkage requested
20/01/10 17:03:23 INFO LuceneRDD: Linker method is cartesian
20/01/10 17:03:23 INFO LuceneRDD: Linker method is cartesian
20/01/10 17:03:23 INFO LuceneRDD: Computing top-k linkage per partition
20/01/10 17:03:23 INFO LuceneRDD: Computing top-k linkage per partition
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=5] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=5] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process initiated at 2020-01-10T17:03:24.580-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process initiated at 2020-01-10T17:03:24.580-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process initiated at 2020-01-10T17:03:24.582-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process initiated at 2020-01-10T17:03:24.582-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=5]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=5]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process initiated at 2020-01-10T17:03:24.581-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process initiated at 2020-01-10T17:03:24.581-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process completed at 2020-01-10T17:03:24.626-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process completed at 2020-01-10T17:03:24.626-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process completed at 2020-01-10T17:03:24.626-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process completed at 2020-01-10T17:03:24.626-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process completed at 2020-01-10T17:03:24.628-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process completed at 2020-01-10T17:03:24.628-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process took 0 seconds...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Index readers instantiated successfully
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=7]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=7]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=0]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=0]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=6]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=6]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=4]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=4]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=11]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=11]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=10]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=10]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=9]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=9]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=2]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=2]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=1]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=1]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=8]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=8]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=3]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=3]Indexed 0 documents
20/01/10 17:03:46 INFO LuceneRDDPartition: [partId=5]Indexing process completed at 2020-01-10T17:03:46.639-05:00...
20/01/10 17:03:46 INFO LuceneRDDPartition: [partId=5]Indexing process completed at 2020-01-10T17:03:46.639-05:00...
20/01/10 17:03:46 INFO LuceneRDDPartition: [partId=5]Indexing process took 22 seconds...
20/01/10 17:03:46 INFO LuceneRDDPartition: [partId=5]Indexing process took 22 seconds...
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Closing index writers...
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Closing index writers...
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Instantiating index/facet readers
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Instantiating index/facet readers
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Index readers instantiated successfully
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Index readers instantiated successfully
20/01/10 17:03:49 INFO LuceneRDDPartition: [partId=5]Indexed 897134 documents
20/01/10 17:03:49 INFO LuceneRDDPartition: [partId=5]Indexed 897134 documents

After the documents , are indexed , there is no information about what is happening on the cluster/process. I'd be nice to be able to trace the process even deeper

yeikel avatar Jan 10 '20 22:01 yeikel

How about an info message when the linkage action takes place since linkage can take a lot of time?

I just saw that you are using the "cartesian" linkage method. Does the broadcastjoin approach does not work for you? Cartesian could be very slow if you don't have a lot of resources.

20/01/10 17:03:23 INFO LuceneRDD: Linker method is cartesian

This is related to #190 .

zouzias avatar Jan 13 '20 13:01 zouzias

How about an info message when the linkage action takes place since linkage can take a lot of time?

Could you please clarify this?

I just saw that you are using the "cartesian" linkage method. Does the broadcastjoin approach does not work for you? Cartesian could be very slow if you don't have a lot of resources.

Sadly I can't do this as I need to use the Query Builders with the "link" method and it is not supported as we discussed in pull request #162

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 13) had a not serializable result: org.apache.lucene.search.BooleanQuery
Serialization stack:
	- object not serializable (class: org.apache.lucene.search.BooleanQuery, value: (home_ad:flt home_ad:blk home_ad:fullview home_ad:gdn home_ad:siu home_ad:sai home_ad:wan)~5)
	- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
	- object (class scala.Tuple2, (0,(home_ad:flt home_ad:blk home_ad:fullview home_ad:gdn home_ad:siu home_ad:sai home_ad:wan)~5))
	- element of array (index: 0)

yeikel avatar Jan 13 '20 14:01 yeikel