sstabledump: Unknown column scylla_cpu_sharding_algorithm during deserialization

[amoskong]

$ rpm -qa|grep scylla scylla-libstdc++73-7.3.1-1.2.el7.centos.x86_64 scylla-conf-3.0.0-0.20190114.37ab553f0.el7.x86_64 scylla-server-3.0.0-0.20190114.37ab553f0.el7.x86_64 scylla-libgcc73-7.3.1-1.2.el7.centos.x86_64 scylla-gdb-7.10.1-30.el7.centos.x86_64 scylla-tools-core-3.0.0-20190114.b9d3384c92.el7.noarch scylla-tools-3.0.0-20190114.b9d3384c92.el7.noarch scylla-libatomic73-7.3.1-1.2.el7.centos.x86_64 scylla-jmx-3.0.0-20190114.18bf97a.el7.noarch scylla-env-1.1-1.el7.noarch scylla-kernel-conf-3.0.0-0.20190114.37ab553f0.el7.x86_64 scylla-3.0.0-0.20190114.37ab553f0.el7.x86_64

• schema attached the schema of keyspace_sst3: sst3_schema.yaml.txt

• workloads

# parepare write workload before upgrade
stress_cmd_sst3_prepare: cassandra-stress user no-warmup profile=/tmp/sst3_schema.yaml  ops'(insert=1)' cl=QUORUM n=10000000 -rate threads=1000 -pop seq=1..10000000

# verify workloads after upgrade
stress_cmd_sst3_verify_read: cassandra-stress user no-warmup profile=/tmp/sst3_schema.yaml  ops'(read1=1)' cl=ALL n=10000000 -rate threads=1000 -pop seq=1..10000000
stress_cmd_sst3_verify_more: cassandra-stress user no-warmup profile=/tmp/sst3_schema.yaml  ops'(read1=1,read2=1,update_static=1,update_ttl=1,update_diff1_ts=1,update_diff2_ts=1,update_same1_ts=1,update_same2_ts=1)' cl=ALL n=10000000 -rate threads=200 -pop seq=1..10000000
stress_cmd_sst3_verify_delete: cassandra-stress user no-warmup profile=/tmp/sst3_schema.yaml  ops'(delete_row=1)' cl=ALL n=1000000 -rate threads=200 -pop seq=1..1000000

• sstabledump error at the begin, the command didn't exit after output error, and continue output normal json text

[scylla-test@rolling-upgrade-l-ak-30-db-node-7a0013b8-0-1 ~]$ for i in `sudo find /var/lib/scylla/data/keyspace_sst3/ -type f`; do echo "sudo sstabledump $i"; sudo sstabledump $i >sstabledump.output 2>sstabledump.err || exit 1; done
sudo sstabledump /var/lib/scylla/data/keyspace_sst3/user_with_ck-9d097050198f11e9b465000000000004/snapshots/1547646512573/la-58-big-Scylla.db

ERROR 03:13:29,845 Error in ThreadPoolExecutor
java.lang.RuntimeException: Unknown column scylla_cpu_sharding_algorithm during deserialization
	at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:321) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:522) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_191]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.0.56.Final.jar:4.0.56.Final]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]
ERROR 03:13:29,857 Error in ThreadPoolExecutor
java.lang.RuntimeException: Unknown column supported_features during deserialization
	at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:321) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:522) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_191]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.0.56.Final.jar:4.0.56.Final]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]
ERROR 03:13:29,857 Error in ThreadPoolExecutor
java.lang.RuntimeException: Unknown column supported_features during deserialization
	at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:321) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:522) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570) ~[apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_191]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3-SNAPSHOT.jar:3.11.3-SNAPSHOT]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.0.56.Final.jar:4.0.56.Final]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]

[amoskong]

The problem still exists when I test with shlomi's fixed stress.jar https://github.com/slivne/scylla-tools-java/commits/slivne_fix_cs_userdefined_retry

[slivne]

my fix is not related to this.

related to https://github.com/scylladb/scylla/issues/4112