java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/tzdb.dat (Structure needs cleaning)

[amoskong]

Loader info

• ami_id_loader: 'ami-063a97bde47353690'

+# loader AMI with c-s ver. 4 and few fixes:
+#  - fix NoSuchElementException
+#  - enable control over both consistency levels: regular and serial
+#  - bring shard awareness
+regions_data:
+  us-east-1:
+    ami_id_loader: 'ami-0b94f0e897d884b1b'
+  eu-west-1:
+    ami_id_loader: 'ami-0ea704a8c9ec08e57'
+  us-west-2:
+    ami_id_loader: 'ami-063a97bde47353690'

Description:

In longevity-5000-tables, we have 5 loader nodes, and many workloads. I saw a cassandra-stress error in the job. The loader uses newer scylla-tools-java.

Job : https://jenkins.scylladb.com/view/scylla-4.1/job/scylla-4.1/job/longevity/job/longevity-5000-tables-newer-loader/1

Test log:

• loader log: https://cloudius-jenkins-test.s3.amazonaws.com/fc556c79-ff14-4516-a23a-9691fa75cc1b/20200618_161136/loader-set-fc556c79.zip
• db log: https://cloudius-jenkins-test.s3.amazonaws.com/fc556c79-ff14-4516-a23a-9691fa75cc1b/20200618_161136/db-cluster-fc556c79.zip

Stress cmd:

cassandra-stress user profile=/tmp/templated_tables_mv4r7sygao.yaml c l=QUORUM 'ops(insert=1, read1=5)' duration=55m -rate threads=2 -errors ignore -node 10.0.136.55

< t:2020-06-18 15:39:58,666 f:remote.py l:306 c:sdcm.remote p:DEBUG > RemoteCmdRunner [[email protected]]: Running command "echo TAG: loader_idx:0-cpu_idx:0-keyspace_idx:1; STRESS_TEST_MARKER=TN9E3LR5H9NAT2RCC697; cassandra-stress user profile=/tmp/templated_tables_mv4r7sygao.yaml cl=QUORUM 'ops(insert=1, read1=5)' duration=55m -rate threads=2 -errors ignore -node 10.0.136.55"...

TAG: loader_idx:0-cpu_idx:0-keyspace_idx:1
mktemp: failed to create directory via template ‘/tmp/tmp.XXXXXXXXXX’: Structure needs cleaning
/usr/share/scylla/apache/cassandra.in.sh: line 32: /cassandra.yaml: Permission denied
Exception in thread "main" java.lang.Error: java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/tzdb.dat (Structure needs cleaning)
        at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:261)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.util.calendar.ZoneInfoFile.<clinit>(ZoneInfoFile.java:251)
        at sun.util.calendar.ZoneInfo.getTimeZone(ZoneInfo.java:589)
        at java.util.TimeZone.getTimeZone(TimeZone.java:560)
        at java.util.TimeZone.setDefaultZone(TimeZone.java:666)
        at java.util.TimeZone.getDefaultRef(TimeZone.java:636)
        at java.util.TimeZone.getDefault(TimeZone.java:625)
        at java.text.SimpleDateFormat.initializeCalendar(SimpleDateFormat.java:657)
        at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:601)
        at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:580)
        at ch.qos.logback.core.util.CachingDateFormatter.<init>(CachingDateFormatter.java:33)
        at ch.qos.logback.classic.layout.TTLLLayout.<init>(TTLLLayout.java:25)
        at ch.qos.logback.classic.BasicConfigurator.configure(BasicConfigurator.java:48)
        at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:164)
        at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:84)
        at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
        at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
        at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
        at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
        at org.apache.cassandra.utils.FBUtilities.<clinit>(FBUtilities.java:75)
        at org.apache.cassandra.stress.Stress.main(Stress.java:59)
Caused by: java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/tzdb.dat (Structure needs cleaning)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:255)
        ... 23 more