sstabledump doesn't work if run as user scylla

[tarzanek]

as user scylla I run sstabledump mytable_map-ka-12345-Data.db -e

but you get

Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.apache.cassandra.cql3.restrictions.StatementRestrictions.<init>(StatementRestrictions.java:166)
    at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:917)
    at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:864)
    at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:852)
    at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:520)
    at org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
    at org.apache.cassandra.cql3.QueryProcessor.executeOnceInternal(QueryProcessor.java:337)
    at org.apache.cassandra.schema.LegacySchemaMigrator.query(LegacySchemaMigrator.java:1043)
    at org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:197)
    at org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:96)
    at org.apache.cassandra.schema.LegacySchemaMigrator.load(LegacySchemaMigrator.java:87)
    at org.apache.cassandra.config.Schema.loadFromDiskForTool(Schema.java:144)
    at com.scylladb.tools.SSTableExport.metadataFromSSTable(SSTableExport.java:26)
    at org.apache.cassandra.tools.SSTableExport.run(SSTableExport.java:192)
    at com.scylladb.tools.SSTableExport.main(SSTableExport.java:44)
Caused by: java.lang.IllegalStateException: Failed to mkdirs /hints; unable to start server
    at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:867)
    at org.apache.cassandra.db.Keyspace.<clinit>(Keyspace.java:74)
    ... 15 more

workaround is to run above command as root (see https://github.com/scylladb/scylla/issues/1045 )

but it will create /hints and /saved_caches

Ideally if it can be overriden using -Dcassandra.storagedir (set in cassandra-env.sh) https://github.com/scylladb/scylla-tools-java/blob/branch-3.0/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L440

[tarzanek]

OK even better workaround is: cassandra_storagedir="/tmp" sstabledump mytable_map-ka-12345-Data.db -e which will override needed access to / just for creating dummy dirs

[syuu1228]

Does not able to reproduce on my side.

[tarzanek]

sorry, forgot to mention, that you need to switch user scylla to a normal user (which some users and customers do), resp. unlock it

[tarzanek]

not saying we need to fix it, but at least this is indexed now and people can find a quick workaround for the command here

[dyasny]

Just ran into this in ticket 1094

[dyasny]

@syuu1228 if you need help reproducing we can assist with that.

I wonder if the best solution would be to simply push cassandra_storagedir="/tmp" into the users' .bash_profile

[vladzcloudius]

@dyasny Why not to fix this in cassandra-env.sh?

[vladzcloudius]

@slivne @eliransin FYI

[eliransin]

@tarzanek it seams that you run the command from the wrong place. Did you try to run it from the Data directory instead of from the actual table directory? I mean: got to the data directory and run: sstabledump ./<keyspace dir>/<table dir>/mytable_map-ka-12345-Data.db -e

Can you give it a try and see if it solves it for you? It works for master but not for enterprise branch. Which version are you using? If this is a required functionality maybe we should backport the fixes that eliminate this error because all approaches (sudoing,data_dirs=tmp,changing cassandra-env.sh) are workarounds in some sense

[tarzanek]

@dyasny - where is the customer running this from? in my old case I don't recall anymore

but anyways @eliransin the error message should say to run this in data dir and not fail with a weird stack

[eliransin]

it is also seems to be resolved in the latest version. If you run it from the wrong place, then it complains about cassandra_storagedir not being configured correctly. My guess (unverified) is that if cassandra_storagedir is not configured, the newer versions sets pwd as the default cassandra_storagedir. We can verify all of that though.

[dyasny]

They have their own user called xdeploy, under which they login as well as run scylla.

I tried to reproduce this internally using the regular scylla:scylla user and changing the user's /etc/passwd line from nologin to /bin/bash. The issue did not reproduce.

In this setup, I guess the user settings are a bit stricter, but the workaround (export the var first) works for them. This is environmental for certain, but it would be good to simply not have these problems in the future and stop relying on environmental quirks

[dyasny]

@eliransin they are on 2019.1.5

[eliransin]

@dyasny IMO we need to either define a clear action item or close this issue. It is not clear even if we want to allow this with other users. According to @slivne (that will correct me if I'm wrong :smile: ) we don't want to allow this. So some possible action item can be to have a more clear documentation about how to use those tools.

[dyasny]

I have no problem with a solution that says "If you aren't under the scylla:scylla or root:root user, please run this command with a specified storagedir variable + an example" in the docs for this utility

On Sun, Mar 22, 2020 at 10:30 AM Eliran Sinvani [email protected] wrote:

@dyasny https://github.com/dyasny IMO we need to either define a clear action item or close this issue. It is not clear even if we want to allow this with other users. According to @slivne https://github.com/slivne (that will correct me if I'm wrong 😄 ) we don't want to allow this. So some possible action item can be to have a more clear documentation about how to use those tools.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scylladb/scylla-tools-java/issues/118#issuecomment-602211913, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGAL5J6I65XBOK3ZGHZAE3RIYOH7ANCNFSM4I5RRCXA .