incubator-heron icon indicating copy to clipboard operation
incubator-heron copied to clipboard

Published 20 hours ago • verified publisher icon apache

Helm deployment break - WIP

Open windhamwong opened this issue 3 years ago • 2 comments

I create this issue for tracking the issue of, deploying Heron to Kubernetes with Helm, having issue with zookeeper. Please feel free to leave your related case and log here.

TODO: I am trying to solve the issue if this is a known issue.

The log of zookeeper is listed as below:

Validating enviornment
ZK_REPLICAS=3
MY_ID=1
ZK_LOG_LEVEL=INFO
ZK_DATA_DIR=/var/lib/zookeeper/data
ZK_DATA_LOG_DIR=/var/lib/zookeeper/log
ZK_LOG_DIR=var/log/zookeeper
ZK_CLIENT_PORT=2181
ZK_SERVER_PORT=2888
ZK_ELECTION_PORT=3888
ZK_TICK_TIME=2000
ZK_INIT_LIMIT=10
ZK_SYNC_LIMIT=5
ZK_MAX_CLIENT_CNXNS=60
ZK_MIN_SESSION_TIMEOUT=4000
ZK_MAX_SESSION_TIMEOUT=40000
ZK_HEAP_SIZE=256M
ZK_SNAP_RETAIN_COUNT=3
ZK_PURGE_INTERVAL=1
ENSEMBLE
server.1=heron-zookeeper-0.heron-zookeeper-headless.default.svc.cluster.local:2888:3888
server.2=heron-zookeeper-1.heron-zookeeper-headless.default.svc.cluster.local:2888:3888
server.3=heron-zookeeper-2.heron-zookeeper-headless.default.svc.cluster.local:2888:3888
Enviorment validation successful
rm: cannot remove '/opt/zookeeper/conf/zookeeper.conf': Permission denied
Creating ZooKeeper configuration
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 92: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 93: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 94: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 95: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 96: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 97: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 98: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 99: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 100: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 101: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 102: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 103: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 104: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 105: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 108: /opt/zookeeper/conf/zookeeper.conf: Permission denied
Wrote ZooKeeper configuration file to /opt/zookeeper/conf/zookeeper.conf
rm: cannot remove '/opt/zookeeper/conf/log4j.properties': Permission denied
Creating ZooKeeper log4j configuration
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 138: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 139: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 140: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 141: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 142: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 143: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 144: /opt/zookeeper/conf/log4j.properties: Permission denied
Wrote log4j configuration to /opt/zookeeper/conf/log4j.properties
Creating ZooKeeper data directories and setting permissions
mkdir: cannot create directory ‘var’: Permission denied
chown: invalid user: ‘zookeeper:zookeeper’
Created ZooKeeper data directories and set permissions in /var/lib/zookeeper/data
Creating JVM configuration file
Wrote JVM configuration to /opt/zookeeper/conf/java.env
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 151: /opt/zookeeper/conf/java.env: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 152: /opt/zookeeper/conf/java.env: Permission denied
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/conf/zookeeper.conf
mkdir: cannot create directory ‘data’: Permission denied
mkdir: cannot create directory ‘/opt/zookeeper/bin/../logs’: Permission denied
2021-10-18 10:22:57,409 [myid:] - INFO  [main:QuorumPeerConfig@135] - Reading configuration from: /opt/zookeeper/conf/zookeeper.conf
2021-10-18 10:22:57,415 [myid:] - WARN  [main:VerifyingFileFactory@59] - data/zookeeper is relative. Prepend ./ to indicate that you're sure!
2021-10-18 10:22:57,425 [myid:] - INFO  [main:QuorumPeerConfig@387] - clientPortAddress is 0.0.0.0:2181
2021-10-18 10:22:57,426 [myid:] - INFO  [main:QuorumPeerConfig@391] - secureClientPort is not set
2021-10-18 10:22:57,432 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2021-10-18 10:22:57,433 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 1
2021-10-18 10:22:57,434 [myid:] - WARN  [main:QuorumPeerMain@125] - Either no config or no quorum defined in config, running  in standalone mode
2021-10-18 10:22:57,439 [myid:] - INFO  [main:ManagedUtil@45] - Log4j 1.2 jmx support found and enabled.
2021-10-18 10:22:57,439 [myid:] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2021-10-18 10:22:57,455 [myid:] - INFO  [main:QuorumPeerConfig@135] - Reading configuration from: /opt/zookeeper/conf/zookeeper.conf
2021-10-18 10:22:57,459 [myid:] - WARN  [main:VerifyingFileFactory@59] - data/zookeeper is relative. Prepend ./ to indicate that you're sure!
2021-10-18 10:22:57,464 [myid:] - INFO  [PurgeTask:FileTxnSnapLog@115] - zookeeper.snapshot.trust.empty : false
2021-10-18 10:22:57,494 [myid:] - INFO  [main:QuorumPeerConfig@387] - clientPortAddress is 0.0.0.0:2181
2021-10-18 10:22:57,494 [myid:] - INFO  [main:QuorumPeerConfig@391] - secureClientPort is not set
2021-10-18 10:22:57,495 [myid:] - INFO  [main:ZooKeeperServerMain@117] - Starting server
2021-10-18 10:22:57,495 [myid:] - ERROR [PurgeTask:DatadirCleanupManager$PurgeTask@142] - Error occurred while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:127)
        at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:79)
        at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
        at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
        at java.base/java.util.TimerThread.run(Timer.java:506)
2021-10-18 10:22:57,499 [myid:] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
2021-10-18 10:22:57,495 [myid:] - INFO  [main:FileTxnSnapLog@115] - zookeeper.snapshot.trust.empty : false
2021-10-18 10:22:57,499 [myid:] - ERROR [main:ZooKeeperServerMain@75] - Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:127)
        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:124)
        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Unable to access datadir, exiting abnormally

windhamwong avatar Oct 18 '21 10:10 windhamwong

Which Heron image were you using? How was the image built?

On Mon, Oct 18, 2021, 6:31 AM Windham Wong @.***> wrote:

I create this issue for tracking the issue of, deploying Heron to Kubernetes with Helm, having issue with zookeeper. Please feel free to leave your related case and log here.

TODO: I am trying to solve the issue if this is a known issue.

The log of zookeeper is listed as below:

Validating enviornment

ZK_REPLICAS=3

MY_ID=1

ZK_LOG_LEVEL=INFO

ZK_DATA_DIR=/var/lib/zookeeper/data

ZK_DATA_LOG_DIR=/var/lib/zookeeper/log

ZK_LOG_DIR=var/log/zookeeper

ZK_CLIENT_PORT=2181

ZK_SERVER_PORT=2888

ZK_ELECTION_PORT=3888

ZK_TICK_TIME=2000

ZK_INIT_LIMIT=10

ZK_SYNC_LIMIT=5

ZK_MAX_CLIENT_CNXNS=60

ZK_MIN_SESSION_TIMEOUT=4000

ZK_MAX_SESSION_TIMEOUT=40000

ZK_HEAP_SIZE=256M

ZK_SNAP_RETAIN_COUNT=3

ZK_PURGE_INTERVAL=1

ENSEMBLE

server.1=heron-zookeeper-0.heron-zookeeper-headless.default.svc.cluster.local:2888:3888

server.2=heron-zookeeper-1.heron-zookeeper-headless.default.svc.cluster.local:2888:3888

server.3=heron-zookeeper-2.heron-zookeeper-headless.default.svc.cluster.local:2888:3888

Enviorment validation successful

rm: cannot remove '/opt/zookeeper/conf/zookeeper.conf': Permission denied

Creating ZooKeeper configuration

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 92: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 93: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 94: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 95: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 96: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 97: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 98: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 99: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 100: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 101: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 102: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 103: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 104: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 105: /opt/zookeeper/conf/zookeeper.conf: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 108: /opt/zookeeper/conf/zookeeper.conf: Permission denied

Wrote ZooKeeper configuration file to /opt/zookeeper/conf/zookeeper.conf

rm: cannot remove '/opt/zookeeper/conf/log4j.properties': Permission denied

Creating ZooKeeper log4j configuration

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 138: /opt/zookeeper/conf/log4j.properties: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 139: /opt/zookeeper/conf/log4j.properties: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 140: /opt/zookeeper/conf/log4j.properties: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 141: /opt/zookeeper/conf/log4j.properties: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 142: /opt/zookeeper/conf/log4j.properties: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 143: /opt/zookeeper/conf/log4j.properties: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 144: /opt/zookeeper/conf/log4j.properties: Permission denied

Wrote log4j configuration to /opt/zookeeper/conf/log4j.properties

Creating ZooKeeper data directories and setting permissions

mkdir: cannot create directory ‘var’: Permission denied

chown: invalid user: ‘zookeeper:zookeeper’

Created ZooKeeper data directories and set permissions in /var/lib/zookeeper/data

Creating JVM configuration file

Wrote JVM configuration to /opt/zookeeper/conf/java.env

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 151: /opt/zookeeper/conf/java.env: Permission denied

/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 152: /opt/zookeeper/conf/java.env: Permission denied

ZooKeeper JMX enabled by default

Using config: /opt/zookeeper/conf/zookeeper.conf

mkdir: cannot create directory ‘data’: Permission denied

mkdir: cannot create directory ‘/opt/zookeeper/bin/../logs’: Permission denied

2021-10-18 10:22:57,409 [myid:] - INFO @.*** - Reading configuration from: /opt/zookeeper/conf/zookeeper.conf

2021-10-18 10:22:57,415 [myid:] - WARN @.*** - data/zookeeper is relative. Prepend ./ to indicate that you're sure!

2021-10-18 10:22:57,425 [myid:] - INFO @.*** - clientPortAddress is 0.0.0.0:2181

2021-10-18 10:22:57,426 [myid:] - INFO @.*** - secureClientPort is not set

2021-10-18 10:22:57,432 [myid:] - INFO @.*** - autopurge.snapRetainCount set to 3

2021-10-18 10:22:57,433 [myid:] - INFO @.*** - autopurge.purgeInterval set to 1

2021-10-18 10:22:57,434 [myid:] - WARN @.*** - Either no config or no quorum defined in config, running in standalone mode

2021-10-18 10:22:57,439 [myid:] - INFO @.*** - Log4j 1.2 jmx support found and enabled.

2021-10-18 10:22:57,439 [myid:] - INFO @.*** - Purge task started.

2021-10-18 10:22:57,455 [myid:] - INFO @.*** - Reading configuration from: /opt/zookeeper/conf/zookeeper.conf

2021-10-18 10:22:57,459 [myid:] - WARN @.*** - data/zookeeper is relative. Prepend ./ to indicate that you're sure!

2021-10-18 10:22:57,464 [myid:] - INFO @.*** - zookeeper.snapshot.trust.empty : false

2021-10-18 10:22:57,494 [myid:] - INFO @.*** - clientPortAddress is 0.0.0.0:2181

2021-10-18 10:22:57,494 [myid:] - INFO @.*** - secureClientPort is not set

2021-10-18 10:22:57,495 [myid:] - INFO @.*** - Starting server

2021-10-18 10:22:57,495 [myid:] - ERROR @.*** - Error occurred while purging.

org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2

    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:127)

    at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:79)

    at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)

    at java.base/java.util.TimerThread.mainLoop(Timer.java:556)

    at java.base/java.util.TimerThread.run(Timer.java:506)

2021-10-18 10:22:57,499 [myid:] - INFO @.*** - Purge task completed.

2021-10-18 10:22:57,495 [myid:] - INFO @.*** - zookeeper.snapshot.trust.empty : false

2021-10-18 10:22:57,499 [myid:] - ERROR @.*** - Unable to access datadir, exiting abnormally

org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2

    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:127)

    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:124)

    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)

    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)

    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)

    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)

Unable to access datadir, exiting abnormally

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-heron/issues/3719, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABMNKQGMHESA3A6JGI47MTUHPZPJANCNFSM5GGJBPYA .

nicknezis avatar Oct 18 '21 11:10 nicknezis

Confirmed that this happens when using helm, setting .Values.zookeeper.security := true.

windhamwong avatar Oct 22 '21 02:10 windhamwong