docker-kafka icon indicating copy to clipboard operation
docker-kafka copied to clipboard

Published 20 hours ago • verified publisher icon ches

Permission denied error

Open panda87 opened this issue 9 years ago • 11 comments

Hi

When I'm trying to run the "More complex local development setup" I'm getting the error below, Someone knows how can I save the data to another location in the file system? i.e the logs folder will be transparent in my /kafka/logs ?

[2015-10-10 19:30:42,236] FATAL [Kafka Server 2], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.io.IOException: Permission denied
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1006)
    at kafka.utils.FileLock.<init>(FileLock.scala:28)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:96)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
    at kafka.log.LogManager.<init>(LogManager.scala:57)
    at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:335)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
    at kafka.Kafka$.main(Kafka.scala:46)
    at kafka.Kafka.main(Kafka.scala)
[2015-10-10 19:30:42,248] INFO [Kafka Server 2], shutting down (kafka.server.KafkaServer)
...

Thanks D.

panda87 avatar Oct 10 '15 22:10 panda87

Are you running Docker on OS X or Windows via docker-machine or boot2docker? In that case you're probably running into this, probably the single worst remaining Docker usability issue for development that I know of:

https://github.com/boot2docker/boot2docker/issues/581

ches avatar Oct 11 '15 21:10 ches

Hi @ches thanks for the quick response. Im running this container on Ubuntu14.04. What do you think can be the problem there?

Thanks

panda87 avatar Oct 12 '15 10:10 panda87

What's the command you're using to run the container, and what are the permissions on the directory that you're mounting for the volume?

ches avatar Oct 13 '15 04:10 ches 

$ mkdir -p kafka-ex/{data,logs} && cd kafka-ex
$ ls -l
total 16
drwxrwxr-x 4  david 4096 Oct 14 10:42 ./
drwxr-xr-x 5 david david 4096 Oct 14 10:42 ../
drwxrwxr-x 2 david david 4096 Oct 14 10:42 data/
drwxrwxr-x 2 david david 4096 Oct 14 10:42 logs/
$ docker run -d \
    --hostname 127.0.0.1 \
    --name kafka \
    --volume ./data:/data --volume ./logs:/logs \
    --publish 9092:9092 --publish 7203:7203 \
    --env KAFKA_ADVERTISED_HOST_NAME=127.0.0.1 --env ZOOKEEPER_IP=EXT_IP \
    ches/kafka

$ docker logs -f kafka
Starting kafka
[2015-10-14 11:10:22,238] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,274] INFO Property advertised.host.name is overridden to 127.0.0.1 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,274] INFO Property advertised.port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,274] INFO Property auto.create.topics.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property auto.leader.rebalance.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property controlled.shutdown.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property default.replication.factor is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,275] INFO Property log.dir is overridden to /data (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property log.dirs is overridden to /data (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property zookeeper.connect is overridden to EXT_IP:2181 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,276] INFO Property zookeeper.connection.timeout.ms is overridden to 10000 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,277] INFO Property zookeeper.session.timeout.ms is overridden to 10000 (kafka.utils.VerifiableProperties)
[2015-10-14 11:10:22,316] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2015-10-14 11:10:22,318] INFO [Kafka Server 0], Connecting to zookeeper on EXT_IP:2181 (kafka.server.KafkaServer)
[2015-10-14 11:10:22,329] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-10-14 11:10:22,336] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:host.name=localhost (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.version=1.7.0_72 (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.home=/usr/lib/jvm/java-7-oracle/jre (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.class.path=:/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/kafka/bin/../examples/build/libs//kafka-examples*.jar:/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/kafka/bin/../clients/build/libs/kafka-clients*.jar:/kafka/bin/../libs/jopt-simple-3.2.jar:/kafka/bin/../libs/kafka-clients-0.8.2.1.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/kafka/bin/../libs/kafka_2.10-0.8.2.1.jar:/kafka/bin/../libs/log4j-1.2.16.jar:/kafka/bin/../libs/lz4-1.2.0.jar:/kafka/bin/../libs/metrics-core-2.2.0.jar:/kafka/bin/../libs/scala-library-2.10.4.jar:/kafka/bin/../libs/slf4j-api-1.7.6.jar:/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/kafka/bin/../libs/snappy-java-1.1.1.6.jar:/kafka/bin/../libs/zkclient-0.3.jar:/kafka/bin/../libs/zookeeper-3.4.6.jar:/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,336] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:os.version=3.13.0-48-generic (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:user.home=/kafka (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,337] INFO Client environment:user.dir=/kafka (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,338] INFO Initiating client connection, connectString=EXT_IP:2181 sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@546ce79d (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,359] INFO Opening socket connection to server EXT_IP/EXT_IP:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-10-14 11:10:22,365] INFO Socket connection established to EXT_IP/EXT_IP:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-10-14 11:10:22,376] INFO Session establishment complete on server EXT_IP/EXT_IP:2181, sessionid = 0x1506602d5790003, negotiated timeout = 10000 (org.apache.zookeeper.ClientCnxn)
[2015-10-14 11:10:22,378] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2015-10-14 11:10:22,438] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.io.IOException: Permission denied
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1006)
    at kafka.utils.FileLock.<init>(FileLock.scala:28)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:96)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
    at kafka.log.LogManager.<init>(LogManager.scala:57)
    at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:335)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
    at kafka.Kafka$.main(Kafka.scala:46)
    at kafka.Kafka.main(Kafka.scala)
[2015-10-14 11:10:22,441] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
[2015-10-14 11:10:22,445] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-10-14 11:10:22,451] INFO Session: 0x1506602d5790003 closed (org.apache.zookeeper.ZooKeeper)
[2015-10-14 11:10:22,451] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2015-10-14 11:10:22,452] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)
[2015-10-14 11:10:22,453] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.io.IOException: Permission denied
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1006)
    at kafka.utils.FileLock.<init>(FileLock.scala:28)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:96)
    at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
    at kafka.log.LogManager.<init>(LogManager.scala:57)
    at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:335)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
    at kafka.Kafka$.main(Kafka.scala:46)
    at kafka.Kafka.main(Kafka.scala)
[2015-10-14 11:10:22,455] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

What do you think?

panda87 avatar Oct 14 '15 11:10 panda87

I haven't tested this approach for awhile, it might take some time before I have a chance to, but I would imagine that a kludgey solution to give you a workaround for now is: chmod 777 data logs.

The longer answer is probably that Docker data volume containers are "the Docker way" to deal with this, but that's ultimately only an indirection. This still boils down to the same root usability needs on the issue I mentioned at first. See also:

http://stackoverflow.com/questions/23544282/what-is-the-best-way-to-manage-permissions-for-docker-shared-volumes https://github.com/docker/docker/issues/7198

Hope that helps for now.

ches avatar Oct 14 '15 13:10 ches

Thanks for the answer, I already tried the 'chmod' command without success. I'll read the links above and I hope this will solve my issue.

panda87 avatar Oct 14 '15 14:10 panda87

@panda87 Did you solve this issue?

SimonMiaou avatar Jan 05 '16 17:01 SimonMiaou

Yes, I set the volume permissions to be ubuntu:ubuntu This is the only way it works

panda87 avatar Jan 05 '16 17:01 panda87

@ches is this suppose to happen when you run on aws ebs?

*edited: I am running it via kubernetes. Tried running as root and also fsGroup 1000 (assuming the newly added "kafka" group is 1000) it returns "kafka.common.KafkaException: Found directory /data/lost+found, 'lost+found' is not in the form of topic-partition"

Any pointers?

MazizEsa avatar Dec 02 '16 10:12 MazizEsa

Sorry for the delayed response @MazizEsa, for anyone else that different lost+found issue has a response on #28.

ches avatar Dec 05 '16 09:12 ches

I was getting this error because /kafka/data was not assigned to the right group/owner. Had to change the group/owner to Kafka and bam! Kafka was up and running.

imjuoy avatar Nov 21 '19 17:11 imjuoy