Avishai Ish-Shalom

yeh that's a good one, but lots of ground to cover. thinking out loud, maybe do a series: 1. modern networking intro - architecture, etc 2. NAT 3. Routed floating...

"DNS peering" is a GCP specific concept afaict. Probably better to cover DNS forwarding and K8s DNS in general. TBH I'm going to give anything which is K8s specific lower...

Details? Latency? Bandwidth? Metrics? Caching/proxies? Latency breakdown: - DNS/name resolution - TCP+TLS handshake - TCP slow start - Header buffering (Head of line blocking) - Body buffering + decompression -...

yeh that's a great topic but quite a large one too. i need to think how to break it down to 15m videos.

Please provide your config On Wed, Jul 29, 2015, 14:51 hellowyf [email protected] wrote: > when logstash write message to hdfs and elasticsearch . elasticsearch's > message is fine.but message in...

you are building the classpath using the catch-all expression `find $HADOOP_HOME/share/hadoop/common/lib/ -name '*.jar'` which includes too many jar, specifically httpcore which is used by manticore. please use a specific list...

`path => "/path/in/hdfs/tweet-%{timestamp}.log"` is probably a bad idea as HDFS is not well suited to many small files. Probably `%{+YYYY.MM.DD-HH}` is better - i assume you are using `%{timestamp}` for...

could you try the following command line? ``` /usr/bin/java -Djava.io.tmpdir=/opt/logstash/shipper/tmp/ -classpath /etc/hadoop/log-deps:/etc/hadoop/log-deps:/etc/hadoop/log-deps/hadoop-hdfs-2.6.0.jar:/etc/hadoop/log-deps/commons-lang-2.6.jar:/etc/hadoop/log-deps/commons-daemon-1.0.13.jar:/etc/hadoop/log-deps/commons-el-1.0.jar:/etc/hadoop/log-deps/commons-logging-1.2.jar:/etc/hadoop/log-deps/hadoop-core-0.20.2.jar:/etc/hadoop/log-deps/commons-logging-1.1.3.jar:/etc/hadoop/log-deps/commons-cli-1.2.jar:/etc/hadoop/log-deps/commons-codec-1.4.jar -Xmx1024M -Xms256M -Xss2048k -Djffi.boot.library.path=/opt/logstash/shipper/vendor/jruby/lib/jni -server -Djava.io.tmpdir=/opt/logstash/shipper/tmp/ -Xbootclasspath/a:/opt/logstash/shipper/vendor/jruby/lib/jruby.jar \ -Djruby.home=/opt/logstash/shipper/vendor/jruby -Djruby.lib=/opt/logstash/shipper/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/shipper/lib/bootstrap/environment.rb logstash/runner.rb agent -f...

I could not reproduce the issue. Can you attach Hadoop's logs? is it possible some other program is holding the lease for this file?

just to make sure, can you paste the exact command line you are using to execute logstash and the logstash configuration?