ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess for shard shardId-000000000000

[ashokmadi]

I am using the KCL library in Python and logging the information as below:

logging.basicConfig(level=logging.INFO) logger = logging.getLogger(name) logger.info("some logger info message")

From the above code, I am expecting to display the information as INFO in Log file, but it is showing as below:

[multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [INFO:some logger info message] for shard shardId-000000000000 some logger info message

It is logging the message twice. I have looked at the DrainChildSTDERRTask.HandleLineResult handleLine(String line) and understand that it logging to logger and JVM.

@Override protected HandleLineResult<Boolean> handleLine(String line) { log.error("Received error line from subprocess [{}] for shard {}", line, getShardId()); System.err.println(line); return new HandleLineResult<Boolean>(); }

I have two questions:

Why my logger.info is going to DrainChildSTDERRTask class though it is not an error? Is logging twice necessary as it is filling the log quickly when it processes loads of streaming messages?

Not sure if this is an error in the KCL library or I am missing something when setting up the configuration in the python code.

Also in the sample_kcl_app you are writing to the sys.stderr is that the expected behaviour? def log(self, message): sys.stderr.write(message)

It would be great if someone can help with this one!

[ravi0912]

So if you see in this link u can see how multilang handle any line goes to console. So the quick solution for logging is to use sys.stdout.write or print.

[montelius]

Hey! 👋

I know this thread is old but I haven't found a suggested solution for this anywhere, so if someone, like me, comes across this thread looking for an answer I thought it would be valuable for them to know what I eventually did to avoid the double logging (can't guarantee it's a good solution though).

I created a custom log configuration XML file and used the --log-configuration flag of the amazon_kclpy_helper.py script to turn off Logback logging specifically for the DrainChildSTDERRTask class. This means only the System.err.println line in the DrainChildSTDERRTask class will output logs.

Below is the content of the log config file. It also sets the logging level in general to ERROR to turn off the noisy logging from KCL.

<?xml version="1.0" encoding="UTF-8" ?>
<!--
/*
 * Copyright 2019 Amazon.com, Inc. or its affiliates.
 * Licensed under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
  -->
<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %d [%thread] %-5level %logger{36} [%mdc{ShardId:-NONE}] - %msg %n
            </pattern>
        </encoder>
    </appender>

    <root level="WARN">
        <appender-ref ref="CONSOLE" />
    </root>

    <logger
        name="software.amazon.kinesis.multilang.DrainChildSTDERRTask"
        level="OFF"
    />
</configuration>