CVD exception on loading AE's & unable to run collection reader

[eswidler]

Hi @tmills

I'm running into an error when running the CVD. This issue seems similar to the one brought up in https://github.com/tmills/ctakes-docker/issues/29, with similar symptoms, but the traceback is different. I'm also not able to run the collection reader, maybe for similar reasons. I'm running this locally on Ubuntu 18.04.

I was able to determine my IP through hostname -I. Note that I was originally trying this on my work network with a 172... IP address, but after noticing your comment in the linked issue, I also tried this at home with a 192... IP address, however this yielded the same result.

I set up the docker containers without deid, and verified that both were running.

~/ctakes-docker$ ./bin/runBrokerContainer.sh 
2739bb4746935aec0ffc08404ac1354e84367125486349abff6ba63bc81a612d

~/ctakes-docker$ docker container logs 2739bb4746935aec0ffc08404ac1354e84367125486349abff6ba63bc81a612d
Setting public ip based on broker_host environment variable of 192...
public_ip set to 192...
Certificate stored in file </certificate/broker_cert>
INFO: Loading '/apache-activemq-5.15.2/bin/env'
INFO: Using java '/usr/lib/jvm/java-1.8-openjdk/bin/java'
INFO: Starting in foreground, this is just for debugging purposes (stop process by pressing CTRL+C)
INFO: Creating pidfile /apache-activemq-5.15.2/data/activemq.pid
Java Runtime: IcedTea 1.8.0_212 /usr/lib/jvm/java-1.8-openjdk/jre
  Heap sizes: current=62976k  free=58697k  max=5592576k
    JVM args: -Xms64M -Xmx6G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/apache-activemq-5.15.2/conf/login.config -Dcom.sun.management.jmxremote -Djavax.net.ssl.keyStore=/certificate/broker.ks -Djavax.net.ssl.keyStorePassword=ZjhiYWUwND -Djava.awt.headless=true -Djava.io.tmpdir=/apache-activemq-5.15.2/tmp -Dactivemq.classpath=/apache-activemq-5.15.2/conf:/apache-activemq-5.15.2/../lib/: -Dactivemq.home=/apache-activemq-5.15.2 -Dactivemq.base=/apache-activemq-5.15.2 -Dactivemq.conf=/apache-activemq-5.15.2/conf -Dactivemq.data=/apache-activemq-5.15.2/data
Extensions classpath:
  [/apache-activemq-5.15.2/lib,/apache-activemq-5.15.2/lib/camel,/apache-activemq-5.15.2/lib/optional,/apache-activemq-5.15.2/lib/web,/apache-activemq-5.15.2/lib/extra]
ACTIVEMQ_HOME: /apache-activemq-5.15.2
ACTIVEMQ_BASE: /apache-activemq-5.15.2
ACTIVEMQ_CONF: /apache-activemq-5.15.2/conf
ACTIVEMQ_DATA: /apache-activemq-5.15.2/data
Loading message broker from: xbean:file:/apache-activemq-5.15.2/conf/activemq-nojournal.xml
 INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@5442a311: startup date [Wed Sep 04 21:16:20 GMT 2019]; root of context hierarchy
 INFO | Using Persistence Adapter: MemoryPersistenceAdapter
 INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
 INFO | Apache ActiveMQ 5.15.2 (localhost, ID:2739bb474693-46193-1567631781453-0:1) is starting
 INFO | Listening for connections at: tcp://2739bb474693:61616
 INFO | Connector openwire started
 WARN | [email protected]@6babf3bf{/,null,STARTING} has uncovered http methods for path: /
 INFO | Connector https started
 INFO | Apache ActiveMQ 5.15.2 (localhost, ID:2739bb474693-46193-1567631781453-0:1) started
 INFO | For help or more information please see: http://activemq.apache.org

~/ctakes-docker$ ./bin/runPipelineContainer.sh 
174df79811b7fb19a246f57728fa7b1c1991b5a335996c1262f4e2994f72f61e

~/ctakes-docker$ docker container logs 174df79811b7fb19a246f57728fa7b1c1991b5a335996c1262f4e2994f72f61e
Certificate was added to keystore
>>> Setting defaultBrokerURL to:https://192...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/apache-ctakes-4.0.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/apache-uima-as-2.10.3/apache-activemq/lib/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Service:AggregatePlaintextFastUMLSProcessor using default view Initialized. Ready To Process Messages From Queue:myQueueName
Press 'q'+'Enter' to quiesce and stop the service or 's'+'Enter' to stop it now.
Note: selected option is not echoed on the console.

So it appears to me the docker containers are running fine. Although one concerning thing I noticed was in the AMQ logs where it says Listening for connections at: tcp://2739bb474693:61616, which appears to be using the container ID as the TCP address.

When I launch the CVD and attempt to load the remoteNoDeid.xml AE, I get the following popup: And in the CVD logs, this is the full error in Java:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-ctakes-4.0.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-uima-as-2.9.0/apache-activemq/lib/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
javax.jms.JMSException: java.io.EOFException
	at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
	at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1949)
	at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1968)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
	at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
	at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
	at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:344)
	at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:219)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
	at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
	... 1 more

No logs appear in either the AMQ or Pipeline containers after this exception.

I also tried running the collection reader example (just running it on an unmodified desc/FilesInDirectoryCollectionReader.xml to use the sample dir). Here is the output of that command before it hangs indefinitely:

~/ctakes-docker$ ./bin/runRemoteAsyncAE.sh tcp://192...:61616 mainQueue -d desc/localDeploymentDescriptorNoDeid.xml -c desc/FilesInDirectoryCollectionReader.xml -o xmis/
Attempting to deploy desc/localDeploymentDescriptorNoDeid.xml ...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-ctakes-4.0.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-uima-as-2.9.0/apache-activemq/lib/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Could not refresh JMS Connection for destination '' - silently retrying in 5 ms. Cause: Cannot send, channel has already failed: tcp://192...:61616

I'm not familiar with AMQ, so possibly it's a setup issue there (i.e. the weird TCP listening address). When I go to the AMQ address that the Pipeline is pointed to (https://192...:61616/), I do get a response, I believe indicating that AMQ is up. Do you have any advice here?