glu icon indicating copy to clipboard operation
glu copied to clipboard

consolestl.sh restart fails with invalid tmp dir error

Open reversefold opened this issue 11 years ago • 11 comments

Doing a stop then a start works fine.

$ ./console-server/bin/consolectl.sh restart
JAVA_OPTIONS=-Dorg.linkedin.app.name=org.linkedin.glu.console-webapp -Dorg.linkedin.glu.console.config.location=/ig/glu/current/console-server/conf/glu-console-webapp.groovy -Dorg.linkedin.glu.console.keys.dir=/ig/glu/current/console-server/keys -Dorg.linkedin.glu.console.plugins.classpath=/ig/glu/current/console-server/glu/repository/plugins/*.jar -Dorg.linkedin.glu.console.root=/ig/glu/current/console-server -XX:MaxPermSize=128m /ig/glu/current/console-server/jetty-distribution-7.2.2.v20101205/bin/jetty.sh restart
Stopping Jetty: OK
Starting Jetty: STARTED Jetty Tue Feb 12 21:28:52 PST 2013
[Tue Feb 12 21:28:52 jenkins@buildmaster03:/ig/glu/current] 
$ 2013-02-12 21:28:53.326:INFO::Redirecting stderr/stdout to /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/logs/2013_02_13.stderrout.log

[Tue Feb 12 21:28:58 jenkins@buildmaster03:/ig/glu/current] 
$ 

console-server/jetty-distribution-7.2.2.v20101205/logs/2013_02_13.stderrout.log

2013-02-12 21:28:53.475:INFO::jetty-7.2.2.v20101205
2013-02-12 21:28:53.498:INFO::Deployment monitor /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/webapps at interval 1
2013-02-12 21:28:53.502:INFO::Deployment monitor /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/contexts at interval 1
2013-02-12 21:28:53.503:INFO::Deployable added: /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/contexts/glu-jetty-context.xml
2013-02-12 21:28:53.550:WARN::Config error at  java.lang.NoSuchMethodException: class org.eclipse.jetty.server.handler.ContextHandl
er.setPort(class java.lang.String)
2013-02-12 21:28:53.550:WARN::Unable to reach node goal: started
java.lang.NoSuchMethodException: class org.eclipse.jetty.server.handler.ContextHandler.setPort(class java.lang.String)
        at org.eclipse.jetty.xml.XmlConfiguration.set(XmlConfiguration.java:464)
        at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:263)
        at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:229)
        at org.eclipse.jetty.deploy.providers.ContextProvider.createContextHandler(ContextProvider.java:73)
        at org.eclipse.jetty.deploy.App.getContextHandler(App.java:97)
        at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:33)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:180)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:481)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:137)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:137)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:50)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:470)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:382)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:339)
        at org.eclipse.jetty.util.Scanner.start(Scanner.java:275)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:114)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:542)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:214)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.server.Server.doStart(Server.java:226)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1061)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:994)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:477)
        at org.eclipse.jetty.start.Main.start(Main.java:623)
        at org.eclipse.jetty.start.Main.parseCommandLine(Main.java:273)
        at org.eclipse.jetty.start.Main.main(Main.java:81)
2013-02-12 21:28:53.552:INFO::Deployable added: /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/contexts/console-jetty-context.xml
2013-02-12 21:28:53.585:WARN::Config error at /tmp java.lang.reflect.InvocationTargetException
2013-02-12 21:28:53.585:WARN::Unable to reach node goal: started
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.eclipse.jetty.xml.XmlConfiguration.set(XmlConfiguration.java:446)
        at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:263)
        at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:229)
        at org.eclipse.jetty.deploy.providers.ContextProvider.createContextHandler(ContextProvider.java:73)
        at org.eclipse.jetty.deploy.App.getContextHandler(App.java:97)
        at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:33)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:180)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:481)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:137)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:137)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:50)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:470)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:382)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:339)
        at org.eclipse.jetty.util.Scanner.start(Scanner.java:275)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:114)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:542)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:214)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.server.Server.doStart(Server.java:226)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1061)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:994)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:477)
        at org.eclipse.jetty.start.Main.start(Main.java:623)
        at org.eclipse.jetty.start.Main.parseCommandLine(Main.java:273)
        at org.eclipse.jetty.start.Main.main(Main.java:81)
Caused by: 
java.lang.IllegalArgumentException: Bad temp directory: /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/null/tmp
        at org.eclipse.jetty.webapp.WebAppContext.setTempDirectory(WebAppContext.java:1077)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.eclipse.jetty.xml.XmlConfiguration.set(XmlConfiguration.java:446)
        at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:263)
        at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:229)
        at org.eclipse.jetty.deploy.providers.ContextProvider.createContextHandler(ContextProvider.java:73)
        at org.eclipse.jetty.deploy.App.getContextHandler(App.java:97)
        at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:33)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:180)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:481)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:137)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:137)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:50)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:470)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:382)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:339)
        at org.eclipse.jetty.util.Scanner.start(Scanner.java:275)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:114)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:542)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:214)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.server.Server.doStart(Server.java:226)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55)
        at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1061)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:994)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:477)        at org.eclipse.jetty.start.Main.start(Main.java:623)        at org.eclipse.jetty.start.Main.parseCommandLine(Main.java:273)        at org.eclipse.jetty.start.Main.main(Main.java:81)2013-02-12 21:28:53.649:INFO::Started [email protected]:8085

reversefold avatar Feb 13 '13 05:02 reversefold

Could you try with glu 4.7.0 which uses a different version of jetty (and a lot of upgraded libraries)?

ypujante avatar Apr 14 '13 21:04 ypujante

Still broken. :-(

reversefold avatar Apr 15 '13 20:04 reversefold

Still broken. :-(

On Sun, Apr 14, 2013 at 2:43 PM, Yan Pujante [email protected]:

Could you try with glu 4.7.0 which uses a different version of jetty (and a lot of upgraded libraries)?

— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/glu/issues/202#issuecomment-16360208 .

Justin Patrin Developer Extraordinaire

reversefold avatar Apr 15 '13 20:04 reversefold

is stop and start working? Is it only restart that is failing?

ypujante avatar Apr 15 '13 20:04 ypujante

That is correct.

On Mon, Apr 15, 2013 at 1:53 PM, Yan Pujante [email protected]:

is stop and start working? Is it only restart that is failing?

— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/glu/issues/202#issuecomment-16410768 .

Justin Patrin Developer Extraordinaire

reversefold avatar Apr 15 '13 22:04 reversefold

I am not a big guru in shell/bash coding. From what I can tell, when you invoke restart it "simply" calls stop and start. The issue is that by this time there is no more JAVA_OPTIONS in the environment. I tried to change consolectl.sh and do

export JAVA_OPTIONS="$JAVA_OPTIONS $JVM_SIZE -Dorg.linkedin.app.name=org.linkedin.glu.console-webapp -Dorg.linkedin.glu.console.config.location=$BASEDIR/conf/glu-console-webapp.groovy -Dorg.linkedin.glu.console.keys.dir=$BASEDIR/keys -Dorg.linkedin.glu.console.plugins.classpath=$PLUGINS_CLASSPATH -Dorg.linkedin.glu.console.root=$BASEDIR"

so that it would be passed down to subprocesses but somehow it did not have any effect. The reason for the bad temp directory is because somehow this JAVA_OPTIONS variable is not available in the subprocesses.

consolectl.sh is controlled by glu and delegates to jetty.sh which comes bundled with jetty.

At this stage I am requesting help on what to do in consolectl.sh to fix this issue as I am unclear how to do it.

Also I wanted to point out that restart is a bit flaky anyway (in jetty.sh) because it calls stop then start right away without every waiting for the server to actually be down which I don't think is very good.

ypujante avatar Apr 17 '13 01:04 ypujante

I think the root cause is that jetty.sh is expecting JAVA_OPTIONS to be an array object and not a string object. Working on a fix.

sodul avatar Jul 17 '13 23:07 sodul

For reference: http://tldp.org/LDP/abs/html/arrays.html

sodul avatar Jul 17 '13 23:07 sodul

So the bug is in jetty.sh, the reason is related to array variables and how restart is implemented.

When doing restart, jetty.sh will call itself in a new shell, the problem is that because it changes JAVA_OPTIONS into an array type, it no longer get passed along.

A potential fix is to update restart to do this: JAVA_OPTIONS="$JAVA_OPTIONS" "$JETTY_SH" stop "$@" JAVA_OPTIONS="$JAVA_OPTIONS" "$JETTY_SH" start "$@"

sodul avatar Jul 18 '13 00:07 sodul

Out of curiosity, do you know if the stop command is synchronous or asynchronous? If async, then I could see an issue in doing start right away as it may take a long time for a webapp to shutdown (I remember those days when working at LinkedIn and shutting down the graph service was writing a cache to the disk.... 20mns to shutdown!)

In which case what works is:

stop wait for stop start

ypujante avatar Jul 18 '13 04:07 ypujante

The stop command is written to be blocking for about 30s: tries 30 times, sleep 1s between each check, then do a kill -KILL. Most of the time this is fine since the console will shut down rather quickly as long as it uses a proper DB.

sodul avatar Jul 18 '13 17:07 sodul