glu
glu copied to clipboard
consolestl.sh restart fails with invalid tmp dir error
Doing a stop then a start works fine.
$ ./console-server/bin/consolectl.sh restart JAVA_OPTIONS=-Dorg.linkedin.app.name=org.linkedin.glu.console-webapp -Dorg.linkedin.glu.console.config.location=/ig/glu/current/console-server/conf/glu-console-webapp.groovy -Dorg.linkedin.glu.console.keys.dir=/ig/glu/current/console-server/keys -Dorg.linkedin.glu.console.plugins.classpath=/ig/glu/current/console-server/glu/repository/plugins/*.jar -Dorg.linkedin.glu.console.root=/ig/glu/current/console-server -XX:MaxPermSize=128m /ig/glu/current/console-server/jetty-distribution-7.2.2.v20101205/bin/jetty.sh restart Stopping Jetty: OK Starting Jetty: STARTED Jetty Tue Feb 12 21:28:52 PST 2013 [Tue Feb 12 21:28:52 jenkins@buildmaster03:/ig/glu/current] $ 2013-02-12 21:28:53.326:INFO::Redirecting stderr/stdout to /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/logs/2013_02_13.stderrout.log [Tue Feb 12 21:28:58 jenkins@buildmaster03:/ig/glu/current] $
console-server/jetty-distribution-7.2.2.v20101205/logs/2013_02_13.stderrout.log
2013-02-12 21:28:53.475:INFO::jetty-7.2.2.v20101205 2013-02-12 21:28:53.498:INFO::Deployment monitor /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/webapps at interval 1 2013-02-12 21:28:53.502:INFO::Deployment monitor /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/contexts at interval 1 2013-02-12 21:28:53.503:INFO::Deployable added: /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/contexts/glu-jetty-context.xml 2013-02-12 21:28:53.550:WARN::Config error atjava.lang.NoSuchMethodException: class org.eclipse.jetty.server.handler.ContextHandl er.setPort(class java.lang.String) 2013-02-12 21:28:53.550:WARN::Unable to reach node goal: started java.lang.NoSuchMethodException: class org.eclipse.jetty.server.handler.ContextHandler.setPort(class java.lang.String) at org.eclipse.jetty.xml.XmlConfiguration.set(XmlConfiguration.java:464) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:263) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:229) at org.eclipse.jetty.deploy.providers.ContextProvider.createContextHandler(ContextProvider.java:73) at org.eclipse.jetty.deploy.App.getContextHandler(App.java:97) at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:33) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:180) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:481) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:137) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:137) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:50) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:470) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:382) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:339) at org.eclipse.jetty.util.Scanner.start(Scanner.java:275) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:114) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:542) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:214) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) at org.eclipse.jetty.server.Server.doStart(Server.java:226) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1061) at java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:994) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.eclipse.jetty.start.Main.invokeMain(Main.java:477) at org.eclipse.jetty.start.Main.start(Main.java:623) at org.eclipse.jetty.start.Main.parseCommandLine(Main.java:273) at org.eclipse.jetty.start.Main.main(Main.java:81) 2013-02-12 21:28:53.552:INFO::Deployable added: /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/contexts/console-jetty-context.xml 2013-02-12 21:28:53.585:WARN::Config error at java.lang.reflect.InvocationTargetException 2013-02-12 21:28:53.585:WARN::Unable to reach node goal: started java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.eclipse.jetty.xml.XmlConfiguration.set(XmlConfiguration.java:446) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:263) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:229) at org.eclipse.jetty.deploy.providers.ContextProvider.createContextHandler(ContextProvider.java:73) at org.eclipse.jetty.deploy.App.getContextHandler(App.java:97) at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:33) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:180) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:481) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:137) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:137) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:50) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:470) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:382) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:339) at org.eclipse.jetty.util.Scanner.start(Scanner.java:275) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:114) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:542) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:214) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) at org.eclipse.jetty.server.Server.doStart(Server.java:226) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1061) at java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:994) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.eclipse.jetty.start.Main.invokeMain(Main.java:477) at org.eclipse.jetty.start.Main.start(Main.java:623) at org.eclipse.jetty.start.Main.parseCommandLine(Main.java:273) at org.eclipse.jetty.start.Main.main(Main.java:81) Caused by: java.lang.IllegalArgumentException: Bad temp directory: /ig/glu/org.linkedin.glu.packaging-all-4.6.2/console-server/jetty-distribution-7.2.2.v20101205/null/tmp at org.eclipse.jetty.webapp.WebAppContext.setTempDirectory(WebAppContext.java:1077) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.eclipse.jetty.xml.XmlConfiguration.set(XmlConfiguration.java:446) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:263) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:229) at org.eclipse.jetty.deploy.providers.ContextProvider.createContextHandler(ContextProvider.java:73) at org.eclipse.jetty.deploy.App.getContextHandler(App.java:97) at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:33) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:180) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:481) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:137) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:137) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:50) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:470) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:382) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:339) at org.eclipse.jetty.util.Scanner.start(Scanner.java:275) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:114) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:542) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:214) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) at org.eclipse.jetty.server.Server.doStart(Server.java:226) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1061) at java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:994) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.eclipse.jetty.start.Main.invokeMain(Main.java:477) at org.eclipse.jetty.start.Main.start(Main.java:623) at org.eclipse.jetty.start.Main.parseCommandLine(Main.java:273) at org.eclipse.jetty.start.Main.main(Main.java:81)2013-02-12 21:28:53.649:INFO::Started [email protected]:8085 /tmp
Could you try with glu 4.7.0 which uses a different version of jetty (and a lot of upgraded libraries)?
Still broken. :-(
Still broken. :-(
On Sun, Apr 14, 2013 at 2:43 PM, Yan Pujante [email protected]:
Could you try with glu 4.7.0 which uses a different version of jetty (and a lot of upgraded libraries)?
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/glu/issues/202#issuecomment-16360208 .
Justin Patrin Developer Extraordinaire
is stop and start working? Is it only restart that is failing?
That is correct.
On Mon, Apr 15, 2013 at 1:53 PM, Yan Pujante [email protected]:
is stop and start working? Is it only restart that is failing?
— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/glu/issues/202#issuecomment-16410768 .
Justin Patrin Developer Extraordinaire
I am not a big guru in shell/bash coding. From what I can tell, when you invoke restart it "simply" calls stop and start. The issue is that by this time there is no more JAVA_OPTIONS
in the environment. I tried to change consolectl.sh
and do
export JAVA_OPTIONS="$JAVA_OPTIONS $JVM_SIZE -Dorg.linkedin.app.name=org.linkedin.glu.console-webapp -Dorg.linkedin.glu.console.config.location=$BASEDIR/conf/glu-console-webapp.groovy -Dorg.linkedin.glu.console.keys.dir=$BASEDIR/keys -Dorg.linkedin.glu.console.plugins.classpath=$PLUGINS_CLASSPATH -Dorg.linkedin.glu.console.root=$BASEDIR"
so that it would be passed down to subprocesses but somehow it did not have any effect. The reason for the bad temp directory is because somehow this JAVA_OPTIONS
variable is not available in the subprocesses.
consolectl.sh
is controlled by glu and delegates to jetty.sh
which comes bundled with jetty.
At this stage I am requesting help on what to do in consolectl.sh
to fix this issue as I am unclear how to do it.
Also I wanted to point out that restart
is a bit flaky anyway (in jetty.sh
) because it calls stop
then start
right away without every waiting for the server to actually be down which I don't think is very good.
I think the root cause is that jetty.sh
is expecting JAVA_OPTIONS
to be an array object and not a string object. Working on a fix.
For reference: http://tldp.org/LDP/abs/html/arrays.html
So the bug is in jetty.sh
, the reason is related to array variables and how restart
is implemented.
When doing restart
, jetty.sh
will call itself in a new shell, the problem is that because it changes JAVA_OPTIONS
into an array type, it no longer get passed along.
A potential fix is to update restart
to do this:
JAVA_OPTIONS="$JAVA_OPTIONS" "$JETTY_SH" stop "$@" JAVA_OPTIONS="$JAVA_OPTIONS" "$JETTY_SH" start "$@"
Out of curiosity, do you know if the stop command is synchronous or asynchronous? If async, then I could see an issue in doing start right away as it may take a long time for a webapp to shutdown (I remember those days when working at LinkedIn and shutting down the graph service was writing a cache to the disk.... 20mns to shutdown!)
In which case what works is:
stop wait for stop start
The stop command is written to be blocking for about 30s: tries 30 times, sleep 1s between each check, then do a kill -KILL
. Most of the time this is fine since the console will shut down rather quickly as long as it uses a proper DB.