Exception "lost connection with Eclipse" when leaving Jetty running overnight and coming back the next morning

[GoogleCodeExporter]

What is the expected output? What do you see instead?

I upgraded to the latest nightly build (1.3.3.201301020723) and now when I 
leave Jetty 8 running overnight and I come back the next morning, I see the 
exception "lost connection with Eclipse , shutting down.".  This is a 
regression from the previous build with Jetty 8.

What steps will reproduce the problem?

1. Install nightly build and start Jetty 9.

2. Come back the next morning.

EXPECTING: Jetty should still be running.

ACTUAL: Jetty is not running.  In the Console output, I see the following: lost 
connection with Eclipse , shutting down.

What Eclipse version are you using ?

Version: Juno Service Release 2
Build id: 20130225-0426

What Run-Jetty-Run version are you using?

  RunJettyRun Jetty9 Support (Optional) 1.3.3.201301020723

What OS are you using ? 32bit or 64 bit?

- OS X 10.7.5

Please provide any additional information below.

Would it be possible to change the nightly build to output the full exception 
in Bootstrap.java and then I can provide additional details?  Thanks!

Original issue reported on code.google.com by [email protected] on 7 Jun 2013 at 3:00

[GoogleCodeExporter]

I am seeing the same issue, but with different versions of things:

* RunJettyRun 1.3.3.201203161919,
* Jetty 8.1.2.v20120308
* Eclipse 3.8.2
* Oracle JRE 1.7.0_40 64-bit
* Ubuntu 12.04, 64-bit

I recently switched from a Oracle JRE 1.6 to Oracle JRE 1.7, and I don't recall 
seeing this problem with JRE 1.6. May or may not be coincidence ... I might be 
able to switch back and try 1.6 again.

Original comment by [email protected] on 30 Sep 2013 at 11:42

[GoogleCodeExporter]

Tried it overnight with Oracle JRE 1.6.0_32 64-bit, and had the same issue. So 
I guess it has nothing to do with Java 7.

Original comment by [email protected] on 1 Oct 2013 at 11:51

[GoogleCodeExporter]

got the same problem.

nobody knows the cause of it?

Original comment by [email protected] on 16 Oct 2013 at 6:53

[pghj]

Same problem here. I'm pretty sure it's caused by resource exhaustion (too many open files).

$ netstat -atpn shows a huge number of these lines: tcp6 1 0 127.0.0.1:56042 127.0.0.1:57728 CLOSE_WAIT 4386/java

(Jetty being run with argument -DrjrEclipseListener=57728 )

These sockets are opened in plugin/bootstrap/runjettyrun/Bootstrap.java every 5 seconds.

line 202: Socket sock = new Socket("127.0.0.1", configs.getEclipseListenerPort());

They are never closed, however. If jetty is being actively used, garbage collection will finalize the sockets and close them. When unused for a long time, garbage collection might come too late and the creation of a new socket fails with "too many open files". The error is lost because the catch block a few lines below discards it:

catch (IOException e) { System.err.println("lost connection with Eclipse , shutting down."); shutdownServer(); }

I suggest putting the socket creation in a try-with-resources block, and invoking e.printStackTrace() or appending e.getMessage() to the error being displayed.

It would also be better if the server side in plugin/src/runjettyrun/Plugin.java (around line 200) used try-with-resources as well.

[xzer]

@pghj I am the current maintainer of this plugin, according to the source you pointed out, it actually seems a resource leak issue. I will try to reproduce it an fix it in the current nightly build.

[xzer]

@pghj I cannot reproduce this issue even I left my eclipse in whole night, however, I fixed the resource leak you pointed out and has released a new nightly build, you can try the newest nightly build and see whether the problem has gone.

[pghj]

According to Eclipse I now have version 1.3.5.201606280300. Shouldn't that have the fix applied? I still see the problem, and as far as I can tell from the byte code of runjettyrun.Bootstrap$1.class, the socked still isn't being closed...

[xzer]

I am sorry that your answer confused me. I checked the compiled class files, the socket related logic is contained in Bootstrap$2.class, not in Bootstrap$1.class. And the Bootstrap$1.class containing logic is about stdin.

And also for the Bootstrap$2.class, I checked the uploaded file and decompiled it, which shows that the socket has been closed correctly.

Could you please confirm that whether there are two verion of RJR installed in you eclipse? Which may be caused by the recent migration. Can you uninstall your RJR and install it again and to confirm it again?

[pghj]

I was rather confused myself. I uninstalled and reinstalled RJR as you said multiple times to no avail. But I do think I understand the problem better now:

For every version of jetty other than 6.1.26, the files runjettyrun.jetty[VERSION]_1.3.5.201606280300.jar carry their own instance of lib/bootstrap.jar. I now see that for runjettyrun_1.3.5.201606280300.jar (jetty 6.1.26), which you were apparently looking at, the problem is indeed fixed. But for every other version of jetty, the corresponding bootstrap.jar contains an unfixed Bootstrap.class (and incidentally has Bootstrap$1.class and Bootstrap$2.class swapped)

Indeed, looking at run-jetty-run/Jetty[VERSION]Support/plugin-jetty[VERSION]/bootstrap/runjettyrun/Bootstrap.java you can see that all these files are still unfixed.

I didn't realise this code was duplicated for every supported version of jetty. Could you apply the fix to all of them?

[xzer]

@pghj ,Thanks very much for your patient and effort.

I didn't realise this code was duplicated for every supported version of jetty. Could you apply the fix to all of them?

This is my fault, I forgot this fact even I had realised that at the first day I got the ownership of this project. I was planning to fix this duplicated source issue, but I forgot it.

Anyway, I have fixed all the bootstrap.java in all versions, and uploaded a new nightly build, please try it again.

[mukundjalan]

I still see this issue. I am using eclipse 4.6.2 with RunJettyRun 1.3.5.201611231530

[xzer]

It seems a long term issue, but could you please tell me how to reproduce it? As I mentioned above, I cannot reproduce it at my machine.

[mukundjalan]

For me it happens when I run Jetty and leave the laptop on stand by for some time. Doesn't happen to my colleagues. It is very random. May be you could try putting your system on stand by and check after 30 mins.

[xzer]

I will try it.

[mukundjalan]

After posting my issue here, I unchecked "Enable Eclipse Listener to prevent Jetty leaks when Eclipse crash" and my Jetty hasn't crashed since last 2 days. Laptop was put on stand by 2-3 times in this period but everything still works fine. I am not sure what this option does but it was an trial & error to uncheck it, & probably that has saved me from the issue to occur again. Hope this information helps.