[Bug] TPS Server problems caused by ELN

[codgamerMC]

I recently encountered problems with ELN, my server reports a TPS of 3/20 and is now almost unplayable, my map is approx. 7500x3000 blocks it's quite big in a Flans mods car it will take you 5-10 min to get from tip to tip, I established an electrical grid all over the map, 5 Power Plants, it's a stable system and no tps problems until I decided to place streetlights all over the map in the 10 roads I built which are long roads, after the placement of over 2000 streetlights and using relays and signals to switch the on and off at night, I started having tps problems, and also I built a monitoring station for my power plant that uses aprox. 30 signal transmitters and it furthered the TPS into 3/20, I would like to know if this is a tick bug or something related in to that nature cause I really like the mod, I had to remove this station to get atleast 6-7 TPS but the main problem is the streetlights, are they supposed to be rendered all the time although player is not in that area? I also realized that streetlights stay on during day time on unloaded chunks that were loaded during the night and causing a jump in kW consumption sometimes leading to power plant security shutdown to protect the system and when restarting the lines, the instant kW and Amperage is too high to safely get the system voltage to 3.2 KV without it exploding, If I could know why the TPS issue happens and how to solve it I will appreciate it cause this map is my custom build on real life as close as possible, project is 3 years into the making and in the recent 4 months I have played with ELN and has changed it to the better!!

[lashtear]

Hm. Several things: Lighting changes are very heavy in minecraft, as they involve lighting recalculations; streetlighting of that sort then necessarily involves quite a lot of map churn and keeping everything chunkloaded makes for a very very large active set of chunks for the server.

Recommend: Use a grid of poles, always energized (3.2kV on the ground, 12.8kV in the air). Then supply individual lights via lamp supplies on dc-dc converters. (200V LED bulbs most likely). Do the switching for day/night via signals into the lamp supply; I recommend using a daylight sensor into a signal proc looking for e.g. <0.2. With signal repeaters that can handle the whole grid on one sensor. Anything doing switching on/off (lamps, processors, etc) needs to be chunkloaded.

Not doing chunkloading will cause it to continue with its last known state.

[codgamerMC]

Thanks for the comment, so few questions, I have a grid system like mentioned but every lets say 200 blocks I have a D.C. D.C. Converter with relay and daylight sensor with the lamp supply because each supply can only reach so far, so what you are saying is instead of using so many sensors just use one and use repeaters to get to the relays? Do I have to remove the lamp supplies or are they fine? Is he problem just the sensors? That is part of the problem the other part that remained unanswered is does the over 2000 streetlights with relays and sensors cause TPS problems cause I have been getting like 7-10 tps ever since, thanks in advanced

[lashtear]

I was primarily recommending use of a grid, with on/off switching handled by the supplies-- which you seem to do. (Charging and discharging an entire grid is expected to be awkward because of the unfortunate grid capacitance in the current electrical model).

You can use separate sensors on each supply if you prefer; shouldn't make a big difference. I do recommend switching via a wireless-signal to the supply rather than a relay to the supply but either should work.

Because of how Minecraft lighting updates work, the transition on/off or off/on will be somewhat lag-inducing. That would show up as a temporary lag after each transition period.

If you're seeing persistent lag, that's different. Can you describe whether that is server-side lag (e.g. handling all those tile entities) or client-side (e.g. rendering lag)? "I don't know" is an okay answer, but we'll have to do more detailed testing. Lowering your render distance may help check that though...

We've tested very large (continental) grids with diverse powerplants but I don't think we've performance tested large lighting arrays recently. I'm sorta curious what the bottleneck there is.

[codgamerMC]

Well to let you know more about my grid system it's approx 7500 blocks wide by 3000 north, the grid supports more or less 2000 streetlights consumption of the grid is more than 60kW of power, after doing all this I noticed the server side lag in TPS and it wasn't client sided, my avg tick ranges from 130-230m/s which is bad, it's constant all the time and then I decided to create a monitoring station for the power plants and used over 30 transmitters and receivers, I changed the config to set the range of the transmitters to 10000 blocks so it could reach from one side of the map to the station without the use of repeaters and using that many transmitters made the lag worst into 3/20 TPS, what causes all this lag is beyond my knowledge and why was it caused, all I know is to my understanding my Processor can withstand this intensity but java isn't using it at all not even more than 2GB of RAM, which is all conserning, thanks in advanced!!

[jrddunbr]

Perhaps the best way to see what the problem is would be to replicate the problem. This would allow us to see how exactly the Minecraft server is behaving on local hardware, and what physical limitation we are hitting with the world you have.

Some useful information you could give us:

• Java command line, including flags, that you use to start your sever (or, in other words, how do you start Java)
• Platform:
  • Windows: Version Name
  • Linux: Output of uname -a
• Version of the JRE you are using: java -version in a command prompt window or shell will do.
• Raw computer specs (ex, processor name, ram speed, size, and type, disk type) or if it's a hosted server, what the specs are that the hosting provider tells you.

My current suspicion is that the main thread of the Java program is hitting the top capacity of one processor core. Once that happens, the TPS begins to fall rather quickly, regardless of how many cores the system has - so your CPU usage may look low, but it may be already overloaded, and this is mostly undetectable without looking at how each core performs (and in general, CPU usage graphs are misleading). Having as much of the world loaded as you do may be causing the problem - if you have ~100,000 chunks loaded, that may be a limitation of the server hardware to do all of the ticks for each chunk loaded. Even 1/100'th of that being loaded (or 1,000 chunks) is a huge number of chunks loaded, and could definitely thrash a vanilla world on a decent server.

Summary

If you can provide us system specs, the entire server directory (configs, world, mods) in an archive file (zip, 7z, tarball, etc), what command line you use to start it (or otherwise), and any other useful information, we can do some more complicated diagnostics locally using some debuggers and server tick monitors.

If you are not comfortable with giving us your world file, in replacement you can give us a handful of screenshots with the critical components that drive your lighting systems, including but not limited to the distribution system, generation system, and other control systems you may have, so that we can re-create the problem. The other information such as the configs and mod-pack would still be useful.

I may ask more specific questions depending upon your platform, but this is all I can think of for now.

[codgamerMC]

I would love to provide the whole server directory only problem is it's about 2-3 GB files and you may or may not be aware, I live in Puerto Rico we been hit by a Hurricane on September and I have electricity but not internet so I rely on my phone ATT connection to connect my self, so I would spend my data plan on sending you that, what I can do is send many screenshots since they don't consume data and send that with all the information, i can comply with everything just that the world file is kind of big to send it, I appreciate all the help!! If you need more information after I send the information this afternoon you can reply and I will reply back, some useful information that you may now have that I can remember and later confirm is that I have Java 1.8073 version or similar, my of PC specs are i7-4790k 4ghz "8" cores 16GB RAM DDR3, 250 GB SSD with 3TB HDD, I will add to this info later on.

[codgamerMC]

I forgot to add that, my RAM speed is 1866 Mhz and Windows 7 ultimate 64bit

[jrddunbr]

Hmm, yeah don't waste your limited bandwidth. Mods can also be rather large, so instead just give us a complete listing of what is in the mods folder, and the Eln configuration file, and some screenshots.

As for CPU specs, your CPU really dominates on single core performance - it's in the top 5 on this list. That is probably not the bottleneck. The rest of your rig looks pretty reasonable, so I think we've got the specs part down - it's probably not the computer's fault. I don't need any more platform info.

Likely, there are too many chunks being loaded, and having Eln talk across the entire world at once is causing a lot of chunk churn. Are you using chunk-loaders at all?

[codgamerMC]

Here you can download a zip file with all the relevant information I think might be necessary, just to point out, this server is for me a real life simulation I am trying to create, so it imitates Puerto Rico's Electrical Power Grid, you will find a Picture of Puerto Rico in dynmap form which is the shape of the actual Minecraft World and I used Paint to draw the grid I created, In yellow you will find the 3,2 KV grid which runs parallel thru all the roads and that's where the streetlight density can be found, almost all of the map is not loaded, chunk loaded areas are small, for example all power plants and some minor places here and there, there are approx. 5000 chunks loaded says OPIS (Also Included in the zip), in red the 12.8 KV Transmission lines, Green dots are distribution substations, in orange you will find the powerplants. I also send pictures of the grid, pictures of the power plants, roads and streetlight setup, if you need more information let me know, it's a very big grid whose consumption is around 60kW. http://www.mediafire.com/file/i4sczdv74n8edn3/ELN_Server_TPS_Problems.zip

[jrddunbr]

Run java -version, and paste the output. You look to be using 32 bit Java if you are having problems using more than 2GB of ram. From what I see in Opis there, you may be just spending a ton of time GC'ing instead of doing game ticks... 13ms/t is fine for the world objects...

Also, can we have the Eln config you have? You didn't include it in the zip from what I see.

PS. Nice map :)

[codgamerMC]

Thanks!! Also thanks for all the help you given me I appreciate it a lot!! Here is the config file and one question, what is GC'ing... kind of ignorant on what that means.

Configuration file

balancing { D:ElnToIndustrialCraftConversionRatio=0.3333333333333333 D:ElnToOpenComputerConversionRatio=0.13333333333333333 D:ElnToThermalExpansionConversionRatio=1.3333333333333333

# Maximum horizontal distance from autominer that will be mined
I:autominerRange=10
D:batteryCapacityFactor=1.0
I:fuelGeneratorPowerFactor=1
D:fuelHeatFurnacePowerFactor=1.0

# Factor to apply when converting real word heat values to Minecraft heat values (1mB = 1l).
D:fuelHeatValueFactor=6.75E-5
I:heatTurbinePowerFactor=1
I:platesPerIngot=1
I:solarPanelPowerFactor=1

# Display more detailed WAILA info on some machines
B:wailaEasyMode=false
I:waterTurbinePowerFactor=1
I:windTurbinePowerFactor=1

}

battery { # How many days it takes for a battery to decay half way I:batteryHalfLife=2 }

compatibility { B:ComputerProbeEnable=true B:ElnToOtherEnergyConverterEnable=true }

debug { B:enable=false }

dictionary { B:chips=true B:tungsten=false }

entity { B:killMonstersAroundLamps=true I:killMonstersAroundLampsRange=9 I:replicatorId=-1 B:replicatorPop=true D:replicatorPopWhenThunderPerSecond=0.008333333333333333 }

fuelgenerator { I:tankCapacityInSecondsAtNominalPower=1200 }

gameplay { B:explosion=true }

general { B:analyticsEnable=true S:playerUUID=b3a3dfcf-996f-4f3e-84e2-53c8a1e1eda9 B:versionCheckEnable=true }

lamp { D:carbonLifeInHours=6.0 D:economicLifeInHours=64.0 D:incandescentLifeInHours=16.0 B:infiniteLedLife=false D:ledLifeInHours=512.0 }

mapgenerate { B:cinnabar=true B:copper=true B:forceOreRegen=false B:lead=true B:tungsten=true }

modbus { B:enable=false I:port=1502 }

simulation { D:cableRsFactor=1.0 I:electricalFrequency=20 I:electricalInterSystemOverSampling=50 I:thermalFrequency=400 }

wireless { I:txRange=10000 }

xrayscannerconfig { B:addOtherModOreToXRay=true B:canBeCrafted=true D:rangeInBloc=5.0 }

[codgamerMC]

Also on a side note, I tried the code: java -server -Xms4G -Xmx8G -jar custom_server.jar It still didn't got beyond 2GB of RAM

[jrddunbr]

GC'ing means Garbage Collect(ing). Java will occasionally go to collect the variables that are no longer in use and remove them from the ram allocation. If you run out of usable RAM within the JVM, it will race to clear as much RAM as possible, even if it's to the determent of the system running it. Eventually, it can just hang your stuff.

I still need your Java version, and whether it's 32 or 64 bit. Like I said before, java -version in a Windows command prompt will do the trick.

Glad to help!

[codgamerMC]

Java Version:

[jrddunbr]

That's 64 bit Java alright. Now, what on Earth is your game doing if not playing the GC game... Perhaps a screenshot from each screen of Opis would help. The tick time spent on what looks to be nothing seems rather large. I'll have to experiment with that later this week.

[codgamerMC]

Do you have any suspicion at the moment of what can be that tick lag? Something to note, I removed ELN from a backup I made and it instantly went up to 20TPS no lag at all, so I assume that the lag is caused by ELN in some way or form, how can you help me experiment on this issue? Extra Info.zip

[jrddunbr]

I suspect it must be something to do with the electrical simulation. Perhaps if you split your electrical network into smaller segments, it would behave better for the time being? Are all of the poles on the 3.2kv and 12.8kV networks connected together across the map? Perhaps just sticking to using local power generation to each quarter of the map would help the electrical simulation in some way, and isolate the networks otherwise, to prevent larger network calculations. Or, perhaps putting some of the DC/DC regulators in isolated mode would help.

The interesting thing I've noticed is that it appears that the time broken up for each tick is mostly spent on something that is not the worlds themselves, based on information in two of the screenshots on the first upload and confirmed by the second - meaning it's either the electrical simulation, or something else funky that we can't see. It's probably not directly related to the chunks, and if it is, it's too many chunks with Eln being loaded on it causing larger calculations for the network.

@Baughn - Do you have any ideas on what's up with the simulation? It's a black box to me.

[codgamerMC]

Yes all the lines are connected as shown in a grid color coded picture I sent you yesterday, I started noticing the problem when I was progressing with more and more streetlights, that meaning more lamp supplies, DC-DC converters, relays, amplifier and day light sensors, when I finished the whole map I noticed that TPS was around 7-10, are there any other programs other than OPIS that can see in detail what is going on, on a side note, this grid is really a large scale simulation.

[jrddunbr]

I was planning to put it into JVisual VM to watch usages. You can try that, too.

[codgamerMC]

I meant to ask you if you knew any programs on how to do that, I don't know how to. How do you watch usages?

[jrddunbr]

Well, if you run the program, you attach it to the currently running server process, and it will show what things are taking what amount of time ("usages" - referring to using a profiler for CPU time). If you see something out of the ordinary while the CPU profiler is running, then you analyze it further. Generally, whatever is lowering your TPS will be repeating a lot and/or taking a lot of time.

Note that there is a lot to watch, there are network threads, tick threads, and other threads related to the GUI, since you're using it in headed mode on Windows (versus running headless mode Java like I usually do in Linux; in that platform, you wouldn't encounter those threads running.) Some mods may also be breaking out into other threads.

Perhaps something like this would be helpful.

[cm0x4D]

@codgamerMC You can try to set some DC/DC converters into "isolation" mode, so the huge simulated grid will be partitioned in smaller subnets and the Electrical Age simulation should theoretically get calculated more efficiently. The downside of this is that the simulation can get a bit less reactive as if there were big capacitors placed on both sides of the DC/DC converter.

[codgamerMC]

Currently I am busy on some work I have to do as of today I can't do more testing, hopefully tomorrow I can do some and I will get back to you guys tomorrow if I can and thanks for all the help, I really appreciate it

[Baughn]

If he uses utility poles, those are always isolated. (At the transformer pole / ground connection, that is.)

[codgamerMC]

At the root of each lamp supply there is the poles then a transformer that is in not isolated mode, relay, lamp supply, day light sensors with amplifiers, this setup repeats I would estimate for over 100 times, since I did this across the map I have been getting this lag, I removed the mod on a backup I made to test and it jumped back to 20 TPS so I assume there is something in the mod that is causing this mysterious decay in TPS, I also noticed that if you use more than 30 signal transmitters in the map over some distance, that will down the server on 6 more TPS, as to why, I don't know, any ideas @Baughn?

[codgamerMC]

Also thanks for the huge help to everybody that has helped in this thread, I love this mod!!

[bloxgate]

I also noticed that if you use more than 30 signal transmitters in the map over some distance, that will down the server on 6 more TPS, as to why, I don't know, any ideas

That's really odd. I remember playing on servers with way more than 30 signal transmitters with no issues. I've noticed you have the transmitter range set to 10 km, perhaps the transmitters are very inefficient with such a large range? Could you try using a significantly smaller range with repeaters, or transmitting your control signals via signal cable?

[codgamerMC]

I have no problem but the thing is that I built like a command center to receive this signals, but the default 32 blocks wouldn't help to transmit the signal over 6 km since my map is like 10 km wide I set it to that, but I had the TPS problems, way before that, anyway to test Tick time other than opis? Thanks for the help

[bloxgate]

You can transmit signals more than 32 blocks using the default range, you just need to put signal repeaters down near the edge of the range.

[Baughn]

The signal blocks raytrace to each other in order to figure out if they're in range or not, and they're not clever about caching data from unloaded chunks. I can absolutely believe that setting an excessively large range would cause lag.