After many weeks of studying the JVM, Flags, and testing various combinations, I came up with a highly tuned set of Garbage Collection flags for Minecraft. I tested these on my server, and have been used for years. I then announced my research to the public, and to this day, many servers have been using my flag recommendations for years and reporting great improvement to garbage collection behavior.
These flags are the result of a ton of effort, and results of seeing it in production on various server sizes, plugin lists and server types. They have proven themselves repeatedly.
I strongly suggest using these flags to start your server. These flags help keep your server running CONSISTENT without any large garbage collection spikes. CPU may be slightly higher, but your server will be overall more reliable and stable TPS.
If these flags help your server, consider donating!
The JVM Startup Flags to use – MC 1.15 (Java 8+, MC 1.8+) Update
Use these flags exactly, only changing Xmx and Xms. These flags work and scale accordingly to any size of memory, even 500MB but 1.15 will not do well with such low memory…)
IMPORTANT – READ – Don’t use ALL of your memory!! PTERODACTYL USERS!
I recommend using at least 6-10GB, No matter how few players! If you can’t afford 10GB of memory, give as much as you can, but ensure you leave the operating system some memory too. G1GC operates better with more memory.
If you are running with 12GB or less memory for MC, you should not adjust these parameters.
If you are using an Xmx value greater than 12G
If you have and use more than 12GB of memory, adjust the following:
NOTICE: If you see increase in old generation collections after this, revert back to the base flags!
Explanation of these changes:
- Base flag set aims for 30/40 to reduce risk of to space issues. With more memory, less of an issue. We can give more to new generation with 40/50, as well as reduce reserve percent since the default reserve will already be larger.
- Region Size increase helps reduce humongous allocations, and speeds up remarking. We need a smaller region size at smaller heaps to ensure an adequate amount of regions available
- We can start looking for old generation memory to reclaim with more of a delay with IHOP at 20 since we have more old generation available to space on CPU.
Java GC Logging
Are you having old gen issues with these flags? Help me help you! Add the following flags based on your java version to enable GC Logging:
-Xloggc:gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=1M
Once you start seeing old generation collections in Timings, grab the logs/gc.log file (same location as your latest.log) and send it to me on Paper Discord to analyze.
GC logging does not hurt your performance and can be left on at all times. The files will not take up much space (5MB)
Technical Explanation of the Flags:
- -Xms matching -Xmx – Why: You should never run your server with the case that -Xmx can run the system completely out of memory. Your server should always be expected to use the entire -Xmx!
You should then ensure the OS has extra memory on top of that Xmx for non MC/OS level things. Therefore, you should never run MC with -Xmx settings you can’t support if java uses it all.Now, that means if -Xms is lower than -Xmx -YOU HAVE UNUSED MEMORY!Unused memory is wasted memory.G1 (and probably even CMS to a certain threshold, but I’m only stating what I’m sure about) operates better with the more memory it’s given. G1 adaptively chooses how much memory to give to each region to optimize pause time. If you have more memory than it needs to reach an optimal pause time, G1 will simply push that extra into the old generation and it will not hurt you (This may not be the case for CMS, but is the case for G1).The fundamental idea of improving GC behavior is to ensure short lived objects die young and never get promoted. With the more memory G1 has, the better assurance you will get that objects are not getting prematurely promoted to the old generation.G1 Operates differently than previous collectors and is able to handle larger heaps more efficiently.
If it does not need the memory given to it, it will not use it. The entire engine operates differently and does not suffer from too large of heaps, and this is industry wide accepted information that under G1 to keep Xms and Xmx the same!
- UnlockExperimentalVMOptions – needed for some the below options
- G1NewSizePercent: These are the important ones. In CMS and other Generations, tweaking the New Generation results in FIXED SIZE New Gen and usually is done through explicit size setting with -Xmn.With G1, things are better! You now can specify percentages of an overall desired range for the new generation. With these settings, we tell G1 to not use its default 5% for new gen, and instead give it 40%!Minecraft has an extremely high a memory allocation rate, ranging to at least 800 Megabytes a second on a 30 player server! And this is mostly short lived objects (Block Position)
Now, this means MC REALLY needs more focus on New Generation to be able to even support this allocation rate. If your new gen is too small, you will be running new gen collections 1-2+ times per second, which is really bad.You will have so many pauses that TPS has risk of suffering, and the server will not be able to keep up with the cost of GC’s.Then combine the fact that objects will now promote faster, resulting in your Old Gen growing faster. Given more NewGen, we are able to slow down the intervals of Young Gen collections, resulting in more time for short lived objects to die young and overall more efficient GC behavior.
- G1MixedGCLiveThresholdPercent: Controls when to include regions in Mixed GC’s in the Young GC collection, keeping Old Gen tidy without doing a normal Old Gen GC collection. When your memory is less than this percent, old gen won’t even be included in ‘mixed’ collections. Mixed are not as heavy as a full old collection, so having small incremental cleanups of old keeps memory usage light.
Default is 65 to 85 depending on Java Version, we are setting to 90 to ensure we reclaim garbage in old gen as fast as possible to retain as much free regions as we can.My Old flag set had this at 35 which was a bug. I had the intent of this flag inverted, as I thought 35 was what 65 does. You should not be using 35 for this number.
- G1ReservePercent=20: MC Memory allocation rate in up to date versions is really insane. We run the risk of a dreaded “to-space exhaustion” not having enough memory free to move data around. This ensures more memory is waiting to be used for this operation. Default is 10, so we are giving another 10 to it.
- MaxTenuringThreshold=1: Minecraft has a really high allocation rate of memory. Of that memory, most is reclaimed in the eden generation. However transient data will overflow into survivor. Initially played with completely removing Survivor and had decent results, but does result in transient data making its way to Old which is not good.Max Tenuring 1 ensures that we do not promote transient data to old generation, but anything that survives 2 passes of Garbage Collection is just going to be assumed as longer-lived.
Doing this greatly reduces pause times in Young Collections as copying data up to 15 times in Survivor space for a tenured object really takes a lot of time for actually old memory. Ideally the GC engine would track average age for objects instead and tenure out data faster, but that is not how it works.
Considering average GC rate is 10s to the upwards of minutes per young collection, this does not result in any ‘garbage’ being promoted, and just delays longer lived memory to be collected in Mixed GC’s.
- SurvivorRatio=32: Because we drastically reduced MaxTenuringThreshold, we will be reducing use of survivor space drastically. This frees up more regions to be used by Eden instead.
- AlwaysPreTouch: AlwaysPreTouch gets the memory setup and reserved at process start ensuring it is contiguous, improving the efficiency of it more. This improves the operating systems memory access speed. Mandatory to use Transparent Huge Pages
- +DisableExplicitGC: Many plugins think they know how to control memory, and try to invoke garbage collection. Plugins that do this trigger a full garbage collection, triggering a massive lag spike. This flag disables plugins from trying to do this, protecting you from their bad code.
- MaxGCPauseMillis=200: This setting controls how much memory is used in between the Minimum and Maximum ranges specified for your New Generation. This is a “goal” for how long you want your server to pause for collections. 200 is aiming for at most loss of 4 ticks. This will result in a short TPS drop, however the server can make up for this drop instantly, meaning it will have no meaningful impact to your TPS. 200ms is lower than players can recognize.In testing, having this value constrained to an even lower number results in G1 not recollecting memory fast enough and potentially running out of old gen triggering a Full collection. Just because this number is 200 does not mean every collection will be 200. It means it can use up to 200 if it really needs it, and we need to let it do its job when there is memory to collect.
- +ParallelRefProcEnabled: Optimizes the GC process to use multiple threads for weak reference checking. Not sure why this isn’t default….
- G1RSetUpdatingPauseTimePercent=5: Default is 10% of time spent during pause updating Rsets, reduce this to 5% to make more of it concurrent to reduce pause durations.
- G1MixedGCCountTarget=4: Default is 8. Because we are aiming to collect slower, with less old gen usage, try to reclaim old gen memory faster to avoid running out of old.
- G1HeapRegionSize=8M+: Default is auto calculated. SUPER important for Minecraft, especially 1.15, as with low memory situations, the default calculation will in most times be too low. Any memory allocation half of this size (4MB) will be treated as “Humongous” and promote straight to old generation and is harder to free. If you allow java to use the default, you will be destroyed with a significant chunk of your memory getting treated as Humongous.
- +PerfDisableSharedMem: Causes GC to write to file system which can cause major latency if disk IO is high – See https://www.evanjones.ca/jvm-mmap-pause.html
Using Large Pages
Also for Large Pages – It’s even more important to use -Xms = -Xmx! Large Pages needs to have all of the memory specified for it or you could end up without the gains. This memory will not be used by the OS anyways, so use it.
Additionally use these flags (Metaspace is Java 8 Only, don’t use it for Java7):
Transparent Huge Pages
Controversial Feature but may be usable if you can not configure your host for real HugeTLBFS. try adding -XX:+UseTransparentHugePages but it’s extremely important you also have AlwaysPreTouch set. Otherwise THP will likely hurt you. I have not measured how THP works for MC or its impact with AlwaysPreTouch, so this section is for the advanced users who want to experiement.
Thanks to https://product.hubspot.com/blog/g1gc-fundamentals-lessons-from-taming-garbage-collection for helping reinforce my understanding of the flags and introduce improvements!
- 5/2/2020: Added +PerfDisableSharedMem, Adjusted MixedGCTarget to 4
- 4/25/2020: Removed OmitStackTraces since it could cause performance issues with some plugins (but not everyone)
- 4/5/2020: Massive refactor of the flag suggestions. Takes a new approach at optimizing pause times. Flags may still be changing. These changes are mandatory for MC 1.15
- 10/4/2018: Removed AggressiveOpts and InitiatingHeapOccupancyPercent. Aggressive is removed in Java 11, and IHOP may hurt performance in Java 11. You should remove them for Java 8 too.
- 8/18/2018: Adjusted MixedGCLiveThreshold to 35 (from 50) to ensure mixed GC’s start earlier.
Added notes about recommended use of 10GB of memory.
Added more flag documentation
- 5/24/2018: Added -XX:+ParallelRefProcEnabled
Lately I have been live streaming while working on various Empire Minecraft or Website related tasks! I like the idea of Live Coding as it invites others to learn from you, and on the flip side, allows others to offer constructive criticism to you to improve your code.
I’ve done some work to be able to split stream to 4 (or more if needed) services at the same time, so you can subscribe and watch on any of the following services:
NOTICE: Due to Mixer being a much superior platform, I’ve decided to end streaming to other platforms so that I can use FTL
Please subscribe and view my streams on Mixer only.
The method I used to stream to multiple before is still below.
- Mixer: https://mixer.com/Aikar
Please subscribe to get alerts on when I go live!
If you are interested in Live Streaming to multiple services at once, follow this guide:
And here is my example config file:
I’m also using this fancy Bash Script using xdotool to automate switching scenes with OBS, install xdotool, assign 2 scenes a hotkey for control + alt + [ and control + alt + ]
I have wrote a few methods (color map credit to mcrcon) that will help with outputting colored text using Minecraft color codes.
Usage is like
console.log(mccolor(c("3", "red Text) + c("a", "green text")));0
Reposting because Google has bugged out and ended up dropping this from Google :/ So making it look new to get it back on Google.
So as any Bukkit developer knows, the API is not thread safe! And to make matters worse, there is no concrete Java Control Flow API in Bukkit.
However at Empire Minecraft our server very heavily depends on our MySQL Database to provide features. So running database queries on the main thread is common but undesired, and Java control flow is needed.
Running queries async creates complicated java control flow issues, need to run this query… now need to access the bukkit api, so return to Sync processing, oh wait, now I need to act again with another database query!
Easy to avoid all that java control flow trouble by running everything sync – but then performance can be hurt.
Therefor, to avoid Java running into this same callback mess, I wrote an elegant Java Flow Control system on top of the Bukkit Scheduler.0
Been a while since I posted, but I’ve recently got a few things down lately I felt the need to share.
A major asset to being a developer is the ability to be productive. Remember, time is money. One thing that always irks me when working with junior developers or designers is them doing actions that I felt could be done much quicker with a proper IDE or practice. So much time is wasted doing repetitive things every single day.
Ever watched someone navigate a folder hierarchy to get to a file they want to open? A good 30 seconds or more can be spent there, especially if that file is on a network mount!
With a proper IDE with good navigation support, navigating to a desired file can be drastically reduced. I personally own a license to IntelliJ IDEA 14 Professional, and use it for my Java projects and until recently Web Projects. I value efficiency so much, that I now have even purchased PhpStorm 8, even though IDEA supported PHP. Namely for things like Debugging, but thats another topic.
PhpStorm tailors the experience to web development more, but some of the key notes are still in IDEA.
Things such as opening a file, if you work in an extensive codebase thats been developed over many years, can likely be pretty deep and take a while. But with IDEA and PhpStorm (And likely other IDE’s too, I know VIM does similar), a Key Combo away gives you a nice search box that you can type in a class name or file name and find the exact file you want.
If that file is on a network mount and you have a varying file structure that you cant remember exactly where that file resides, how many minutes per ACTION are you wasting opening up that file?
Then inspections… The ability to see errors in your code before you even run it. I’m not talking about just syntax errors here. Typo in a variable name? That’s not a syntax error, but a good IDE will show you that variable is unused, and another variable hasn’t been defined yet, giving you a sign you have a problem before you even leave your editor.
This is just the tip of the iceberg. IDE’s are built to save you time, yet so many people feel they are hard core because they built a website in Notepad++. You are not hard core, you are wasting time. Use the tools that have been built to make you more productive, and strive to increase your productivity.
I personally recommend Jetbrains products as they have a STRONG focus on productivity (Help -> Productivity Guide) and get more small productivity features all combined into 1 product that many other smaller editors have. You have so many goodies to play with to make you more productive. But the key point, use an IDE, be productive, save you / your company time, and stop trying to show off by claiming you did it all in Notepad. The only people your impressing is juniors who don’t know any better.
If you have a good developer skillset, don’t waste your precious time on inefficiencies in workflow.0
I spend quite a lot of time improving the CraftBukkit’s core code, in order to make it perform better, more efficient and overall do things in a better way. I’m responsible for some of the biggest performance improvements that’s been made to the Minecraft Server.
However, if you’re not apart of the Spigot community, it’s likely you’ve not seen my name before. That’s because I’ve barely had any PR’s accepted to the CraftBukkit project.
It’s not for a lack of trying, it really all boils down to politics and policies.
I feel that CraftBukkit’s PR Policy is a reason the project went further downhill. There are countless people like myself that want to work to make the project better, but do not have the time to deal with the politics.
I’m speaking of the “holier than thou” attitude that the (original) CraftBukkit team (well, specifically certain members who handled PR’s) attitude.
It’s not that I’m lazy, it’s quite the opposite: I’m BUSY. I work a full time job to pay the bills, then another full time job to run my own Minecraft Server. I don’t have the time to contribute to Bukkit or Spigot, but I do it any ways especially when it can benefit my own server.
Every experienced developer knows designing a solution and testing and working all the bugs out is the hard part of writing code, not the syntax or styling details. However, COUNTLESS Pull Requests are closed on Bukkit purely due to minor things.
I look at it this way, if all that is “wrong” is a simple white-space on a single line or a 3 line change, then just make the fix yourself when pulling it!
This PR for example: https://github.com/Bukkit/CraftBukkit/pull/1352
Only feedback was formatting errors essentially which I resolved, then it sat for many months untouched. It then of course conflicted with newer commits.
The problem is Bukkit thinks everyone contributing is sitting around, waiting to make the next change to get it accepted. But, we can’t do that. We have other things to work on. My focus has been switched and I’m now working on the next task.
To have to stop what I’m doing, switch back to a different project/checkout, add a space and recommit and push… It’s hard to justify stopping work on something significant for such a small thing.
Essentially, Bukkit thinks we should lose 10-30+ minutes of our time, and most importantly: break concentration (and we all know how bad that is) for meaningless politics just so that the Bukkit team member doesn’t have to do a single change to the code, to save them 2 minutes.
I disagree with this ideology completely. When you are a team member of a successful open source project, your goal should be to make that product great. If someone took their time to solve a complicated issue, and bring it to production readiness, it is fairly reasonable for you to spend 5-30 minutes when pulling it to fix up tiny things, verify it works yourself, and fix up things that you want changed that does not interfere with the logic of the change itself (styling, whitespace, imports, etc).
The only reason you should be sending a PR back to the submitter is if you have a concern with the implementation, or need more details to understand the change yourself.
A project maintainers time is no more important than the contributors, and remember, its not the contributor who’s going to get the mass-praise over a major new change – its the project itself and its developers, even when they didn’t even do the hard part of a change.
Be happy that someone is helping you improve your product, and work WITH them to improve your product.
Mojang: My suggestion to you – Put someone in charge of the project who is enthusiastic about making the project great. Someone who gets excited when someone is working on “the next big change” for them, and is eager to work with them and discuss it and push for it to be included.
Please build a team of FRIENDLY people, people who are not going to personally insult everyone who tries to have low level talks, and someone who is not going to try to make others feel lesser than them.
Loosen up the PR process, and give people realistic hope that a PR might actually be pulled before the next major upgrade that conflicts the PR.
Another thing to do is encourage low level discussion and “potential” talk. Numerous times I tried to discuss the feasibility of whether or not CraftBukkit would even CONSIDER an idea. I wasn’t about to go spend 12+ hours working on a change to just be told “We don’t want to maintain a change of that scale” or “we don’t think the API should offer that”.
But every time I was told to go do it then bother them with it… And discussing the idea without showing results was met with insults.
That can not happen any more.
Spigot isn’t perfect, but is a much better place than Bukkit was, so if you can at least match Spigot, then you will have done great things for the Bukkit project.
If any Mojang member (Dinnerbone?) wants to discuss this more with me, I’m on IRC (Esper, Spigot, FreeNode) as Aikar every day.
Here’s hoping Dinnerbone is true to his word that Bukkit will be better than before!
I’ve written aother node module to provide Google Traceur support for Node.JS
If you are not familiar with Traceur, read up on it here: http://code.google.com/p/traceur-compiler/
And the features it provides: http://code.google.com/p/traceur-compiler/wiki/LanguageFeatures
npm install traceur
then all require() calls can be traceur syntax files.
Note you can’t use Traceur syntax on the file that includes Traceur,
since this extends require() itself to go through Traceur first before
being passed to normal node require().
So, I was reading over the V8 docs yesterday and found a really neat feature in the C++ API… Interceptors.
Interceptors let you provide a global getter/setter on an object so you can get a callback for ANY property request/set without specially adding a setter/getter by name for it. This is something that I’ve found lacking in V8 JS Land to mimic PHP’s __get and __set methods, but it appears it does exists, just only on the C++ side!
So I wrote Magic this weekend, my first full 1.0.0 (1.0.1 actually, slight fix) release of a module to the public (advertised). This lets you play with a Magic Object with your own custom getters/setters (Interceptors) and do creative things with it.
Hopefully some modules that use promised based approaches will find creative ways to use these magic objects and create useful things out of it!
Download/Instructions @ GitHub0
Just wanted to update on what i’m currently working on.
Nova is pretty much in a releasable state except that I ran into another idea to do at work and it was something I could use asap at work so I began work on it.
I’m creating an alternative package installer for NPM, with a few key differences:
- No symlink hell: Maybe no symlinks at all. I’m designing it to install a package from NPM without any symlinks (i think I may make it so that if you link to a non JS file as a module it will symlink, but undecided yet as that feature doesn’t even work on default NPM, so possibly wont do any symlinking)
- Lightweight packages: A module package will not include tons of references to other dependencies in its folder. It will simply be the folder that was published to npm (usually the project on github)
- Per Project basis: Modules will be installed to a specific projects libs folder and not on a global level. Letting you manage the packers needed for your specific project, and no needing to worry about file permissions, and no need to worry about deployment ensuring your server has all the needed npm packages installed at the correct version.
The recursive symlinks of npm destroy some IDE’s and overall makes a hard to figure out folder structure. They also do not play as nice for windows, causing it confusion.
So that was the main reason for making this. As I could no longer work on my project due to the symlinks (yes it’s an IDE bug that should be fixed, but the linking isn’t really necessary, at least not recursively)
Also, the idea of global modules does not serve well for packaging modules into a projects version control. Trying to configure npm to install to a projects folder instead of global was a nightmare.
So barenpm is designed to be a tool for people who want to bundle packages as part of their project and included in a much leaner fashion.
I also am writing this in order to show to Isaacs so he may take some ideas from this implementation of the install method and hopefully get NPM to provide these features and not need symlinks. That is my ultimate goal,
but I needed the solution now and could not wait for NPM to officially support these ideas.
If you want to check out the current code, hit it up on my github http://github.com/aikar/barenpm
It’s not done, but it’s getting close.
I will finish up Nova documentation and do a proper release afterwards.0