System Administration

Tuning the JVM – G1GC Garbage Collector Flags for Minecraft

Introduction

After many weeks of studying the JVM, Flags, and testing various combinations, I came up with a highly tuned set of Garbage Collection flags for Minecraft. I tested these on my server, and have been used for years. I then announced my research to the public, and to this day, many servers have been using my flag recommendations for years and reporting great improvement to garbage collection behavior.

These flags are the result of a ton of effort, and results of seeing it in production on various server sizes, plugin lists and server types. They have proven themselves repeatedly.

I strongly suggest using these flags to start your server. These flags help keep your server running CONSISTENT without any large garbage collection spikes. CPU may be slightly higher, but your server will be overall more reliable and stable TPS.

The JVM Startup Flags to use

Use these flags exactly, only changing Xmx and Xms. These flags work and scale accordingly to any size of memory, even 500MB)


Recommended Memory

I recommend using up to 10GB, No matter how few players! If you can’t afford 10Gb of memory, give as much as you can, but ensure you leave the operating system some memory too. G1GC operates better with more memory.

Going over 10GB may start to become subjective, but should be ok. But very few servers really need more than 10GB.

If you are running with 10GB or less memory for MC, you should not adjust these parameters.

Higher Old Generation Memory Needs

If you have a high player count and use more than 10GB of memory, and are seeing old generation lag spikes, you may want to adjust the following:

  • -XX:G1MaxNewSizePercent=60
  • -XX:G1NewSizePercent=40

Technical Explanation of the Flags:

  1. -Xms matching -Xmx – Why: You should never run your server with the case that -Xmx can run the system completely out of memory. Your server should always be expected to use the entire -Xmx! You should then ensure the OS has extra memory on top of that Xmx for non MC/OS level things. Therefore, you should never run MC with -Xmx settings you can’t support if java uses it all.Now, that means if -Xms is lower than -Xmx – YOU HAVE UNUSED MEMORY! Unused memory is wasted memory. G1 (and probably even CMS to a certain threshold, but I’m only stating what I’m sure about) operates better with the more memory its given. G1 adaptively chooses how much memory to give to each region to optimize pause time.If you have more memory than it needs to reach an optimal pause time, G1 will simply push that extra into the old generation and it will not hurt you (This may not be the case for CMS, but is the case for G1)The fundamental idea of improving GC behavior is to ensure short lived objects die young and never get promoted.With the more memory G1 has, the better assurance you will get that objects are not getting prematurely promoted to the old generation.G1 Operates differently than previous collectors and is able to handle larger heaps more efficiently. If it does not need the memory given to it, it will not use it. The entire engine operates differently and does not suffer from too large of heaps, and this is industry wide accepted information that under G1 to keep Xms and Xmx the same!
  2. UnlockExperimentalVMOptions – needed for some the below options
  3. TargetSurvivorRatio: I’m sure your all use to seeing this one suggested. Good news! It’s actually a good flag to use :DThis setting controls how much of the Survivor space is ABLE to be used before promotion. If survivor gets too full, stuff starts promoting to Old Gen. The reason behind this is to be able to handle memory allocation spikes.However, MC allocation rate for most part is pretty steady (steadily high…..), and when its steady its safe to raise this value to avoid premature promotions.
  4. G1NewSize Percent: These are the important ones. In CMS and other Generations, tweaking the New Generation results in FIXED SIZE New Gen and usually is done through explicit size setting with -Xmn.With G1, things are better! You now can specify percentages of an overall desired range for the new generation.With these settings, we tell G1 to not use its default 5% for new gen, and instead give it 50% at least!Minecraft has an extremely high a memory allocation rate, ranging to at least 800 Megabytes a second on a 30 player server! And this is mostly short lived objects (Block Position)Now, this means MC REALLY needs more focus on New Generation to be able to even support this allocation rate. If your new gen is too small, you will be running new gen collections 1-2+ times per second, which is really bad.You will have so many pauses that TPS has risk of suffering, and the server will not be able to keep up with the cost of GC’s.Then combine the fact that objects will now promote faster, resulting in your Old Gen growing faster.Given more NewGen, we are able to slow down the intervals of Young Gen collections, resulting in more time for short lived objects to die young and overall more efficient GC behavior.if you run with a larger heap (15GB+), you may want to lower the minimum to say 40%, but don’t go lower than 30%. This will let G1 have more power in its own assumptions.
  5. G1MixedGCLiveThresholdPercent: Controls when to include Mixed GC’s in the Young GC collection, keeping Old Gen tidy without doing a normal Old Gen GC collection. When your memory is less than this percent, old gen won’t even be included in ‘mixed’ collections. Mixed are not as heavy as a full old collection, so having small incremental cleanups of old keeps memory usage light.
  6. AlwaysPreTouch: AlwaysPreTouch gets the memory setup and reserved at process start ensuring it is contiguous, improving the efficiency of it more. This improves the operating systems memory access speed.
  7. +DisableExplicitGC: Many plugins think they know how to control memory, and try to invoke garbage collection. Plugins that do this trigger a full garbage collection, triggering a massive lag spike. This flag disables plugins from trying to do this, protecting you from their bad code.
  8. MaxGCPauseMillis=100: This setting controls how much memory is used in between the Minimum and Maximum ranges specified for your New Generation. This is a “goal” for how long you want your server to pause for collections. 100 is equal to 2 ticks, aiming for an at most loss of 2 ticks. This will result in a short TPS drop, however Spigot and Paper both can make up for this drop instantly, meaning it will have no meaningful impact to your TPS. 100ms is lower than players can recognize.
  9. +ParallelRefProcEnabled: Optimizes the GC process to use multiple threads

Using Large Pages

Also for Large Pages – It’s even more important to use -Xms = -Xmx! Large Pages needs to have all of the memory specified for it or you could end up without the gains. This memory will not be used by the OS anyways, so use it.
Additionally use these flags (Metaspace is Java 8 Only, don’t use it for Java7):

Code:
 -XX:+UseLargePagesInMetaspace

Thanks to https://product.hubspot.com/blog/g1gc-fundamentals-lessons-from-taming-garbage-collection for helping reinforce my understanding of the flags and introduce improvements!


Changelog

  • 10/4/2018: Removed AggressiveOpts and InitiatingHeapOccupancyPercent. Aggressive is removed in Java 11, and IHOP may hurt performance in Java 11. You should remove them for Java 8 too.
  • 8/18/2018: Adjusted MixedGCLiveThreshold to 35 (from 50) to ensure mixed GC’s start earlier.
    Added notes about recommended use of 10GB of memory.
    Added more flag documentation
  • 5/24/2018: Added -XX:+ParallelRefProcEnabled

 

0

Apache Macros – Simplify your config

Using Apache Macros

Many people host small time hobby websites or even websites for family members, friends and clients on a single server. This will lead to quite a lot of repetition for the same apache site definitions over and over again. Thankfully Apache Macros mod will solve many of these issues. This mod will let you create config templates, that can then be re-used over multiple sections of code, allowing you to pass in variables to fill in on use.

Lets get it installed!

sudo apt-get install libapache2-mod-macro
sudo a2enmod macro

Now, you can start using Macros in your site definitions, to replace common configurations.

» Official Documentation for Apache Macros

Examples of Apache Macros

In this you can see a pretty simple Domain macro. This will set the ServerName, and sets a www. alias. Now we look at the Site macro. In this example you will see it calling other macros for Log, GrantAccess and ForceDomain.

To use this, one could simply add inside of the <VirtualHost> this line:

Use Site mysite.com

And then accessing mysite.com would redirect to www.mysite.com, and log to /var/log/apache2/sites/mysite.com_access.log. Needless to say that likely cuts out 99% of the configuration you’re doing for a simple wordpress site you host for a relative.

And since the Apache Macros are parsed at config load, there’s no impact to your servers performance for using it!

For enterprise grade setups, you’re likely already using Puppet to get the same benefits and only running 1 product per server any-ways, but for those of us kicking it hobby level, Apache macros helps quits a bit! Enjoy 🙂

0

Filtering Spam before Forwarding Email with Postfix/SpamAssassin

One feature many cPanel/Shared Webhosts has is an option to forward your email to a different address. Very useful if you want to have multiple email addresses but check it all in one place (Gmail) like I do. But if you’re like me, you’ve likely migrated onto your own dedicated server you manage yourself, and its likely your making mistakes with email forwarding and filtering spam!

The problem is that that when you receive spam, you are also forwarding spam to your email provider, which makes them upset with you and tarnishes your servers IP address. I did this for years! I always thought that Gmail would be smart enough to see the path in the headers to realize it was forwarded – but then thinking about it – why would Gmail trust me that those servers actually sent the email and that I didn’t just spoof those Received: lines to blame someone else?

When I recently migrated my host, I put in a lot more effort into filtering the spam before it even hits Gmail, and learned quite a few things.

Filtering Spam with Postfix

First Off: Initial Connection Client Checks – These stop a majority of the spammers, and its so simple!
Add this line to your /etc/postfix/main.cf:

smtpd_client_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination reject_rbl_client zen.spamhaus.org reject_rbl_client bl.spamcop.net reject_rbl_client cbl.abuseat.org reject_unknown_client permit

This will enforce a lot of restrictions on the client, namely the Zen Spamhaus check, which knocks out so many spammer connections!

Filtering Spam with SpamAssassin

If you haven’t already installed SpamAssassin, do so now. There is a bit to this than I want to put into this post, so follow this sites guide: http://plecko.com.hr/?p=389

His instructions look spot on to me. Key thing I did not do on my setup and I just realized I needed to do: enable CRON=1! I’ve been running with stale SA Rules… But his guide covers it!

Next up is this page: http://wiki.apache.org/spamassassin/ImproveAccuracy

One thing it mentions is missing Perl Modules that SpamAssassin can try to use. For me, I had to run these commands to get them all installed.

sudo apt-get install libgeoip-dev
sudo cpan Geo::IP Mail::DKIM Encode::Detect DBI IO::Socket::IP Digest::SHA1 Net::Patricia

I don’t know what some of them are for, but SpamAssassin is obviously trying to use them, so give them to it!

Passing SPF Checks

Then there is SRS Rewriting. One problem with forwarding email is that it makes every one of your emails now fail SPF checks, because it looks like your server is sending mail for InsertBigNameDomain.com which does not authorize you to send mail on their behalf.

SPF is considered a “broken” implementation, and it is preferred that system admins use DKIM instead as a way to verify authenticity of an email, so ideally you need to rewrite the return path to be your own server name instead.

I used this guide: https://www.mind-it.info/forward-postfix-spf-srs/
Which summarizes down to

sudo apt-get install cmake sysv-rc-conf
cd /usr/local/src/
wget https://github.com/roehling/postsrsd/archive/master.zip
unzip master
cd postsrsd-master/
make
sudo make install
sudo postconf -e "sender_canonical_maps = tcp:127.0.0.1:10001"
sudo postconf -e "sender_canonical_classes = envelope_sender"
sudo postconf -e "recipient_canonical_maps = tcp:127.0.0.1:10002"
sudo postconf -e "recipient_canonical_classes = envelope_recipient"
sudo sysv-rc-conf postsrsd on
sudo service postsrsd restart
sudo service postfix reload

Now when you inspect a received emails header, you will see that the ReturnPath is now something like  <SRS0+9CLa=52=paypal.com=service@starlis.com>
And your SPF will now pass (You do have SPF records set right for your domain?)

Dropping the Spam

Now the final part… getting rid of that spam before it goes to Gmail!

In /etc/postfix/header_checks (you likely will need to create this file), add this simple line:

/^X-Spam-Level: \*{5,}.*/ DISCARD spam

then in /etc/postfix/main.cf:

header_checks = regexp:/etc/postfix/header_checks

This will drop the spam, but you may want to only drop higher level spam, so instead you could change the 5 to a 7, and then add to your /etc/spamassassin/local.cf (might already be there commented out):

rewrite_header Subject *****SPAM*****

This makes it so that any spam that doesn’t get dropped, has SPAM prepended to the header, which Gmail suggests you do if you do end up forwarding spam to Gmail.

With this approach, low score (5-6) spam will be forwarded but makes Gmail happy that you told them its spam ahead of time, and 7+ spam won’t even bother forwarding.

Taking these steps will help you maintain a good mail sending reputation (Hopefully I don’t have to repair mine too much…). Good luck 🙂

Final note for Gmail users

And one final step if you are using Gmail, ensure EVERY email address that you receive mail from that is forwarded to Gmail is added as a “Send Mail As” account. Gmail uses this list to know it is a forwarded address, and will be more lenient in spam rules. I don’t know if other ESP’s do this, but Gmail has requested you do this if you forward mail to them.

0

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 221 other subscribers

I am Senior Software Engineer and Entrepeneur. I am an enthusiast and love creating things. I operate my own side company in my free time called Starlis LLC, working in Minecraft.

I enjoy doing things right and learning modern technologies.