minecraft

Tuning the JVM – G1GC Garbage Collector Flags for Minecraft

After many weeks of studying the JVM, Flags, and testing various combinations, I have come up with these flags as the most ideal combination to use, backed with SCIENCE.

I strongly suggest these flags to start your server (Which BTW: You really should be using Paper instead of Spigot: https://paper.emc.gs – Paper is a drop in replacement for Spigot and all plugins should still work – but with paper, please do not ask for support in #spigot IRC channel – ask in #paper )

Use these flags exactly (only modify the Xmx and Xms) for max memory of 10GB and LOWER. These flags work and scale accordingly to any size of memory, even 500MB)

These flags help keep your server running CONSISTENT without any large spikes. CPU may be slightly higher, but your server will be overall more reliable and stable TPS.

If you are running with 10GB or less memory for MC, you should not adjust these parameters. (I use 10GB myself)

If you for sure need more than 10GB (Hopefully you are 150+ player server) use these changes:

    • -XX:G1MaxNewSizePercent=60
    • -XX:G1NewSizePercent=35
    • -XX:InitiatingHeapOccupancyPercent=15

Explanation of flags:

  1. -Xms matching -Xmx – Why: You should never run your server with the case that -Xmx can run the system completely out of memory. Your server should always be expected to use the entire -Xmx! You should then ensure the OS has extra memory on top of that Xmx for non MC/OS level things. Therefore, you should never run MC with -Xmx settings you can’t support if java uses it all.Now, that means if -Xms is lower than -Xmx – YOU HAVE UNUSED MEMORY! Unused memory is wasted memory. G1 (and probably even CMS to a certain threshhold, but I’m only stating what I’m sure about) operates better with the more memory its given. G1 adaptively chooses how much memory to give to each region to optimize pause time. If you have more memory than it needs to reach an optimal pause time, G1 will simply push that extra into the old generation and it will not hurt you (This may not be the case for CMS, but is the case for G1)

    The fundamental idea of improving GC behavior is to ensure short lived objects die young and never get promoted. With the more memory G1 has, the better assurance you will get that objects are not getting prematurely promoted to the old generation.

    G1 Operates differently than previous collectors and is able to handle larger heaps more efficiently. If it does not need the memory given to it, it will not use it. The entire engine operates differently and does not suffer from too large of heaps.

  2. UnlockExperimentalVMOptions – needed for some of the others specified
  3. TargetSurvivorRatio: I’m sure your all use to seeing this one suggested. Good news! It’s actually a good flag to use :DThis setting controls how much of the Survivor space is ABLE to be used before promotion. If survivor gets too full, stuff starts promoting to Old Gen. The reason behind this is to be able to handle memory allocation spikes.However, MC allocation rate for most part is pretty steady (steadily high…..), and when its steady its safe to raise this value to avoid premature promotions.
  4. G1NewSize Percent: These are the important ones. In CMS and other Generations, tweaking the New Generation results in FIXED SIZE New Gen and usually is done through explicit size setting with -Xmn.With G1, things are better! You now can specify percentages of an overall desired range for the new generation.

    With these settings, we tell G1 to not use its default 5% for new gen, and instead give it 50% at least!

    Minecraft has an extremely high a memory allocation rate, ranging to at least 800 Megabytes a second on a 30 player server! And this is mostly short lived objects (BlockPosition)

    now, this means MC REALLY needs more focus on New Generation to be able to even support this allocation rate. If your new gen is too small, you will be running new gen collections 1-2+ times per second!!!

    This is bad! You will have so many pauses that TPS has risk of suffering, and Spigot might be unable to catch up TPS with the cost of GC’s.

    Then combine the fact that objects will now promote faster, resulting in your Old Gen growing faster…. This is bad and needs to be avoided.

    Given more NewGen, we are able to slow down the intervals of Young Gen collections, resulting in more time for short lived objects to die young and overall more effecient GC behavior.

    if you run with larger heaps (15GB+), you may want to lower the minimum to say 30%, but don’t go lower than 30%. This will let G1 have more power in its own assumptions.

  5. InitiatingHeapOccupancyPercent/G1MixedGCLiveThresholdPercent: Controls when to include Mixed GC’s in the Young GC collection, keeping OldGen tidy without doing a normal OldGen GC collection.On larger heaps(10GB+), you can raise InitiatingHeap to around 20 to reduce CPU usage, but I wouldn’t go higher than that. And you also need to REDUCE the Maximum New Percentage to around 60. If you use 80% NewGen Max, you must keep this at 10 then. It doesn’t hurt to leave it at 10, but “effeciency wise” you can improve it to 20 if you reduce your new gen size, but if you start seeing Old Gen GC, lower it back.
  6. AlwaysPreTouch: AlwaysPreTouch gets the memory setup and reserved at process start ensuring it is contiguous, improving the efficiency of it more.

Also for Large Pages – IT’s even more important to use -Xms = -Xmx! Large Pages needs to have all of the memory specified for it or you could end up without the gains. That memory CAN NOT be used by the OS anyways, so let something use it!
Additionally use these flags (Metaspace is Java8 Only, don’t use it for Java7):

Code:
 -XX:+UseLargePagesInMetaspace

Thanks to https://product.hubspot.com/blog/g1gc-fundamentals-lessons-from-taming-garbage-collection for helping reinforce my understanding of the flags and introduce improvements!

————-
Update 5/24/2018: Added -XX:+ParallelRefProcEnabled

0

Advanced Minecraft Marketing Analytics

I’ve mentioned a few times the power of my analytics system in the Spigot community and recently in AdminCraft on Reddit. Most advertising campaigns simply track impressions and clicks. But to a server owner, we need more data than that…. We need to know who actually joined the server and continues to play!

I have designed such a system, where I can track clicks, how many joined the game server, then out of those, who stay to the day 5, 15, 30, 60 and 90 markers! I can see when campaigns generate a lot of clicks, yet low players, or worse: players who don’t actually stay.

This article will give a rough idea on how I did it incase someone is curious to implement it on their own.

I want to be straight that the implementation isn’t perfect or the most optimized, but hey it gives you data where you normally would not have data, so that’s a clear winner!

Requirements / Disclaimer

This system only works for advertising campaigns that involve website clicks. Campaigns that serve purely as a banner and a server connect address can not be realistically tracked.

My implementation all revolves around Google Analytics, but the concept can work for any form of marketing system you want to adapt it to.

You should be using analytics.js (I do not think this works with ga.js, the old Analytics system) on every page of your site.

You will also need to be an experienced developer or have one on hand.

I WILL NOT HELP ANYONE IMPLEMENT THIS. I do not do contract work / consulting services, but feel free to ask questions on Spigot IRC.

Tracking that click

First off, my system requires that all inbound campaigns need to land the user on the website. If someone see’s your server connect address and connects directly, then you can’t track that – but hey! That’s usually a FREE conversion , so don’t complain 🙂

Here is an example database table that I use:

CREATE TABLE `conversions` (
 `clientid` varchar(255) CHARACTER SET latin1 NOT NULL,
 `convert_id` varchar(255) CHARACTER SET latin1 NOT NULL,
 `ip` int(10) unsigned NOT NULL,
 `ua` varchar(255) CHARACTER SET latin1 DEFAULT NULL,
 `date` int(11) unsigned NOT NULL,
 `lp` text CHARACTER SET latin1,
 `converted` tinyint(4) unsigned NOT NULL DEFAULT '0'
 PRIMARY KEY (`clientid`),
 KEY `ip` (`ip`),
 KEY `converted` (`converted`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

our goal is to assign a Google Analytics Client Id to every user. Additionally, I assign a unique ID to every user separate to the client ID, as some users may have addons that constantly change their Google Analytics Client ID, so this prevents inserting an entry into DB every time their client id changes.

When a user comes in and has a Google Client ID set, assign them a convert ID. then insert an entry into the database.

Explanation of fields:

  • clientid: Google Analytics Client ID
  • convert_id: Field you generate to associate to a session
  • ip: users IP address – to be passed later to Google Analytics for conversion and match to a newly joined player
  • ua: Users User Agent – to be passed later to Google Analytics for conversion
  • date: current time when seen
  • lp: URL the player first landed on – so you can know what web page they visited.
  • converted: has this record triggered a conversion not.

So, once you have their IP and GA client ID tagged into the database off that first page load… You got them 🙂 If they join, you will know what campaign they came from.

Monitoring Conversions

Now that you have them in the database, you need to have a monitoring system that checks your game server. This is the trickiest part of pulling this off, as how you do this will really vary based on your servers data infrastructure.

We have a users database table that we can easily query on and find who has played in the past few hours and has not triggered conversion events. So we run a query, get the users IP address, and then look for matching conversion table entries on the same IP address that is also not marked as converted.

In the event 2 people joined on same IP from 2 diff devices, you would have 2 entries in the conversions table, and then each would match up to each other, still resulting in 2 conversions. If 2 accounts join but only 1 device hit the website, then only 1 conversion should be marked initially, but they are likely to then trigger a 2nd conversion event once they hit the website with a new device (that doesn’t match the convert_id you generated).

Now… the actual marking of a conversion when this happens!

Setting up your conversion event

Now you need to configure Google Analytics to have a goal to count the conversions.

Go to Admin > Your Property > Your Primary View > Goals

Add a new Goal using an Event like so:

This step doesn’t have to be done first, but it does need to be done to view it and there’s no benefit to do it after rather than before.

Google Analytics Measurement Protocol

The way the Google Analytics tracking system works is extensively documented, and even encouraged for you to do your own backend server side reporting of the event on behalf of the user!

Read all the documentation: Google Analytics Measurement Protocol.

Now we’ve got a list of rows from the conversion table of who joined the game server, with their GA Client Id IP and URL.

Now your backend monitoring system needs to call out to Google Analytics in the exact same fashion the client browsers do, but I’ll save you the trouble, here is the code I use to do this:

 

I do even further events like Day 5/15/30/60/90 tracking, so I pass in other events for those. To support that type of data, you will have to analyze your data and figure out the best way to compute who should be converted for those cases.

Tagging Every Inbound Link

Now you have the infrastructure set up, you need to ensure every direct campaign you control is tagged! This involves setting utm_* parameters on your inbound links.

Learn more about UTM Parameters

I personally use our emc.gs url shortner, then create links that redirect to pages with the utm parameters, so that I can do “reddit.emc.gs” and it will tag that link with the Reddit tracking parameters. If you want to set up a url shortener like we have, check out YourLS.

I have modified it for our needs though, making it so &gac = &utm_campaign, &gas = &utm_source and &gam  = &utm_medium for quick changes of properties like reddit.emc.gs/?gam=ad-4

I suggest setting up a redirect system on your main website domain and not an alternate domain like we do, because many advertising systems dislike domain mismatches, and we had to set up empireminecraft.com/go/foo to redirect to emc.gs/foo which then redirects back to the target of emc.gs/foo just to trick some of those platforms.

Enjoy having data about your campaigns and who sends you the best traffic 🙂

0

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 221 other subscribers

I am Senior Software Engineer and Entrepeneur. I am an enthusiast and love creating things. I operate my own side company in my free time called Starlis LLC, working in Minecraft.

I enjoy doing things right and learning modern technologies.