Tuning G1 GC for Cassandra

Tuning G1 GC for Cassandra is too complicated, but it can make a big difference in cluster health.

Symptoms:

• High p99 read/write latencies (because of long GC pauses)
• High CPU causing lower read throughput (because of low GC throughput)
• Dropped mutations (because of full GC collections on write-heavy clusters)

Here are some options that made a difference for me:

• JVM: options for getting GC details out for inspection
  -XX:+PrintGCDetails
  -XX:+PrintGCDateStamps
  -Xloggc:/var/log/cassandra/gc.log
• JVM: options for having enough buffer for collections
  # Pre-allocate full heap
  # Pre-size new size for high-throughput young collections
  -Xms24G
  -Xmx24G
  -Xmn8G
• JVM: options for avoiding longer pauses (do reference scanning concurrently with app)
  # Have the JVM do less remembered set work during STW, instead
  # preferring concurrent GC.
  -XX:G1RSetUpdatingPauseTimePercent=5
  # Scan references in parallel to avoid long RSet scan times
  -XX:+ParallelRefProcEnabled
• JVM: options for better young collection throughput (avoid copying short-lived objects)
  # Save CPU time by avoiding copying objects repeatedly
  # Improve collection throughput by making heap regions larger
  -XX:MaxTenuringThreshold=1
  -XX:G1HeapRegionSize=32m
• JVM: option cocktail to reduce risk of long mixed collections
  # Avoid to-space exhaustion by starting sooner, capping new size, and being more aggressive during mixed collections
  -XX:InitiatingHeapOccupancyPercent=40
  -XX:+UnlockExperimentalVMOptions
  -XX:G1MaxNewSizePercent=50
  -XX:G1MixedGCLiveThresholdPercent=50
  -XX:G1MixedGCCountTarget=32
  -XX:G1OldCSetRegionThresholdPercent=5
  # Reduce pause time target to make mixed collections shorter
  -XX:MaxGCPauseMillis=300
• JVM: option to get extra buffer for use in allocation emergency
  # Reserve extra heap space to reduce risk of to-space overflows
  -XX:G1ReservePercent=20
• JVM: options for top collection throughput during pauses
  # Max out the parallel effort during pause
  # Set to number of cores
  -XX:ParallelGCThreads=16
  -XX:ConcGCThreads=16
• Cassandra: option to avoid excess spikes of garbage from compaction
  # Reduce load of garbage generation & CPU used for compaction
  compaction_throughput_mb_per_sec: 2
• Cassandra: option to aggressively flush to disk on write-heavy clusters
  # Reduce amount of memtable heap load to reduce object copying
  memtable_heap_space_in_mb: 1024  # instead of default 1/3 heap

The net effect of the above combined settings is as follows:

• for a read-heavy cluster on i3.4xlarge:
• young collection p90 pause times around 50ms
• mixed collection p90 pause times around 90ms
• no Full GCs, no dropped mutations
• for write-heavy clusters on r5.2xlarge:
• young collection p90 pause times around 175ms
• mixed collection p90 pause times around 175ms
• no Full GCs, no dropped mutations

Tuning process:

1. Turn on GC logging
2. Gather pause times for young collections, mixed collections, and any full collections
• get logs for at least 2-3 cycles of young => mixed/full transitions
3. Decide which of the above you want to optimize for, pick a single set of settings
• Apply the settings to one node on one rack
• Decide whether it had the desired effect
• Tweak and repeat on single node until you get to a stable point
4. Apply settings to all nodes on one rack
• Wait for a peak traffic period or apply stress
• Compare results from non-tuned racks with the tuned rack
• Tweak and repeat on single rack until settings are rock solid
5. Apply settings to full cluster
• Wait for a peak traffic period or apply stress
• Make sure settings are rock solid for full cluster
6. Start again on step 2 until you have nothing left to tune

Another Poem, With You

With You

Piece of pie
Clear blue sky
Mountain lakes
Leaves, and rakes

And when I am with you,
Two hearts filled with joy.
Our time passes softly,
Two hearts, one girl, one boy.

Teach me love
Teach me caring
I'll teach you happiness
I'll teach you daring

And when I am with you,
Two hearts filled with joy.
Our time passes softly,
Two hearts, one girl, one boy.

Another Poem, Longer With You

Longer With You

It took a little bit of time
for me to see and make them rhyme,
and when beneath the pen, the words, they stuck,
it took a bit more time to see, and pluck
them out for you, to read,
and us to hear, and laugh,
and spend a day, a year
together, and whether you want to
or need to depart, the mem'ry
of this time together
might smart,
but mem'ry is short
and time will start to pull
and stretch and fill our heart
with new times together
when never we part.

Another Poem, Knowing Where

Knowing Where

Came home this evening
  at quarter to six.
Met with the Bishop
  who told us to fix
sev'ral mistakes
  on forms one, five, and six.

We're going on a mission to
  serve our dear Lord.

Where will He send us?
  Where will we go?
Over the oceans,
  or closer to home?
Who will we serve with?
  How long to be gone?

We're going on a mission to
  Serve our dear Lord.

Fam'ly will miss us;
  we'll miss them too.
Places we've lived in will
  see someone new.
Trust in our Savior, and
  live in His care.

We're going on a mission, and
  He'll meet us there.

Another Poem, Riff Off 1 Cor 13

Perfect?

Completeness:
What I sought
when I was small.

Clarity:
What I thought
that I could call.

Calm:
What was not
were I to fall.

But if it fails,
or vanishes away,
or if I'm blind,
or mostly so today.

There is a promised tomorrow,
when that completeness will come
when I will see face to face
and know as I am known.

Another Poem, Finding Encouragement

Here is another poem, born out of a desire to help my daughter get through the last few weeks of high school.


Born Again

Say your piece.
Toot your horn!
Do your thing.
Sing your song!

Then look and see
If it was what
You meant to say or do
Then cut.

And if you meant to do it different,
Or if you notice something new,
Then aren't you glad you started out,
With confidence to see the world,
To learn and do?


Sometimes it takes a lot of confidence to overcome the resistance you feel to take the next step in life.

Another Poem, Hymn Text

Here is a poem that came out when I couldn't sleep and was thinking about an earlier attempt at writing a hymn for the Church of Jesus Christ of Latter-day Saints, of which I am a member.


Learning and Growing

Learning and growing are part of life's purpose,
Even when happiness seems far away.
Peace in this life comes from completely accepting
The plan of our Savior, beginning His way.

Jesus desires to teach us and guide us,
He will protect us as we find our way.

When evil conquers it cannot remain long,
If faced with commitment and faith in the Lord.
Confidence, trust in our saving Companion,
Is what will enable our learning His Word.

Jesus desires to teach us and guide us,
He will protect us as we find our way.


We are making another hymnbook, and I think I'll submit it.  See https://www.lds.org/church/news/church-announces-plans-for-new-hymnbook-and-childrens-songbook?lang=eng.

Finally Stable Caps to Ctrl Mapping

Over the years, I have tried so many ways on Linux to map Caps Lock to Ctrl:

• Xorg config (that didn't work on a tty console)
• Inputrc (really weird rules that depended on initial state)
• Gnome keyboard config

Finally there is a stable way to remap Caps Lock to Ctrl.  It is relatively simple and it is at the lowest level in Linux.

Thanks to this page for all the details: https://wiki.archlinux.org/index.php/Map_scancodes_to_keycodes but it took a lot of effort to parse out what actually needed to be done.

Here is a distilled version:

0) Make sure Caps Lock is OFF

1) Create the following file:

/etc/udev/hwdb.d/10-caps-modifier.hwdb

---

evdev:atkbd:dmi:*            # built-in keyboard: match all AT keyboards for now

  KEYBOARD_KEY_3a=leftctrl   # bind capslock to leftctrl

2) Run  the following command:

$ systemd-hwdb update

3) Run the following command to make the remapping take immediate effect:

$ udevadm trigger

This was tested on Ubuntu 18.04.2 LTS running Xfce.  The remapping works great in tty consoles, and the X input system.

Poem for Grandma

I wrote a poem for my Grandma for Mother's Day.

Years

For when along
the road you've come,
you pause,
to see the race,
how it was run.

The mornings when imaginations sang
The afternoons when, interrupting
work and sun, the lightning cracked.
Then came the rain.
The evenings when all work was done.

Some goals were met.
Some riches won.
Some people left.
And not just one.

And then you stood.
Alone.
Dismayed.

But through
this turn and
that meander,
we kept,
we stayed,
we held each other.