Procrastination is Your Friend

My friend told me:

    Procrastination is your friend when it's time for bed.

I think that is a true statement.

Stress on a Cracked Foundation

Think about a house standing strong.  Imagine a crack in the foundation.  Maybe the ground under the house has settled unevenly.  Maybe things have shifted since the house was built.

If it has gone long enough that the foundation is cracked, there is a failure waiting to happen.  Putting enough stress on a cracked foundation will lead to a dramatic failure.

This is like living at the edge of health: physical, social, emotional, or spiritual.  If you aren't constantly investing in repairing and strengthening your foundation, you are more susceptible to unexpected failure.

I want to eat healthy food, keep in touch with good friends, keep enough space in my life, and stay strong in my faith.  So that when the storms of life come, I can get through the challenging times.  When I ask God for help, I can be more confident I will be able to receive His help to bridge me back to a stable and happy future.

Discover, Receive, Commit

Yesterday, I was learning with my peers in a priesthood quorum at church.  The topic was answered prayer.

The combined message of three scriptures stood out to me:
- Matt 6:8 discover vs beg
- Jacob 4:10 receive vs command
- James 1:5-7 commit vs worry

The combination of receiving God's help, discovering what He has in store, and being pre-committed to act on His generous & challenging prompting is a magic combination to me.

It helps me feel a sense of sufficiency when facing my set of challenges today.  I'm grateful for the learning & strengthening environment afforded to me in my priesthood quorum.

Tuning G1 GC for Cassandra

Tuning G1 GC for Cassandra is too complicated, but it can make a big difference in cluster health.

Symptoms:

• High p99 read/write latencies (because of long GC pauses)
• High CPU causing lower read throughput (because of low GC throughput)
• Dropped mutations (because of full GC collections on write-heavy clusters)

Here are some options that made a difference for me:

• JVM: options for getting GC details out for inspection
  -XX:+PrintGCDetails
  -XX:+PrintGCDateStamps
  -Xloggc:/var/log/cassandra/gc.log
• JVM: options for having enough buffer for collections
  # Pre-allocate full heap
  # Pre-size new size for high-throughput young collections
  -Xms24G
  -Xmx24G
  -Xmn8G
• JVM: options for avoiding longer pauses (do reference scanning concurrently with app)
  # Have the JVM do less remembered set work during STW, instead
  # preferring concurrent GC.
  -XX:G1RSetUpdatingPauseTimePercent=5
  # Scan references in parallel to avoid long RSet scan times
  -XX:+ParallelRefProcEnabled
• JVM: options for better young collection throughput (avoid copying short-lived objects)
  # Save CPU time by avoiding copying objects repeatedly
  # Improve collection throughput by making heap regions larger
  -XX:MaxTenuringThreshold=1
  -XX:G1HeapRegionSize=32m
• JVM: option cocktail to reduce risk of long mixed collections
  # Avoid to-space exhaustion by starting sooner, capping new size, and being more aggressive during mixed collections
  -XX:InitiatingHeapOccupancyPercent=40
  -XX:+UnlockExperimentalVMOptions
  -XX:G1MaxNewSizePercent=50
  -XX:G1MixedGCLiveThresholdPercent=50
  -XX:G1MixedGCCountTarget=32
  -XX:G1OldCSetRegionThresholdPercent=5
  # Reduce pause time target to make mixed collections shorter
  -XX:MaxGCPauseMillis=300
• JVM: option to get extra buffer for use in allocation emergency
  # Reserve extra heap space to reduce risk of to-space overflows
  -XX:G1ReservePercent=20
• JVM: options for top collection throughput during pauses
  # Max out the parallel effort during pause
  # Set to number of cores
  -XX:ParallelGCThreads=16
  -XX:ConcGCThreads=16
• Cassandra: option to avoid excess spikes of garbage from compaction
  # Reduce load of garbage generation & CPU used for compaction
  compaction_throughput_mb_per_sec: 2
• Cassandra: option to aggressively flush to disk on write-heavy clusters
  # Reduce amount of memtable heap load to reduce object copying
  memtable_heap_space_in_mb: 1024  # instead of default 1/3 heap

The net effect of the above combined settings is as follows:

• for a read-heavy cluster on i3.4xlarge:
• young collection p90 pause times around 50ms
• mixed collection p90 pause times around 90ms
• no Full GCs, no dropped mutations
• for write-heavy clusters on r5.2xlarge:
• young collection p90 pause times around 175ms
• mixed collection p90 pause times around 175ms
• no Full GCs, no dropped mutations

Tuning process:

1. Turn on GC logging
2. Gather pause times for young collections, mixed collections, and any full collections
• get logs for at least 2-3 cycles of young => mixed/full transitions
3. Decide which of the above you want to optimize for, pick a single set of settings
• Apply the settings to one node on one rack
• Decide whether it had the desired effect
• Tweak and repeat on single node until you get to a stable point
4. Apply settings to all nodes on one rack
• Wait for a peak traffic period or apply stress
• Compare results from non-tuned racks with the tuned rack
• Tweak and repeat on single rack until settings are rock solid
5. Apply settings to full cluster
• Wait for a peak traffic period or apply stress
• Make sure settings are rock solid for full cluster
6. Start again on step 2 until you have nothing left to tune

Another Poem, With You

With You

Piece of pie
Clear blue sky
Mountain lakes
Leaves, and rakes

And when I am with you,
Two hearts filled with joy.
Our time passes softly,
Two hearts, one girl, one boy.

Teach me love
Teach me caring
I'll teach you happiness
I'll teach you daring

And when I am with you,
Two hearts filled with joy.
Our time passes softly,
Two hearts, one girl, one boy.

Another Poem, Longer With You

Longer With You

It took a little bit of time
for me to see and make them rhyme,
and when beneath the pen, the words, they stuck,
it took a bit more time to see, and pluck
them out for you, to read,
and us to hear, and laugh,
and spend a day, a year
together, and whether you want to
or need to depart, the mem'ry
of this time together
might smart,
but mem'ry is short
and time will start to pull
and stretch and fill our heart
with new times together
when never we part.

Another Poem, Knowing Where

Knowing Where

Came home this evening
  at quarter to six.
Met with the Bishop
  who told us to fix
sev'ral mistakes
  on forms one, five, and six.

We're going on a mission to
  serve our dear Lord.

Where will He send us?
  Where will we go?
Over the oceans,
  or closer to home?
Who will we serve with?
  How long to be gone?

We're going on a mission to
  Serve our dear Lord.

Fam'ly will miss us;
  we'll miss them too.
Places we've lived in will
  see someone new.
Trust in our Savior, and
  live in His care.

We're going on a mission, and
  He'll meet us there.