Ways of Blindness

We want hard 

And feel fast

and overlook 

the others' grief

others' sorrow 

We can't see 

Because we look

and want to look

above the rest

actively miss him

or mishear her 

Where would we

Get calm and reflect? 

is not the other

real too?

What about inside?

Where does the look go

that we miss

actively lie

by misseeing

When life is short

And hearing takes long

And what we miss

Will not repeat 

Every sight

Every together

Every one

Counts

Oneita

he told his mom, I think she might... be the one

she told him... go and buy a ring

she drove past when he was getting home

before he knew her friends would come 

they met, they saw, they joined as one

but then he saw the writing on the wall and signed

she went with and welcomed little Jim

then back home where Steph came in

he saw and welcomed her ambitious words

she'd wait again, when he wasn't home

they wanted more, but then they came another way

two red heads, first John, then Phil

and growing up they brought their light

and shone and burned, she loved them right

but then he could not drive himself back home

it broke her heart when Phil was gone

but she still had Jim, and John

they'd call, invite and hold so many little ones 

who all grew up, and left to live their own

then off to England, to help them write

evr'y name and date and place of 1881,

and back again, and to Nauvoo to count, and write, and do

but crack, it didn't hold, and Jim was gone 

way before his due

when Milton came she welcomed him

they lived and loved a wave of little ones 

the heart that loved then beat no more

and quiet then, and lonely too

and slower then, and slower too

she loved and cared and started to review

and count and tell and hold all them

all those who wandered through 

then two angels came to care and live and watch

they stayed, she lingered there

then softly and quietly, in the night, she too was gone

she'd taken flight, and hoped that Jim would hug her tight 

Procrastination is Your Friend

My friend told me:

    Procrastination is your friend when it's time for bed.

I think that is a true statement.

Stress on a Cracked Foundation

Think about a house standing strong.  Imagine a crack in the foundation.  Maybe the ground under the house has settled unevenly.  Maybe things have shifted since the house was built.

If it has gone long enough that the foundation is cracked, there is a failure waiting to happen.  Putting enough stress on a cracked foundation will lead to a dramatic failure.

This is like living at the edge of health: physical, social, emotional, or spiritual.  If you aren't constantly investing in repairing and strengthening your foundation, you are more susceptible to unexpected failure.

I want to eat healthy food, keep in touch with good friends, keep enough space in my life, and stay strong in my faith.  So that when the storms of life come, I can get through the challenging times.  When I ask God for help, I can be more confident I will be able to receive His help to bridge me back to a stable and happy future.

Discover, Receive, Commit

Yesterday, I was learning with my peers in a priesthood quorum at church.  The topic was answered prayer.

The combined message of three scriptures stood out to me:
- Matt 6:8 discover vs beg
- Jacob 4:10 receive vs command
- James 1:5-7 commit vs worry

The combination of receiving God's help, discovering what He has in store, and being pre-committed to act on His generous & challenging prompting is a magic combination to me.

It helps me feel a sense of sufficiency when facing my set of challenges today.  I'm grateful for the learning & strengthening environment afforded to me in my priesthood quorum.

Tuning G1 GC for Cassandra

Tuning G1 GC for Cassandra is too complicated, but it can make a big difference in cluster health.

Symptoms:

• High p99 read/write latencies (because of long GC pauses)
• High CPU causing lower read throughput (because of low GC throughput)
• Dropped mutations (because of full GC collections on write-heavy clusters)

Here are some options that made a difference for me:

• JVM: options for getting GC details out for inspection
  -XX:+PrintGCDetails
  -XX:+PrintGCDateStamps
  -Xloggc:/var/log/cassandra/gc.log
• JVM: options for having enough buffer for collections
  # Pre-allocate full heap
  # Pre-size new size for high-throughput young collections
  -Xms24G
  -Xmx24G
  -Xmn8G
• JVM: options for avoiding longer pauses (do reference scanning concurrently with app)
  # Have the JVM do less remembered set work during STW, instead
  # preferring concurrent GC.
  -XX:G1RSetUpdatingPauseTimePercent=5
  # Scan references in parallel to avoid long RSet scan times
  -XX:+ParallelRefProcEnabled
• JVM: options for better young collection throughput (avoid copying short-lived objects)
  # Save CPU time by avoiding copying objects repeatedly
  # Improve collection throughput by making heap regions larger
  -XX:MaxTenuringThreshold=1
  -XX:G1HeapRegionSize=32m
• JVM: option cocktail to reduce risk of long mixed collections
  # Avoid to-space exhaustion by starting sooner, capping new size, and being more aggressive during mixed collections
  -XX:InitiatingHeapOccupancyPercent=40
  -XX:+UnlockExperimentalVMOptions
  -XX:G1MaxNewSizePercent=50
  -XX:G1MixedGCLiveThresholdPercent=50
  -XX:G1MixedGCCountTarget=32
  -XX:G1OldCSetRegionThresholdPercent=5
  # Reduce pause time target to make mixed collections shorter
  -XX:MaxGCPauseMillis=300
• JVM: option to get extra buffer for use in allocation emergency
  # Reserve extra heap space to reduce risk of to-space overflows
  -XX:G1ReservePercent=20
• JVM: options for top collection throughput during pauses
  # Max out the parallel effort during pause
  # Set to number of cores
  -XX:ParallelGCThreads=16
  -XX:ConcGCThreads=16
• Cassandra: option to avoid excess spikes of garbage from compaction
  # Reduce load of garbage generation & CPU used for compaction
  compaction_throughput_mb_per_sec: 2
• Cassandra: option to aggressively flush to disk on write-heavy clusters
  # Reduce amount of memtable heap load to reduce object copying
  memtable_heap_space_in_mb: 1024  # instead of default 1/3 heap

The net effect of the above combined settings is as follows:

• for a read-heavy cluster on i3.4xlarge:
• young collection p90 pause times around 50ms
• mixed collection p90 pause times around 90ms
• no Full GCs, no dropped mutations
• for write-heavy clusters on r5.2xlarge:
• young collection p90 pause times around 175ms
• mixed collection p90 pause times around 175ms
• no Full GCs, no dropped mutations

Tuning process:

1. Turn on GC logging
2. Gather pause times for young collections, mixed collections, and any full collections
• get logs for at least 2-3 cycles of young => mixed/full transitions
3. Decide which of the above you want to optimize for, pick a single set of settings
• Apply the settings to one node on one rack
• Decide whether it had the desired effect
• Tweak and repeat on single node until you get to a stable point
4. Apply settings to all nodes on one rack
• Wait for a peak traffic period or apply stress
• Compare results from non-tuned racks with the tuned rack
• Tweak and repeat on single rack until settings are rock solid
5. Apply settings to full cluster
• Wait for a peak traffic period or apply stress
• Make sure settings are rock solid for full cluster
6. Start again on step 2 until you have nothing left to tune

Another Poem, With You

With You

Piece of pie
Clear blue sky
Mountain lakes
Leaves, and rakes

And when I am with you,
Two hearts filled with joy.
Our time passes softly,
Two hearts, one girl, one boy.

Teach me love
Teach me caring
I'll teach you happiness
I'll teach you daring

And when I am with you,
Two hearts filled with joy.
Our time passes softly,
Two hearts, one girl, one boy.