Metrics

ROI

How do you measure success?  Try creating a list of and prioritizing.  What comes to the top?  Is it quantifiable?  Image an arts organization.  Do they measure success by the number of attendees?  How beautiful or though-provoking the pictures are?  The media coverage of the exhibit?  Letter to the editor (either for or against)?  Is it the ‘you must see’ event in your community?  The donations received?

What is your metric for measuring success?

If you sat in a moving vehicle watching just the GPS display without looking out the window, how would it change your perception of travel?  My two kids create this virtual journey for themselves on the way to school.  They were fascinated to watch the route were taking, actual speed, and direction of travel.  Suddenly they shouted, ‘you are speeding!’  The GPS listed the posted speed limit on the screen and they noticed that we were now going 40 MPH in a 35 MPH zone.  What they did not see was that we were in a line of traffic that was all moving at this speed. 

One of the challenges with snapshots, dashboards, or any measurement instrument is that they do not tell the whole story.  The GPS unit provides accurate location and selected data.  What is does not address is the environmental issues.  Is there snow on the roads?  Am I being tailgated or is the driver in front directionally challenged?  Is there a passing zone or what is the speed of moving traffic.  If our driving skills were evaluated just in the context of watching a GPS unit there may be periods of alarm.  But when the realities of the real-time variables were considered, an appropriate course of action is being executed.


What do you measure?  How do you evaluate the data?  What questions does it raise?

When one does not equal one

I have had the fortune of staying at Hyatt hotels while on business for the past couple of years.  They have been generous in how they accommodate me and I have been loyal.  Recently, my membership level was downgraded and according to the membership agreement they were correct.  I spoke to another frequent guest who experienced the same demotion.  He had stayed with Hyatt around the world in the past year.  The cost of the rooms he booked at a Park Hyatt in Europe, Asia and Australia often went for 15 times or more compared to one night at the local Hyatt Place.  Hyatt institutionalizes a formula that the number of nights stayed is the metric that determins your loyalty level.  So a $60 night is the same as a $800 night.  Would you rather have a customer staying 50 nights at $79 a night ($3,950 gross) or a guest lodging 30 nights at $800 per night ($24,000 gross)?

Not all things are equal.  Be thoughtful about how you measure loyalty.  I am now shopping around for alternatives on my coming trips because the benefit of staying with Hyatt can be easily replicated by another chain.  Loyalty is not always about the most, it may be about how sticky you can make an experience.  One poor decision can dissolve years of loyalty.  Reward loyalty but also understand what behavior you are supporting.

What Now?

If you were unable to use numerical or financial metrics that your organization commonly relies on to demonstrate its effectivness, how would you describe your cause’s impact?

  • If you could not reference the amount of money raised, what ways would you describe the success of your fundrasing gala?
  • If you had no attendance numbers, how could you paint a picture that relates to the success of a concert or lecture you produced?
  • If you could not list the names or donation levels, what type of annual report would your organization produce?
  • If someone asked you to produce a 30-second YouTube video highlighting your cause, what images would you show if you could not use words.


Measuring Success or Something

“Not everything that can be counted counts, and not everything that counts can be counted”

John Allen Paulos, The Way We Live
 The New York Times Magazine, May 16, 2010

There is a fascination with measuring our progress or lack there of in all facets of our lives and those of the organizations that employee us.  Many of the fundraisers in the nonprofit world keep track of the number of donors, the dollars contributed, the average size of gift, the number of donors who increased or decreased their gift, and the favorite category of LYBNTs (Last Year But Not This).  I am reminded of Jim Collin’s example in, Good to Great and the Social Sector where he highlighted a cultural organization that measured success by annecdotaly tracking number of taxi drivers who recommended the organization’s programs when they drove out-of-town guests from the airport to downtown.  The Robin Hood Foundation in New York employees an economist full-time to measure the potential impact a program might have on the future earning power of youth living in poverty.

A key conversation Mr. Paulos highlights is the subtle shift that weighted criteria can have on the final ranking.  It sounds simple but in fact many of us are so desperate to quantify our cause’s impact that we gobble-up the data without thinking about its context.  As Mr. Paulos suggest, “counting and aggregating- have important implications for public policy.”  

What if we asked the following questions before we every looked at the bar graphs, pie charts and reports, ‘What did we intend to measure?  What assumptions did we make?  What criteria did we use?  What flaws are inherent with the data?’

Measuring Impact

I use a metric to measure the exertion of my cross-country ski training (at least when I am not recovering from viral meningitis, then I just use the “am I standing upright” measurement).  I outline a purpose for each workout and give it a percentage grade when I am done with 100% representing my workout matching perfect my purpose.  I multiple that by the duration of the workout, 0.25 to 5 hours and then multiple by the intensity with a scale of 1 very easy and 5 a maximum effort.  An example of the metric: my purpose the other day was to ski a long distance workout at level 2 (relatively easy pace) with minimal stops and focus on a specific skiing technique.  In reality I skied 3.5 hours and focused on technique but the intensity was higher than I planned since I had to fight a headwind and skied harder than I had intended.  The calculation: purpose index is .75 (75% of my purpose) x 3.5 hours x 3 (level 3 intensity)= 7.87 index.  Had I honored my purpose for the workout the formula would have been 1.0 (100%) x 3.5 hours x 2 (level 2 intensity)= 7.0.  That does not mean the intended workout was less effective than the actual session but it puts a metric on the exertion.  I then can monitor the impact of the workouts on my races and recovery.


So how can this type of metric be applied to your enterprise?  Many nonprofits lack a perfect metric (as noted by Jim Collins in his Good to Great and the Social Sector, a follow-up book to his best seller Good to Great).  So we invent our own.  Jim notes that a symphony in the Midwest decided to anecdotally follow the number of taxi cab drivers who mentioned the symphony as an attraction while driving tourists from the airport to downtown.  It would be extremely difficult to track this as a metric.  The symphony’s assumption was that if a taxi driver was promoting the symphony then they were well positioned as an attraction in the community.  So what value works for your cause?  World Bicycle Relief could track the number of bikes distributed into an African community but if nobody is using the bikes then the metric is flawed.  So perhaps you track the number of miles ridden on the bike, the number of times the bike is ridden each day, the gross weight of the supplies carried on the bike, or the time saved by riding a bike compared to walking.  Perhaps these measurements get you more information than just the number on units distributed. 


Return on investment (ROI) is a common refrain in the nonprofit board room, a carryover from the corporate world.  In some cases this is a very effective measurement.  The Robin Hood Foundation targets poverty in New York City.  The foundation employees an economist to measure the effectiveness of their grants.  They have established an index that measure the cost-benefit ratio of their grants and support.  Their approach is very focused and scientific.  The majority of the foundation’s board members come from the corporate world and this is a language they speak with ease.  Presenting the information in this manner meets the culture of the board and staff and can be explained readily to the high-energy, high-fiance world of Wall Street.  They now their audience and they are delivering the message in a business terminology.  


What do you need to measure?  Who is your audience?  How can you most effectively monitor the impact of your programs?  Is it scientific or anecdotal?  Are you collecting right data?  Is it relevant to your donors and customers?