Measuring success

Is job retention the right success metric for us to measure? What might be a better measurement of success?

Tell us how YOU would measure success in this competition. We’d love for you to get creative and think outside the box!

@ymedan, @LiliGangas, @serinity, I’d like to ask your opinion on this.

The latest version of our prize design would challenge teams to develop and deploy a scalable training, job placement, and retention solution to:

  • Train 500 workers in 30 days for an occupation that provides a living wage and is growing faster than the national average.
  • Place those 500 workers into new occupations with 60 days.
  • Retain those workers in their new jobs for at least 90 days.

@jordangiali’s question relates to the third point, but we also want to know if you would suggest changing or adding any criteria.

Thank you!


I would suggest counting in weeks - 4/6/12 for train/place/retain respectively. I would also try to quantify what “living wage” and “growing faster” mean.

Absolutely! Those aren’t terms we want to define ourselves. We are thinking of using Dr. Amy K. Glasmeier’s Living Wage Calculator to assess “living wage” and the Bureau of Labor Statistics’ Occupational Employment Statistics program to define “growing faster than the national average”.

@NickOttens - interested to watch this prize design evolve in the context of COVID19. Are the short and / or long term implications of COVID19 job displacement being considered in defining “growing faster than the national average” as the BLS national average will be biased now in a few ways:

  1. Backward looking indicator based on some assumptions of a tight labour market (previous of Jan 2020)
  2. Research has shown that automation and other job disruption occurs in recessions, making some occupations / skills obsolete vs the labour market picture in 2019 (Nir Jaimovich & Henry E. Siu, 2020)

I would suggest looking at the BLS projection system as this is a 10 year outlook on jobs based on a pretty sound methodology (2018-2028), in conjunction with broader automation risk or AI potential indicators to give you alternative views of how a job might evolve in the future (5-10 years).

On defining success:

  1. Job retention measured against a baseline measure from talent supply data sources would probably be more informative (i.e. what is the average attrition or time spent in this job for a comparable population sample?) - with this, you will ultimately battle a question of causal effect of the program in assessing what would have happened if you did nothing with the participants.
  2. Other measures to consider are pre and post survey measures to get a sense on a) career / skill confidence and b) career / skill satisfaction for a more holistic measure on longer term impact. These are prone to self-reporting biases, but give another perspective.

Thank you so much for your comment, @tjforsyth! I’ll ask the rest of the team to read as well and follow up if they have any questions.

In the meantime, we’d love more input on this question. @eperkins, @mhenrynickie, @AlmaSalazar, if you have any thoughts on this, please let us know!

@NickOttens I would also suggest bringing in elements of “how to manage change”, “how to continuously learn”, beyond conventional technical training so that students get access to tools that might become relevant at multiple times in their lives. Almost harking back to the adage: “teach a person one skill and they can find a job, teach a person how to learn and they can find a career”. In the spirit of the audacity that this group aspires to, i suggest we set some loftier goals then purely skill building and job finding. This can be a milestone in the journey but cannot be the destination. I worry that if we do not set our sights ambitiously, we might be failing in our responsibility and the opportunity to make a real meaningful difference as part of this collective.

Let’s not forget the most important stakeholder – the workers themselves. If you are dealing with vulnerable populations, you may want to assess baseline and follow-up measures of satisfaction and self efficacy, as well as movement towards a longer-term goal (for example taking a class). I work in public health and the #1 “social determinant of health” is income/education. We all want to fix, fix, fix but we go for the low-hanging fruit – the things that are necessary for survival but not the bigger, harder issues like income and education – or empathy. We get them food, maybe housing, help with utilities, etc. All things on the bottom of Maslow’s hierarchy. But we forget about well-being and thriving, parts of “love and belonging.” We may want to use Cantril’s Ladder to assess their outlook on life. Or maybe PAM or WIN measures to show improvement. Retention is important as a short-term metric, but the average job is held about 3 years among everyone anyway – but I’ll about I don’t know if 90 days is a success; does the target population hold jobs for much shorter? The metric should reflect success vs. reality. Getting at the reasons people stay or go is important, too – did they get a raise? Are they getting trainings to help them grow and advance? Do they like the work? Do they see it as a dead-end job that takes them from their kids and only helps them live paycheck to paycheck, or is it a stepping stone towards something better that they believe they can achieve?

Those are excellent points! Thank you, @ukarvind and @pepsicola28.

Any metrics for diversity?

I wonder how relevant it is to compare these metrics during Covid vs pre-Covid.

I appreciate that these are specific metrics.

I also think the retention metric time Frane is not a very long period although workers choosing to leave is different that workers not succeeding.

A couple of thoughts on this:

  • 90 days is a very short time frame. I understand the constraints of the contest, but many firms have grace periods that are that length. It would be easily imagined that workers keep jobs for 90 days but not 6 months (or a year). Long term outcomes matter.

  • What about earnings? Given the current economic situation, I might be fine if the worker-firm match lasts 2 months before the worker moves but trainees continue to earn good incomes.

  • The process by which workers are matched to firms is going to be very important. That does not seem to be incorporated in to the evaluation. I would argue that matching is just as important as the training.

  • More abstractly, the definition of “living wage” occupation and “growing faster than average” is going to be based on pre-virus data. How do you define these categories given the new normal?

Thank you for your feedback, @shurder! I can address a couple of your points:

We want to partner with Workforce Development Boards for this. They would do the matching, so teams competing in our prize competition can focus on the training.

Good point! But I’m not sure we can do better than use pre-crisis data. It’s probably too soon to assess how these metrics will be affected by the pandemic?

Sounds good. I would be curious for a bit more detail on the placement process that the Workforce Development Boards will use.

@shurder Thank you for your feedback!

We’ve interviewed representatives from a number of workforce boards to understand how they can help teams place workers into jobs. In general, the boards have close relationships with local employers. They collect data on who is hiring and for which particular roles, along with the skill sets that are needed. The boards will provide these data to the competing teams and help them make contact with local employers.

Teams will also be able to form and leverage partnerships with additional organizations, such as staffing agencies. XPRIZE also plans to administer an online collaboration platform where teams can connect with mentors and other stakeholders that could lead to placement opportunities. Teams will have up to 60 days to complete the placement process.

@jordangiali thanks for the information.

Designing evaluation criteria with these intermediaries playing such an important role may be difficult. A solution might have better performance simply because it is paired with / has access to a more effective set of placement partners, which could depend on factors beyond the solution team’s control.

If you haven’t yet, you may want to consider which one of these evaluation models you want to pursue, because the design would be different:

Option 1: Placement partners (e.g. Workforce Dev Boards) are controlled for in evaluation. For a given workforce board partnership, solution X was more effective than solution Y. This could also involve random assignment of solutions to placement partners.

Option 2: Effectively leveraging placement partners is part of the challenge of the solution design. A solution that has better results because it builds better strategic partnership should get higher evaluation scores, even if all other aspects of the solution are not superior to others.

@shurder Thank you, this is really valuable feedback.

For Option 2, how would you suggest measuring a team’s performance in building partnerships? Total number of partnerships formed?

I would look at this as rapid reskilling, and measure team performance not only on partnerships, but on number of people served, industry, etc. The DeBruce Foundation might be interesting to contact as a potential partner. They have an online tool that is designed to help lower-income, middle income Americans evaluate their skills and provide insight into reskilling. It’s called the Agile Work Profiler.