One of my favorite jokes goes something like this:
Whenever a new position opens up, and I have to sort through a stack of resumes, the first thing I do is take half of them at random and throw them away.
I don’t want unlucky people working in this department.
As with many good jokes, it’s funny because it has a nugget of truth to it.
Anyone who’s run an accelerator, a grant program, reviewed book/screenplay manuscripts, or been in charge of hiring has an impossible challenge: how do you find the proverbial needle in the haystack?
The answer is usually some variation on, “start sifting.”
Sorting through a massive pile of potential applicants is time consuming, boring, and feels pointless.
But we know that the results are undesirable. Being unable to fully and objectively review applicants perpetuates the existing weaknesses of the selecting culture, misses out on diamonds in the rough, and it is costly.
And it’s not like it’s a malicious problem – it’s just an effect of the system.
We just don’t have a better way to do it.
Or do we?
Option 1: Automate or outsourcing the task to less costly labor. Perhaps we give the young, inexperienced pre-screener a set of very simple rules of thumb like “Graduated from Harvard/MIT/Stanford” or “ready to work 40-hours-a-week on their venture.”
There are lots of problems with this. The first is that it inadvertently bakes in the flaws of our current systems as it prefers people who tend to graduate from top-tier schools or who can afford to work full time on their venture without going hungry or neglecting their family. You end up with a very young, white, and male demographic.
It also leaves out the ‘x-factor’. There are always factors beyond the obvious and quantitative that can signal a great applicant.
Option 2: Use Experienced Judges.
People who can spot that x-factor are rare, busy, and expensive. So we don’t use them until late in the process. When we do, we often use them in ways proven to be ineffective – for instance putting 3 judges in a room for 8 hours and sending applicant after applicant to them all day long. Studies show the applicants coming in at 5pm have a much lower chance of being accepted vs applicants coming in at 9am.
Another issue with these high-cost judges is that you often can’t get enough of them to achieve the diversity of perspective you need to spot the best applicants.
Why?
Because these people cost a lot of money, and moreover, are hard to find!
Oh, and on top of that, because of the cost of processing applications, there is NO practical way to provide feedback to the vast majority of people you reject. So, you can’t maintain a relationship with potential future applicants, nor give them a clear path to improvement.
How much of that applies to your system? What if you could do something better, more convenient, and at lower cost? Usually that requires a technological miracle. But in our case we just need to take advantage of a very old miracle – peer selection.
The Peer Selection Solution:
At Valley Venture Mentors (VVM) we created a selection process that improved on all of these issues – reduced cost, saved time, AND increased the quality of applications we accepted.
In short, it’s something of a miracle solution. And, like many miracles, we stumbled upon it largely by accident.
Let me show you how it works.
Step 1: Peer Selection
- The applicants submitted their applications, as normal. VVM’s applications, for example, had two sections:
- Pitch – Section describing the venture
- Rest – Everything else, notably: Entrepreneurs’ names, contact info, demographic data, etc.
- Drawing inspiration from research on blind auditions, we created packets containing 20 randomly selected Pitches without information about the founders. If you don’t know that the founder is female, of color, LGBTQ, etc. then that cannot influence your decision. (It isn’t perfect because if someone had a startup called “Black Girls Code” that information will be in the Pitch and the reviewer might make some educated guesses. However, we solve this later on in the process…)
- Give each applicant a packet and a limited amount of time to read, score, and provide feedback to every Pitch in their packet.
- Why 20? We wanted to have each company score as many of their peers as possible to get a more accurate score, but we had to balance that with the amount of time that would take the applicants to complete. Our goal was to have the whole process take < 4 hours so that the applicants wouldn’t get tired and start scoring people too harshly.
- This was all done with Google Forms and Google Sheets, so no costly or confusing software was required!
- We had a few simple checks to see if someone was gaming the system. We kept an eye out for things like:
- If you scored teams low that everyone else scored high (or vice versa).
- If you seemed to give random scores or score everyone the same.
- If you gave little or no feedback.
- Then we used some spiffy math to adjust for the fact that some people grade generously, some harshly, some with a tight spread, some use the whole scale, etc.
- We disqualified anyone who we felt was trying to game the system. In theory, this could be tricky. In practice, of the very few people who tried to game the system, 100% of them ended up with low scores from their peers anyway. I assume that would not always be the case, but it was the 3 years I managed the process.
Simultaneously we had a Control Group – a group of the usual kinds of people who are used as judges – the expensive people. We had them review a random packet and compared their scores to the crowd’s. The scores were 80% the same.
Benefits of Our Peer Selection System
- Faster – Processed hundreds of applications in 1 week
- Cheaper
- Almost no “expensive” talent (just the 5 control group judges)
- A small portion of time from two staff people
- No expensive software!
- Better
- Dramatically reduced unconscious bias.
- Every applicant received feedback from more than a dozen peers.
- Tests the applicants’ commitment and willingness to work for the position (if they submit no peer review scores, you can safely disqualify them).
- The activity, one of the first any entrepreneur would experience working with us, showed applicants how serious we were about being a community and living by our values. It helped us find our kind of entrepreneurs and let them know we were right for them. In short, it made them like us. A lot.
That’s how it worked. Knowing this, you can build one for your program. If you’d like help I can put you in touch with someone you could contract to help you. Reach out to me on LinkedIn
Paul G. Silva 