Every externally advertised position in the Government of Canada includes a list of selection criteria (or requirements) which lay out the skills and experience an applicant needs to demonstrate in order to be considered for the position. These selection criteria fall into both essential (required) and asset (nice to have).
Getting the selection criteria right is critical to the success of the staffing process. All applicants must be assessed against each of the essential criteria, so adding too many could lead to job competitions with many steps for applicants to complete. On the other hand, the list of essential criteria is also the first filter managers have to keep unqualified people from applying. So too few essential criteria and managers may find themselves with a very large number of applicants, and an insufficient way to distinguish who will be the best choice for the position. So the stakes are pretty high for getting the selection criteria right.
The Hypotheses
There will be an optimal number of selection criteria in terms of producing a hire.
There will be an optimal number of selection criteria in terms of reducing time to staff.
More people will apply to job advertisements when there are fewer essential criteria (or requirements).
Adding more essential criteria will increase the overall time to staff.
Adding more selection criteria in total will increase the overall time to staff.
With the right interventions, managers can be guided to adopt a targeted number of selection criteria (essential and asset) for their job advertisements. We need to be able to guide managers to use an optimal number of essential criteria, or there will be no benefit from uncovering it.
The Experiment
For all staffing competitions advertised on Talent Cloud we kept track of key data including:
Number of essential criteria
Number of asset criteria
Total number of selection criteria (essential + asset)
Number of applicants
Number of assessments
Success of hiring process
Time to staff
The correlations between these data points is what we are interested in for testing our hypotheses. In addition, we added a few different platform interventions to see if we could influence the number of essential criteria managers used.
"When choosing selection criteria, it can be difficult to decide how many is too many... and how many is not enough."
Platform Interventions
We never blocked a job advertisement from going on the site because of too many essential criteria. In fact, in the first year of the live platform, we deliberately provided no direct guidance to managers on how many selection criteria to use, while we studied behaviours, choices, and outcomes. (This caused a fair number of failed processes.)
After the first year, we added information for managers in the instructions for the job advertisement tools, drawing on the data from failed vs. successful job processes.
When we eventually launched our Job Advertisement Builder we added some of these nudges into the platform. We knew it would be tempting for managers to keep adding skills, especially when they are only a click away, so we built in real-time feedback to let managers know when they were within the targeted number of essential skills. Based on the behavioural sciences at the time, we went with numbers, emojis and colour indicators (based on the standard stoplight red-orange-green).
The Results
This was one of the most complex experiments to unpack what we were seeing because of the number of factors involved. Here’s what we think we’re seeing, but ultimately a much larger study with a larger sample size will be required to validate these findings:
There is an intersecting relationship between the number of selection criteria (essential and asset), the number of assessment steps, the length of each assessment step, the number of applicants the job advertisement attracts, the success rate of the hire, and the overall time to staff.
Processes with fewer selection criteria attracted a higher number of applicants. This increased the time to staff at the initial screening stage of the hiring process.
As the number of selection criteria increased (essential and asset), managers added additional assessment steps. This increased the time to staff at the assessment stage of the hiring process, and in the development time for assessment materials. (We found, on average, each essential criteria was assessed in at least two ways by the hiring manager, and assets were assessed in 1-2 ways.)
We found no correlation between the number of selection criteria and the overall time to staff. We suspect that as the number of selection criteria changes, factors impacting time to staff (volume of applicants, number of assessment steps, manager energy levels and enthusiasm) may counteract each other to negate any time savings.
Processes with more than 8 essential criteria failed to attract the number of applicants required to leave a sufficient number of top applicants in the process after the assessment stage. Effectively, it left a single top applicant in the process, and if that person left, the hiring process failed. (See diagrams on the hiring funnel in Optimizing the Volume of Applications in this section of the report.)
Processes with fewer than 4 essential criteria weren’t tested on the platform. While it’s possible that 1-3 essential criteria may produce a strong hire, it’s difficult to take into account the skill requirements associated with a 5 factor match (hard and soft skills) with fewer than 4 essential criteria. Based on the selection criteria chosen by managers, we believe it would not be possible to run an effective 5 factor match with 3 or fewer essential criteria (although this might be desirable for large generic pools of talent).
We did find a correlation between the number of essential criteria and the chance of a successful hiring outcome. Based on the overall balance of all these factors, our recommendation for the optimal range of essential criteria is 4-6, and no more than 8 selection criteria in total (essential + asset combined). That means, if a manager wants a lot of essential criteria, they will need to have few assets.
The number of asset criteria had no bearing on the results. But the number of essential criteria mattered. There appeared to be a range where the number of essential criteria yielded more success. A manager with 5 essential criteria appeared to be as successful as a manager with 6 or 7 essential criteria, as long as they all had no more than 8 selection criteria in total. (Managers with 4 essential criteria fared the best, but the sample size was very small.)
We were successfully able to design platform interventions that nudged managers to adopt these target ranges for the number of selection criteria, which had a positive impact on the overall results for hiring outcome.
Managers don’t necessarily like the nudges that behavioural scientists think are effective. The presence of a guidance tool was applauded by managers, as was the interactivity in terms of feedback when they added or reduced the number of criteria. But ultimately, they simply didn’t like having to work with a tool that had smiley and sad faces. On the nudges themselves, we conclude that the colours and the number range are valuable, but next time we’d swap out the emojis for something a little more subtle. (Of course, then we’d have to test and see if it still worked…)
Insights
These findings represent weak signals, but they would likely be worth pursuing with a larger study. In that context, it’s important to point out that Talent Cloud uses selection criteria made up of a single skill, whereas other platforms sometimes use selection criteria made up of multi-component experience requirements. This could impact an applicant’s perception of whether or not a selection criteria was really one requirement or several described together. Any future study would need to take this into account.
Regardless of the results of the research on the optimal number of selection criteria, we found that the inclusion of nudges was surprisingly effective in changing manager behaviour patterns. Our nudges combined numbers with emotional and colour indicators, and were interactive, so as the manager changed the number of essential and asset criteria, the nudge was updated to reflect this change. The usefulness of this addition to the platform could have broader implications for changing manager behaviours in other types of platforms as well.
We also found that when it comes to designing selection criteria, HR advisors remain an invaluable resource for managers.
