Exploring strengths through "Scalar Interviews"
A novel technique combining the depth of open questions with the precision of closed questions for enhanced interviews
When it comes to interviewing tech candidates, there are two main camps: Those who like open questions and those who like closed questions as well as many flavors in between (see here for an overview of what those entail).
After having done more than 1000 interviews in the span of 3 years with iOS, Android, Backend Engineers as well as Technical Program Managers and Engineering Managers; I got a chance to A/B Test both styles. By looking at my success rate as well as the quality of the hired candidates allowing me to optimize my methods.
One thing that struck me was that certain qualities such as motivation and learning capability of the candidate were easier to discover using open questions whereas other qualities were easier to spot with closed questions.
That got me thinking on how I could get the best of both worlds. This led me to devising my own system which I call Scalar Interviews and I’d love to share my thoughts with you.
But before we begin, let’s look at the two camps.
Open Questions
Open questions are the more common of the two camps. The general belief is that such questions allow the candidate to offer more detail whilst demonstrating their communication skills.
Pros:
Gets the candidate talking which is good as the more the candidate talks the more data points you’ll be able to collect in order to come to a conclusion
You get to observe how the candidate thinks about a problem
As open-questions are more difficult to prepare for in advance by a candidate, you get to see if the candidate is just winging the question on the spot or whether they actually have their own opinion based on their own experience
Open questions allows to measure the wide the candidate’s knowledge is on a specific domain
Cons:
Having the candidate talk means that the control of the interview will mainly be in the hands of the interviewee. If, as an interviewer, you’re bad with dealing with a “Talker”, you might not be able to get the information that you’re trying to get out of the candidate
Scoring such questions is more subjective. As the interviewer, the answers are very much open to interpretation of the interviewer which means that you might get different results from two interviewers interviewing the same candidate using the same questions
Most open questions can make the candidate feel uneasy as they’ll tend to (consciously or unconsciously) think about the expected answer which may involuntarily affect the quality of the received answers
Because open questions take longer time to answer, as an interviewer you get to ask fewer questions in a given time frame than closed questions
Some examples of open questions are:
What do you see as your strengths?
What type of working environment do you excel in?
Describe a complex project that you were assigned to. What approach did you take to complete it? What was the outcome?
Give me an example of a high-pressure situation you faced in the last 3 months. How did you handle it?
Closed Questions
Closed questions are a little less popular than the other camp however they can also provide valuable information to the interviewer.
Pros:
Closed questions allows to measure how deep a candidate’s knowledge is on a specific domain
Extremely easy to score the candidate (one could literally transform the questions in to a scorecard which can be used by the interviewer to assess the candidate)
The answers are extremely objective. If you ask the candidate “Have you ever deployed a service to production using Terraform” the answer is either a “Yes” or a “No”. There is therefore no room for the opinions/feelings of the interviewer that will affect the scoring of the candidate
Since it’s extremely easy to answer such questions, as an interviewer you can ask about 5–7 times more questions during such an interview compared to an interview full of open questions, thereby gathering more evidence for a decision.
Cons:
Closed questions don’t help the interviewer deep-dive into the “Why”
As most answers are quite short, as an interviewer you don’t get to see how good a candidate is in explaining their thoughts
Such questions in some cases can be too narrow. If you’re asking a question about whether or not the candidate has experience with X, the candidate might answer with a no but in fact they might have experience with an analogous technology
Such questions can put ideas into respondents’ minds as when one is asked whether or not they’ve ever used technology X, the chances are higher that they’ll respond with a “Yes” than a “No”
Some examples of closed questions are:
Do you have experience working remotely?
List your top two priorities for improving the technology infrastructure?
Where can someone go to learn more about what you do?
Which products have you used out of the GCP toolset?
Scalar Interviews
With scalar interviews, I’m trying to get the best of both worlds: the long answers that you’d get out of open questions, as well as a narrow domain you’d get out of closed questions.
The idea is very similar to that of a personality test where each question is tailored to measure a certain dimension.
The first step is to decide on which dimensions matter for you and the position you’re hiring for the most.
In the latest role that I was hiring for, I had made the following list of dimensions (think Andy Grove model):
Independence: Does the candidate just do what they’re told, or do they ask questions, challenge requirements and try to understand what problem is being solved
Quality: Does the candidate care about tests, believe in writing clean code and consider other best practices
Thrive in chaos: Is the candidate able to navigate in a start-up environment where there isn’t much structure yet and still deliver value to the company
Once it’s clear to me which dimensions I’d like to measure, the second step is to build my scalar questions around these dimensions. Scle-based questions need to provide the candidate with a spectrum on which they have to position themselves (and an opportunity to explain why they’ve positioned themselves there).
With scalar questions, it is important to set the scene properly. The interviewer has to explain what the spectrum is by giving examples of the pros and cons of both sides. This allows the interviewee to see that there is no right or wrong answer and basically they’re being asked for their personal preference (rather than a secret answer that the interviewer is looking for).
For the dimensions mentioned above, the following scalar questions could be used where the first of the question is setting up the interviewee for the question and the last sentence is the actual question:
“When I look at engineers, I see a spectrum where on one side I meet engineers whom I call “Pleasers”. Pleasers tend to say yes to everything (which is great from a manager’s perspective as when something needs to get done the engineer jumps on to it). However in some cases, such engineers say “yes” to things that they don’t quite understand which means that they sometimes fail to deliver on time (or fail to deliver at all).
On the other side of the spectrum, I also meet engineers whom I call “Over Cautious”. Over cautious engineers mostly say “no”. They start asking questions and pushing back the work as much as they can. However when such engineers say “yes”, then they are extremely accurate on their estimations and almost always deliver on time. Where are you on this spectrum and why?”“When I look at engineers, I see a spectrum where on one side I meet engineers who are skeptical when it comes to testing their code. Such engineers believe that a good test is like a unicorn (a dream that one chases but never finds). They are aware of the fact that creating and maintaining a good test suite is very difficult and doesn’t always have a high return on investment.
On the other side of the spectrum, I also meet engineers who are obsessed with testing. Such engineers believe that any code base should have 100% test coverage. Without a good test suite, they believe that one can’t guarantee the safety of their code nor can they refactor their code. Where are you on this spectrum and why?”“Companies have a life-cycle. On the one side you have enterprises (Google, Tata, Yandex, etc…) which are very well structured. They’re literally a well oiled machine where everyone has their place and it all works together in harmony, meaning that an engineer can focus on their small domain. The downside, however, is that you don’t mean much to that company: if you break, they’ll just replace you with another cog.
On the other side of the spectrum, you have start-ups/scale-ups where there is no structure and you end up doing way more than what you’ve actually signed up for. The upside however is that you mean a lot to that company. With one great idea, you could literally change the future of the company. Where on this spectrum do you feel the happiest and why?”
Scalar questions don’t always have to have two points. They could also have three or more (although I would strongly advise against going over three as having more dimensions seems to confuse the interviewee).
One such great question (devised by Eugene Braginets) is the following:
I believe that there are three types of engineers; tech-driven engineers (those who enjoy working on architecture, performance, algorithms, etc…), user-driven engineers (those who enjoy working on UI, UX, Clear API design, clean code, clear error messages, etc…) and finally social engineers (those who enjoy to coach junior devs, onboard new comers, give tech talks, write blog posts, etc…). I assume that you’re not one or the other but rather a mixture of the three. If you were a pie chart, how large would the slices be and why?
This question helps the interviewer understand what the interviewee cares about the most as well as why they care about it so much. It still helps the interviewee stay inside a specific domain (the domain in this case being the responsibilities of an engineer) yet still allows the interviewee to explain their reasoning (hopefully backed up by some examples).
Once the interview is done, it becomes trivial to map the answers into the desired type of candidate that you’re aiming to hire.
Summary
Hereby a summary of the Pros and Cons of Scalar Questions:
Pros:
It allows the interviewee to talk about their experiences and opinions on the specific domain that the interviewer is interested in.
It creates a fun experience for the interviewee as they get the feeling that they’re creating a character in a role-playing game. Of course, as we all know, happy candidates give better answers than those who feel stressed
Any open or closed question can be converted into a scalar question which gives a lot of flexibility in the interview style
Cons:
As candidates are less used to hearing such questions, they sometimes catch the candidate off guard (although, as they answer the questions with their beliefs they never fail to answer)
It takes a little more time to set the stage for the question. With closed as well as open questions the question itself is literally a single sentence whereas with scalar questions one has to make it clear that there is no side of the spectrum which is better than the other side (both sides has it’s pros and cons)
Such questions are still subject to the candidates’ self-reflection as, for example, the candidate might think that they’re a pleaser but their past manager might disagree
As scalar questions are something that I’ve ended up using in an emergent manner (instead of it being based on a psychological theory) I would love to hear your questions/comments/ideas on whether or not this makes sense.
This article was originally published on LeadDev.com on November 05, 2024.