Episode 22

Marc Roulet on the Complexities of Hiring in the Modern Data Landscape

TIM: Marc One thing I'd love to hear from you is when you think about hiring data talent right now in this market, what are the leading challenges? the leading pain points that you see at the moment

MARC: I think finding the perfect cultural fit for the organization is something that is always important and true, and maybe that's also just because of the company I'm currently in, because at SevDesk there's a lot of focus put on making sure that we have a really good composition in not only our team but in the entire company. So the first company I ever joined that actually did a culture slot In my interview process, there was a half an hour where I met with their culture ambassadors. That, to me, was really telling of how important that topic is for them, so maybe that's why it's more top of mind for me, but that being said, I still think that it's no matter how big your organization is or how big your team is, that's really important to understand, like making sure you get the right person for the right job at the right time. And that's also because I think team dynamics make a huge difference in the performance. It's not so much only the individual person; it's how does he or she really fit into your team at the moment. You're not going to have a great team if you're only hiring rock star senior principal analysts. That's not going to work. There's no harmony in that. You need diversity in the team, maturity in background, and all of these things. So that, to me, is more like this: what is important to me, obviously given the current dynamics around like AI developments, is potentially having to adapt your hiring process to the realities of an AI world. Candidates can look up answers on the fly where most people do remote interviews, which kind of changes I would say a take-home case study used to have a very different potential clarity because you would know that somebody really did that work. Now they still obviously did the work, but they maybe had some different health than they may have had a couple of years ago in doing that. So I think you have to adjust what kind of case studies you provide, for instance, and what's also always true might be even more true now. I would say two plus years ago there really was a scarcity of talent; it was really hard to find good candidates, and you really had to woo them and make sure, like, you really want to put the company and your role in the best light. and you would be lucky if you got, like, good talent. I think that has changed a bit now, so I think it's back to at least in the data and analytics field, it's back to reducing the CV to interview mismatch quota because you want to reduce the number of interviews that you need to do because you'll have a lot of CVs on your table. So I think that, but that was probably also the case 10 years ago; it's just it's gone through a change cycle.

TIM: Yeah, and I guess hiring goes through that from time to time, and as you say, a couple of years ago it was scarcity, so then it was all about reducing the hiring process to as few steps as possible. Don't do anything that's going to scare a candidate off. Like, we've just got so few to choose from. I remember in Australia in particular, maybe even more so than Europe, because we had closed borders during COVID, we had no migration at all for two years, and in our, particularly in our tech and data sector, I'd say the vast majority of people are over-skilled migrants. So if we don't have that inflow of new people, we don't have anyone to do any of the jobs, so it led to such strange market conditions where some of the salaries, as you can imagine, shot up by 50 percent in just six months, and employers were fighting over such a small number of candidates. That's where you get the only work: four days a week, work from home forever. Apparently, what is it, pet insurance? I heard that as maybe a bit of a joke one in America, I'm not sure if that ever happened to you, but now it's, yeah, we're back to a more normal market. I don't think it's that drastically bad for candidates, but how do you view this over a longer horizon?

MARC: That's actually, I think, a really good way of putting it. It's healthy again. I know a couple of years ago I had several situations where candidates first accepted an offer and then a couple of weeks later or even up to only a week before they were supposed to start the job, and in Germany, right It's very normal for candidates to say, I'm joining in three months, because there's a notice period of three months that is completely standard, and then a week before they were supposed to join, which is then a very long time later after signing the contract, they're like, Oh, sorry, I found a new job. I found a better one. I just changed my mind because they could really pick and choose, like cherry-picking, which I mean beyond the fact that I think that's really poor taste to do that, like you shouldn't do that, but it was a reality that they could. I don't think that's the case anymore. You'll still have really good candidates that will have that opportunity, but the job market is in a more healthy, balanced state. I would agree with that.

TIM: Yeah, and I remember even ourselves at the peak of COVID hiring a graduate engineer, and they were really good. They were with us for four months, fresh grad. They left in four months for, I think, 70 percent higher salary, and we were paying market rates when we employed them, so, like, things move so quickly. I feel like that's probably not that helpful to the hiring market because then you start to get this kind of distrust built in if candidates are sitting there going, I've got another offer for 20,000 dollars more. I can't not take it. Like, it's fair enough in a way, and anyone would probably be tempted by that. Exactly. They say candidates don't leave because of money; they leave because they're unhappy in their job or they're unhappy with their manager, but those kinds of increases just don't hold true anymore. Yeah, and particularly if they haven't joined yet, they haven't built up that socialization in the business, then they probably still view it as an opportunity. It's really interesting. It will say how that tendency to do that or not varies drastically by market, so that would be considered quite rare in Australia, I think, as well to not join after you've already signed a contract. but in some of the markets where we've worked and recruited in the Philippines, it's quite normal that people would renege and actually not turn up or tell you, so it's quite common that you go Oh, where's Joe? Oh, they haven't turned up. Oh, they must have another job; it's baked into the way they do it, which would annoy me. no end

MARC: that is definitely a cultural thing I remember the first time that happened to me I was like personally offended I was like how could somebody do this this is they abused my trust it's a job i okay they did first of all friend of mine everything's okay but in the end of the day I still think that it's common courtesy to not do something like that a contract like A contract is a contract. It's like a binding handshake deal. You said you're going to do this, so you have to.

TIM: Yeah, I agree with you, but we shouldn't take it too personally. As you say, one thing you mentioned in passing was that cultural fit has always been important, is important now, and will probably be important in the future, and you also said that your current organization has a very clearly defined culture. Then I'm interested to hear about your thoughts. Did you notice it got measured in the hiring process? Do you try to measure in the hiring process? Do you ever feel like there's almost a trade-off between things that are a bit more measurable and things that are a little bit less measurable, like culture? love to hear your thoughts there

MARC: That's a really good question because obviously you can't measure culture; it's not something that you can put on a scale. You can't test somebody in the sense that you can test coding skills; it doesn't work, and culture is also a very fuzzy concept, right? Many companies have a culture that is a couple of posters on the wall. But I think the basic thought behind that culture process is two-sided. I think one reason it was that it was important for the company to really show that they're serious about the topic, and that made an impression on me as a candidate that they would take the time to have a culture ambassador that has nothing to do with my day-to-day job. I think it was a I'm pretty sure one of the there were two people, and one of them was from customer support. Nothing to do with my analytics and data background, and all they wanted to do is understand how I interact with my organizations in the past that I've worked with. What is that? What is the concept of culture in a job context actually mean for me and things like that? and I try to look out for those cues as well. I probably wouldn't call that culture; I would call that more like attitude in the sense of, We have a high-performance culture, so to me that also is more of what kind of person are you? Are you willing to go the extra mile? Are you motivated, or is this for you just a 9 to 5, put the pen down? and it's not about working overtime; it's more about do you care? Are you passionate? Maybe that's the right thing. I really care about data and analytics; to me, it's hard to understand how somebody can't have a passion for their job, and if you can't feel that in an interview, you're definitely not going to feel it in day-to-day So that would be one of the aspects of culture that I think you can't measure honestly, but you can feel it. It sounds really, I don't know. It's hard for me to describe because I think On the one side, you're trying to be as objective as possible in recruiting, but it is virtually impossible to do that. Because it's a really human and personal interaction, there's a certain aspect to it that is numbers. It's how many years of job experience do you have? What kind of companies have you worked for? What kind of reputation do they have? How good did you solve a case study? How well are you able to communicate verbally and in writing, and whatever you decide, whatever's important to your role? All of those you can, I think, very well Kind of check some boxes, but once you get into that actual interview, I think all of that is gone; it's basically very much about how the person is interacting. Being able to respond to your questions and all these How do you interact with them? and That's a really personal thing to do, but to get back to the original question of, Like, how do you measure culture? I think everybody in their head likes high-performance culture. Is this person super driven? A community-driven company that cares about doing after-work social activities does that resonate? An open feedback culture does somebody get personally offended if you challenge them on what's that weird gap in your CV? I've had people that got offended. I don't think I have to be answering this kind of question right now, but you're clearly not open to critical questioning and feedback. If you can't do that at a job interview, you won't do that in real life at your job, and so I think that's how you can basically vet for culture.

TIM: Yeah, I have to say I'm personally a little bit in two minds about this over the years because when we first started hiring for our business, we didn't think of this at all; we didn't really have a clearly defined culture, at least not one that was written down that we'd consciously thought about and discussed. We hired our first set of people that he affirmed for various reasons didn't work out. It was at that point that we sat down thinking, I'm like, we're missing something here, so we tried to think really concretely about what our values are and maybe what we want them to be, so it's slightly aspirational as well as what it currently was. We managed to reduce it down to eight; it's still probably too many, but still, that's what we got. and then we thought, okay, can we somehow align our hiring process to this? What we came up with was then trying to see if we could measure any of these in the hiring process. Where I got to was the fact that a lot of them, a lot of companies, would tend to do, like, a behavioral interview for a value and be like, Give me an example of when you've done X as the value. So it could be like teamwork or leadership or communication skills, or, to your point, maybe it's openness to criticism, but I felt like that's very easy for a candidate to get through if they're well prepared and they thought through it and they know what to expect; like a coached candidate is going to nail that interview. But what you did was interesting because it aligns with what we settled on, which was actions speak louder than words. If there's a way that they could have a chance to actually demonstrate it in their behavior in the process, that would be way more indicative, so the fact that you would dig in and almost critique them a little bit and see how they react. I think that's really clever because you can't, unless you're super clever as a candidate, really think forward to that. Hang on, this is part of the test, and so that's really fascinating that you took that approach.

MARC: Yeah, so one thing that I believe works the best to really understand how a person is to put them in a real-life environment, and given that many interviews have happened fully remote, that doesn't work as well anymore, but what I used to do is I used to try to make sure that I add a half an hour coffee talk or even have them join the team. The candidate had to join the team for lunch because they're sitting at that lunch table. They will forget that they are actually still in an interview process because it is still an interview process, but for them Hey, I'm at lunch with a couple of people, and they will start acting their normal selves. Basically, they let their guard down, and then they just act their normal selves, and I have had some very positive experiences. I've also had some very awkward ones where I'd like to say that I don't know if I would say that in an interview process, but okay, here we are, because they just completely forgot that this is actually still an interview, and they started talking about there's nothing wrong with talking about personal things, but personal things that are borderline like you shouldn't be sharing this with your colleagues even at work, let alone if you don't have the job yet. So that was, I think, a real revelation for me to show, like, the formality of the interview has people on their guard, and knowing, like, I need to say they are expecting me to say this, and so I will try to understand. I'd say an intelligent person will not oftentimes say what they think; they will say what the person thinks or what they think the person wants them to hear or to say.

TIM: Yeah, and that, I guess, is part of the challenge of the interviews and why they are often not the best indicator of whether or not someone's going to excel in the role because it's a bit of an act; it's a lot about crafting these narratives, these stories that are quite biased towards people who are maybe quite confident, who are good blaggers, who've thought through a lot of this, and are well planned. Maybe some of those things are correlated with success anyway, like being really well planned and organized. Fair enough; who wouldn't want that in a candidate? But I feel like for a lot of data roles that are the individual contributor, largely technical kind of role, I could see a lot of interview structures. probably selecting the best people a lot of the time. What do you think about that?

MARC: There's two aspects of that: one of them is you don't have to be really good, like Speaker, and very social, to just do good preparation, I think. So what I always try to look out for is if somebody really did their homework, and that starts with if I ask something about the company and you have no idea about the company, you didn't do your homework. If, like the presentation, one of the parts of the take-home test is a presentation and there's typos in there and it's just really sloppy, That's not good preparation, so that's something where I think you can try to remove that bias towards more vocal, communicative, really self-assured people, and I try to really ignore if people get nervous. Things like that do not matter to me at all, but where I've seen even like the most, what we say, orative, like very suave communicator, where it falls apart is if you dig on questions because they'll have their answers ready. but if you go and say, Can you elaborate on that a little more? no I really want to understand that I need to go dig deeper if they don't know what they're talking about. If they've just prepped a couple of taglines, that's what my experience is. What separates a good interviewer from just somebody who doesn't really care is that I was just trying to ask their question, like, give me an example of where you had a challenging interaction in your previous company and how did you deal with that? and then they answer that, and they're like, Okay, okay, move on. They had an answer to that, but if you then go, Okay, but why was that now a problem? How? Why did you decide to deal with that? What could have been an alternative path? Do you think you really solved that in a good way? you can really challenge the person that they have to explain that more. that and that has nothing to do with good or bad communication; that's just people. Then their brain really starts to work; they start to reflect, hopefully, on that real situation that happened, and that's where you can tell if they did or not or if they just thought of a stupid answer to provide because they knew this question was going to come anyway.

TIM: Yeah, digging down and not just taking the first response is so important, and it must not only help the candidate demonstrate that depth of knowledge or understanding but also how intricately they were actually involved in a project. I know a lot of interviewers would complain about a candidate saying, referring to a team project they were on, and it turns out they did this tiny sliver of it, and really they weren't actually involved at all, and once you start digging, that becomes immediately apparent. Very interestingly, as an analogy, I feel like traditional interviews then are almost sometimes too superficial and allow a candidate who's at a high level talker to get through. Maybe the first round doesn't matter as you go through a hiring process that lessens as soon as you start speaking to the hiring manager themselves; you can really dig in. I wonder now if politics is quite similar because politicians have been used to just talking in these kinds of sound bites. They just need to get through that quick Q&A session that is not going to dig in, but now that's changing because we're moving to the podcast format. People are going to go on for three or four hours at a time. The interview is going to have an opportunity to really dig in, and so I wonder if politicians are going to have to become a bit more depth-oriented, not just superficial and flashy. and candidates would have to do the same if they want to pass a really in-depth interview, and that's a long, long boat to draw. What do you think about that?

MARC: I don't think that politicians are going to develop to be more knowledgeable; unfortunately, I think it's the opposite. It's been proven that politicians don't have to know very much at all; all they have to do is have really good talking points, and it doesn't even have to be fact-based; it just has to be something that people care about and people are emotional about, and that works. but that's always worked, so I would say no politics is a different beast that I don't want to get into, but candidates I think I'm a fan of long interview processes, and I know candidates hate them, but I think it's really good to spend a lot of time with candidates and have those different aspects. So if we do a panel, we purposefully have people from different roles that are then basically having the clear mandate to vet a candidate, like from a more technical perspective, from a product perspective, from an engineering perspective, and that takes time. Even if we do it in a panel, we try to be a little nicer to candidates. I've seen processes where you have to sit down for a half an hour or 45 minutes with the engineering director and the product director, especially in functional roles that have intersections or overlaps with many different areas. That makes a lot of sense because then you can get grilled for 45 minutes. By a CTO or an engineering director, that's a difficult conversation to have if that's not your core field of expertise, but it's a necessary step. I definitely don't think you have to go like assessment center-style hiring process that is maybe a little over the top, but longer processes with in-depth conversations, not just these very superficial repetitive stuff I've seen that as well. Basically, you go in, and the people are asking the same questions. Everybody's asking those same questions because all they're trying to understand is what type of person you are and what your team dynamic is. That's one important part, but it is again only one important part. There are many other aspects that make a good candidate from a really great candidate, which is really hard to identify in an interview. To be honest, you have to go with gut feeling sometimes at the end, but we try.

TIM: It sounds like the process you're running, at least at the moment, is optimized to reduce the chance of a bad hire at some level, like to really make sure you get the right candidate, as you said, for the right job at the right time. Is that partly driven by the conditions in Germany, where you say you have that three-month notice period and also quite a long probation, if I'm not mistaken?

MARC: six months

TIM: So it's difficult to have a candidate in week one and go, This is not the right candidate; I'm going to get rid of them. It's not like a sort of cutthroat American market where if you make a hiring decision, you just fire them and move on. It's quite difficult to do that, so does that almost breed a little bit of risk aversion? do you think

MARC: Technically, you have six months to make sure that your candidate really is good because in those six months, you can let go of any candidate with a two-week notice. Anytime you do not have to, you do not have to provide a reason; you could just say, I've just decided this is not going to work," and I'll say that. Sorry to let you know, but obviously that's not very nice to do that. But you can so interestingly, actually, there shouldn't be that high of a risk aversion to hiring people where you might not be sure the reality is that most, I think most hiring managers don't make use of probationary periods in at least Germany because I think, to a certain degree, some hiring managers feel to them like it's like a personal mistake, right? That was something very personal to no I didn't hire the wrong candidate. I also think that there's this human interaction part to it to say, But that's not very nice, letting somebody go that just found this new job, because it's a very tedious process to find a good new job. But it shouldn't be risk aversion that's leading you to your hiring process; you should be hiring to make sure you get the best candidates, and I just learned this statement from an HR colleague: when in doubt, I was like, okay, yeah, I see where you're coming from; if you have a doubt about a candidate, don't hire them. I cause I don't think that's fair. Don't hire a candidate where you're not quite sure with having him in the back of your mind. I can just let him go in two months anyway if I don't like him if it doesn't work out, because that doesn't seem very fair, but if you have, I have done that. I have hired candidates where I thought, I'm not sure about this part of this; I think this could be really great, but there's something in there that I think it might not be, and they've turned out to be really good candidates, and in the back of my head I did have that thought: if it doesn't work out, I can let them go within the six-month probationary period. because that's my job as a hiring manager to make sure I get the best talent in the world into the team, and if it doesn't work out, I have to make that tough call to let people go on a probationary period.

TIM: That's interesting, and the phrase so when in doubt, out"—I get that, and I've certainly seen that firsthand. I noticed that the longer a hiring process is, the lower the chance of a candidate ultimately getting an offer, especially if there are more and more people involved, because someone will always find a reason to not like a candidate, I think. If there are enough people involved, especially if there's any kind of subjectivity in the criteria of what you're evaluating, someone will come up, someone will get out of bed on the wrong side in a day, having had a dead coffee, their girlfriend yelled at them or whatever, and they'll come into the interview at 9 am, and they probably won't be in the right mindset to evaluate a very grey thing like is this person the right fit for our team. So I feel like it's got a slightly dangerous downside and one that isn't often measured, like you. Everyone remembers making a bad hire; nobody remembers not making a good hire because there are thousands of candidates to choose from, and so I feel like there's another side to that coin.

MARC: Yeah, you never know what great candidates you actually missed, although when I say that when in doubt, out, I think that boils down to does the hiring manager have a doubt? It would never work if you say as soon as anyone in the interview process has a doubt, to me, everybody else in the hiring process is a consulting person that helps the hiring manager get a very differentiated, broad picture and feedback. The way we handle it at the moment, and I like that, is even if people say, You, I would have voted no; I wouldn't hire this person. The critical question then behind that is, is that a veto, or is that just so you wouldn't hire them? And if somebody gives a veto, that basically is that I have so many red flags here. I don't think I could work together with this. This would be a huge problem. I've done that once. I hired against the resistance of the team because I was like, No, I think this is a perfect fit for you. It was a disaster. It did not work out. I would never do that again. But beyond that, if people just—I don't think it was the best candidate. I'm not so sure. Do you try to collect the reasons behind it? Try to get that full picture, and only the hiring manager knows exactly what they're looking for, and I think has the full picture of the candidate, the position, and the team dynamics and what they actually need right now? And it's their call, and what I meant is if you, as a hiring manager, have doubts Probably best not to hire the person.

TIM: I think we added the phrase captain's call to our dictionary in Australia about eight or nine years ago. One of our prime ministers is a fan of that phrase. It sounds like, Yeah, captain's call, you're the one ultimately accountable for this person being successful. You have the full picture, taking the data points from other people, but at the end of the day, it's your decision.

MARC: Yeah. I think that's correct. That would be a captain's call in that moment because you're the captain of your hiring process.

TIM: Do you feel like there are ways currently that you could make the hiring process more objective? Have you put in place any things? Are there any interesting examples of making processes a bit fairer, a bit more objective, that you've come across?

MARC: I think in the first step, a semi-automated screening process could help because that will naturally remove bias that people have in vetting CVs, because I think it already happens there. That's already the first place where people will have their prejudices baked into their quick screening because if you do a 30-second screen of a CV, You read so much information where it is super easy to be like, I don't find this candidate good, even though actually if you would just look at the raw information that's there that is relevant, if you define this as my relevant set of information, this many years of experience, these different tools and technologies that they have to have used… and you can define all that. I think that definitely would help, and I think AI-based recruiting processes will make that even better because those have existed in the past. Potential is because it makes it a little more nuanced, because if you just say, I'm going to cut everybody with less than five years of work experience, or I'm going to cut out everybody that doesn't have a PhD, You might be missing out on some good candidates, but if you can have an algorithm that really basically gives you a quality score based on the things provided, I think that can help. I think the second thing that would help—I don't know how much that's a thing in Australia, but I'm seeing more and more CVs that don't have pictures, don't have age, and don't have… There's always a name, but removing all these things that are just human That's why it was interesting. I was doing some research on AI tools, just generative AI, and then I also did some on HR, and I found a tool that was advertising that they could automate screening through one-way interviews. I guess it's you just record yourself talking, and then people can watch five minutes of it and then make a decision, and I thought that sounds like the opposite of removing bias because all you're doing is looking at people and how they talk for five minutes. There's no substance in that; it's just, Do I like your face? That's like, how can that be a good improvement of a recruiting process? Because I think you want to remove that, and what I'm getting at is I think it's important to have a panel of different people with different backgrounds in your hiring process, and I do mean here now technical backgrounds in the sense of engineering, marketing, product, and data analysts; they will have a very different criteria for what makes a good candidate. I think that's a good thing. I am very skeptical of trying to really standardize interview processes because, in my experience, it doesn't work, and that's because I've lived through the experience, and I've found it to be really, for a lack of a better word, silly because it was so flawed in its process, this highly standardized process. interview where everybody gets a fair chance and it knows exactly what's expected of them But that removes the whole purpose of an interview process.

TIM: It removes the purpose in what sense? Because it's too manufactured or something like that.

MARC: If you want to understand how good somebody's coding skills are, you can do a coding challenge. That is objective, but if you try to understand how good somebody is in their strategic thinking, You in their application of data knowledge to product management How well do you understand business processes, systems, and thinking? You name it. Whatever you're looking for, that is an inherently individual person discussion that has to unfold dynamically in an interview process. Actually, no interview should be the same because even if you give them a take, take this challenge home. Don't give them a lot of context; everybody will come up with an entirely different interpretation of also numbers, way to visualize it, way to communicate it, structure; it's all different, and so I was in an interview process for what's known as the FANG; that is, the big American ones: Facebook, Amazon, Apple, Netflix, and Google, I think, is what that stands for. and they have completely tried because they have so many candidates to standardize, which basically I got a pamphlet beforehand, seven days in advance, that said this is exactly what you're going to expect, and this is what they're looking out for, and these are the answers you basically should be giving. I was like, What's the point of this interview? Then it seemed really weird because the idea was to obviously remove bias and make sure everybody gets a fair chance, but what then happens is you get that, and then if you provide exactly that, every interviewer is bored out of their minds because they've heard it a hundred times because everybody says the exact same thing. and then that leads to Like interviewers saying, Yeah, I don't care about those answers; tell me something else, and then they just start to chat with you for a half an hour, and I was like, Okay, that was obviously completely off the point; that's why it doesn't work. I think standardizing that part of the interview process doesn't work. There are certain aspects that work; there's some where you just have to allow for that individual communication and exchange between interviewer and interviewee. That's what it's there for; it's to have that unexpected turn, and all of a sudden you ask a question, and somebody lights up because that's what they're totally passionate about, and then they like to talk about it, and they want to go deep into it or whatever it is. I think that's what it's there for. That's what I mean when I say I think standardization of the human interaction and interview process doesn't work.

TIM: Yeah, it's so tricky, isn't it? Because, as you say, if it sounds like they've overstandardized that it's mechanized, it's so specific that it doesn't allow for a candidate to deviate from the very narrow set of things you're discussing with them in the way that a take-home project does, because it could just go off in any direction. and the way they approach it says so much about how they think and their skills in so many different ways, whereas if you narrowly restrain them, by definition, they can't impress you outside the box the way or the reason they would do it, I assume, is because they have such a stupendous amount of data over hiring patterns for years with, as you said, tens of thousands of candidates where they can optimize and actually measure some of the stuff that most companies aren't measuring around conversion rates at different steps and success on the job. I'm assuming there's a reason they've done it, but it sounds like they've taken it too far and now it's ineffective because, as you say, the interviewers themselves are deviating from it and realizing that it's too mechanized. Maybe there's something in between that would still be objective and structured but would allow candidates to demonstrate those things outside the box. I don't know what you reckon.

MARC: That's an interesting thought, like where should the pendulum swing? Because obviously you don't want this complete freestyle; you do want some more common patterns. What I try to do is I try, since every job that I'm hiring for is slightly different, I won't ask the exact same questions in every interview. Even if I'm hiring for an analyst now, I'm hiring for an analyst again. It'll be is it a junior, mid-level, or senior? Which team is he in? Where is that team currently, and what do I need? What I try to do is ask the same questions repetitively in the interview process of the candidates because that at least allows me to compare the answers that were given. Sometimes I would say I have a core set of five to seven questions, and then I'll have additional ones, and I'll just deviate from that narrative based on the interactions I'm having with the candidate or whatever they prepared and are presenting, but that's something I do try to do. I remember when I was a fresh lead I got a pamphlet. It was like, It's 300 questions across different dimensions that you can ask for a good candidate experience, and I remember I would, like, Marc those that I really liked, and then I started using those again and again, and I think that's where I came from. That's how I developed that pattern of recognizing asking the same questions definitely helps because then you're really judging on the same questions, so that's a good thing to do. Not only are you judging on the same case study, you're also judging on the same questions. Interviews should not be completely freestyle because that allows for way too much bias. But they also shouldn't be completely standardized to the degree that then people are taking trainings to AC or interview; there's something wrong with that. That's like doing an SAT preparation test for a college; that's weird.

TIM: I feel like also part of the detriment for those companies is because they have so many candidates who are likely to share this information and put it online, so it's like a whole prep industry of how to get a job at FANG, whereas for a smaller business that would never be the case, so maybe with a structured approach it might work a little bit better for them because the questions aren't leaked. the answers aren't leaked The interviewers probably haven't also asked the same question 8,000 times that year because this is not the same volume of candidates, so maybe they've shot themselves in the foot a little bit with how successful they've been.

MARC: Yeah, to a certain degree, I think so, and that was the same 15 or 20 years ago. I think so. The consultancies were trying to be clever and ask brain teasers, and then everybody was just trying to prepare on how to best solve absurd brain teasers like how many tennis balls fit into a jumbo jet.

TIM: The other one that was really common I remember was what's the angle between the minute and hour hand on a clock if it's 3:15 because it's meant to be like a trick question, and so that's one I've personally had in multiple investment banking and management consulting interviews. So yeah, again the candidates start to prep, and then it's really not as spontaneous as it should be.

MARC: Which basically I think all it says is you always have to innovate on your interview process; probably that's the key thing. Don't always use the same case, which, by the way, I have also. I've both been in a case study, and I've been the culprit that used like a super outdated case study where candidates were like really irritated, like, because oftentimes case studies are like a semi-reflection of your business, but then it was like some old, like, internationalization concept, and you had to look into the numbers ad hoc. It's like you guys are planning to do this. Oh no, no, this is four years old. That's not a good thing; you shouldn't be building on the same case study for years. You should always update it to make sure it reflects the current way technology can be used in the state of your organization and to stay innovative to have new ideas that you bring into how you do the interview.

TIM: Yeah, underappreciated tip. Actually, I can think of two relevant examples. I can remember one business coming to us just for a bit of feedback. They'd gotten a referral to speak to us, and they shared with me that their case study that they were using to hire a data analyst, and this business didn't really have an analytics team. They were doing their best to hire someone that was well without that outside of their normal remit, and the questions looked reasonable looking through each of them, and then they got to one, and it was getting the candidates to do some kind of analysis in VBA; specifically, you had to use VBA in Excel. There's the problem because, oh yeah, they remembered, oh yeah, some candidate had mentioned something about VBA and macros or whatever, and not knowing anything about it, I'm like, Yeah, it's not 2007 anymore, or whatever; there are other tools they're going to use, and it's only just negative. It's actually not only just neutral; it's negative 'cause the candidate will then be thinking, Oh my God, am I going to get into Excel hell? Are you telling me there's no warehouse? I can't even. Maybe I'm not even allowed to use Python. You're going to force me to use macros and VBA? No thank you.

MARC: Exactly, yeah, that's exactly what I was going for. It's important for both sides; it can be very off-putting to a candidate.

TIM: Yeah, absolutely. You mentioned earlier innovating your hiring process. AI is innovating everything. How have you seen it impact the hiring process already, or the way that candidates approach the way that companies approach it? What do you imagine is going to happen over the next couple of years?

MARC: I think the one part where it's become really apparent is that it's changed the dynamic of take-home case studies that include a lot of text and research those candidate processes. I think they have become a lot more difficult for hiring managers to navigate because it has become so much easier to just ask ChatGPT to summarize this question. To then you use it, what's the I forgot the name of the tool for PowerPoint. You just build your slides for yourself. There's so much where I've been now over the past 12 months. I've been in interviews where I thought, I'm 99 percent sure you didn't do that yourself, and you can't tell candidates you can tell candidates to not do it, but you would also argue what's the point? That will do that in their daily business as well, so how do you vet a candidate for good research and storytelling? Because not everything can be answered by ChatGPT and summarized, I think that part is the most difficult one that's really changed, obviously around process improvements. those those I think I've seen that just automated scheduling CV screening, though that part of the hiring process I think is more heavily impacted, but I think you were going more for the actual interview process or how you deal with candidates there. I think it's yet I haven't made up my mind on how I potentially need to actually change, like, entirely change the process. like we still provide our candidates with the data set and tell them to prepare it because, yeah, you can use assistance for that, but in the end, you still have to do the work, and it still gives us a good sense of how you interpret the data and how you visualize it, but there are more assistants out there that can help you do that than there were in the past. So basically I think it's getting harder to really understand how good a candidate really is because you don't know how much assistance and help they're getting in the background.

TIM: Yeah, and I wonder if, like, on the take-home case study front Maybe it's just approaching it in the same way we would have a couple of years ago, which is they do the case study; there was always a tiny chance that maybe they hadn't done it all themselves. Anyway, they might have gotten their friend to help them with it. you never know It's unlikely because it's such a specific skill set that it's not like it must happen to be in the data community or whatever; they might not have a lot of people they could lean on. And so in that follow-up interview where you're digging into it, You, it probably becomes quite apparent straight away if they haven't done it because they wouldn't; they would only have that superficial knowledge of it. Whereas if they'd literally written every line of code, researched it, and written every word of that document, they should be able to go to a fairly low depth of understanding. I imagine it might be the same with ChatGPT; if they've just bungled it into ChatGPT and edited a few things here and there, maybe they can answer superficial questions, but they're unlikely to be able to really buttress that as you dig deeper and deeper. Yeah, and that's back to the point of the interview, which is really important. If we're talking about an individual contributor analytics data analytics position, there's a take-it-home case study. They take it home; they prepare it; they have to send it in advance; they have to provide their code in advance.

MARC: And then there will be an analyst or an engine and analytics engineer in the interview that will challenge them on why they decided to build that code the way they did, and that's where you then get into a conversation, and you can't fake it. That's not going to work, so you can fake it at home, but once you have somebody sitting in an interview asking you why you decided to build your code structure the way you did, If your answer is I don't know," basically that's what Copilot suggested. Then you have a problem, and then you're not going to pass the interview. I think it shifts the responsibility and vetting more into the interview, actually even more into the interview. I don't think, not yet at least, do I think that there's no point in doing the take-home but forcing people to do it live? You have a half an hour because I do acknowledge that people have different ways of working, and I think that is biased towards people that can deal with high-stress situations in a half an hour quickly coded. don't know if that really is a fair process

TIM: No, I can say personally I would have hated that. I remember my last role that was in an office, which is a few years ago now. I would write a fair bit of SQL if someone came over to my desk and said, Hey, can you just run me this quick query? and I had to sit there typing SQL, I became like a seal. like I didn't actually have fingers anymore; I was just mashing the keyboard wildly. I'm like, Just go away; give me five minutes; I'll come and give you the data in a minute. I can't sit here under this pressure with you staring at me.

MARC: Yeah, don't watch me doing this. Yeah, although I've had a really good interview process that was memorable to me was for when I was in finance. I would join an FP&A interview process, and they would have a seemingly very simple set of charts already visualized and a slight outline of this situation. I think it was like market growth or whatever. and then they would give that to a candidate and say, You have 20 minutes to look at this, and then we're going to have a discussion about what you read there. What do you actually see in those numbers? and it was amazing how different the outcome was from what people were able to interpret, and I think people in an analyst and FP&A are also analysts of different types, but anyway, they should be able to do that. So there is merit to spontaneous preparation. Just coding, I think it's a little different. I think some people really need their zone, their music, and exactly their environment that they've been accustomed to to be able to really excel on that, and I think that's fair. I want to give the people the time at home to do that.

TIM: Indeed, Mark. One final question: Can you think of a hiring hero or anyone in your experiences who's done hiring in a great way? This could be a person or even a business. Has anyone sprung to mind?

MARC: So in my hiring experience, I've always tried to learn from peers that I got to join in the interview process, and there was definitely one that stood out that was my former boss and CFO at Mobile; her name was Patrizia, and I'd never met someone that was able to do what we talked about, the digging deeper. and it was on point. The questions were really tough but in a friendly manner that it felt completely appropriate because I think there's always a balance of you want to have a nice and positive interview environment, but you also want to challenge the people you want to get that you really want to dig deeper. and how do you balance that? And she did it like it was perfect every time. I was like, How are you doing it? It was amazing to just watch, and I've tried to emulate that when I do interviews because I tried to give the candidates a positive experience because I'm trying to also—I want them to join my company, right? Like I want them to join my team, but at the same time I want to ask tough questions, but how do you balance that? That, to me, is the perfect combination.

TIM: wonderful