Managing Teams with The Iron Triangle of Engineering Leadership with Colin Henry

David: Colin, it’s nice to have you on, man.

Colin: Ledge, thanks for having me.

Ledge: Can you give a two- to three-minute background about yourself and your work so the listeners get to know you a little bit?

Colin: Sure. My name is Colin Henry. I’ve been a director of engineering and senior manager for companies big and small. I’ve spent some time with GE. I’ve spent some time with Hewlett-Packard. I’ve spent time with some smaller startups like Apptio and Simply Measured.

I live here in Seattle, Washington with my wife and my dog just trying to generally do good work and lead as best as I can.

Ledge: Off-mike, you and I were talking a little bit about the experience of moving in and out of engineering tactical doing things and the leadership of engineers and also maybe the whiteboard time of just deep hardcore strategy. And I theorize that maybe you can’t do all those things at the same time as much as we want to.

What has your experience been on the balance of those maybe three core areas?

Colin: I think you hit something on something very true. For the listeners, we’re talking about “You can do, you lead, or you can strategize.” And pick two. Anyone who has ever dealt with the iron triangle knows you can only pick two.

That’s a triangle most people try and stretch in all directions every given day; and I think it’s true of people, in general. We try to do everything all at once and as many times as possible.

For me, spending time in those different segments while only doing two at a time has kind of been the balance in my career and has kind of given me, at least, in the conversations I have, be the “jack of all trades” kind of moniker stapled to my forehead because I’ve done the “do”; I’ve done the leadership; and I’ve strategized.

But, to your point, it’s pick two. You spend the time leading and strategizing but you don’t get your hands in the muck everyday. You can lead and you can do. If you’ve ever spent any time as a dev lead or a program manager, that’s your bread and butter every single day.

And then, the strategize-and-do piece is always this weird situation because you’ve got the people who are leading and doing and you’ve got the part where people are strategizing and leading. Both of them feel like “Okay, we’ve got this covered” but they’re not looking from “Okay, how does this strategy that you’ve laid on the whiteboard with your bosses or your exec team six months ago actively translating down to boots on the ground?”

I’ve found myself in that situation more than a couple of occasions where you kind of looked at your CEO and it’s like, “The strategy you were going after is not actively getting translated or if it is getting translated, it’s not working. You have to try again.”

And even with all the KPIs, sometimes, it really requires you to have that one person who is not going to lie to you and go out with you for a beer regardless of what level they’re at and say, “It’s not working. We need to do something different.”

Ledge: You talk about KPIs. That’s interesting because KPIs of development team ─ I’ve read endless articles that talk to tons of tech leads. There doesn’t seem to be a lot of agreement around those. I just wonder, okay, in the trenches there, you’ve worked for some really big sort of stalwart engineering and some startups. So you know.

What should be measured for a dev team?

Colin: It’s kind of what Peter Drucker used to say: “What gets measured gets managed.” And you have to ask yourself what really matters for your team in either showing a) they’re productive or b) they’re delivering results of some measure between those two.

So, for me, it was always ─ particularly in the larger companies because they always have their big set programs with their metrics and all that kind of stuff. And I was always kind of a startup guy who was thumbing my nose at everybody. I wasn’t getting the support from the executives which drove the program managers absolutely nuts.

But I’ve always looked at it from two axis: one, individual productivity and, two, the team delivering on promises.

These metrics work in both the Agile kind of world and the guys who are still doing Waterfall and even the guys who are doing Scrummerfall, the guys who sit somewhere in between.

I look at successful builds. I look at total coverage. And what I mean is test coverage there. And that always can be kind of a woo-woo metric but I get pretty specific on it on the tactical side.

I look at “What’s our churn if we’re doing Agile? What are our commits versus what we complete? And what do we toss out?” because if you’re dealing with a team that is newly formed, you’re not going to have the steady metrics of “Okay, we know we consistently deliver x amount of work over the defined time period.”

So I actually work with the guys above, and it’s like “These metrics are going to be all over the map probably for about three to six months while we figure this out. And you’re going to have to be comfortable with that.”

I grew up in a military family and one of the things my dad drilled into my head was VUCA ─ volatility, uncertainty, complexity, and ambiguity. And you have to be comfortable with all those if you’re starting a new engineering team ─ and even when you’re fixing an engineering team.

I’d walked into a couple of situations where they were like just “Help, please! Anything you can do. Move the needle.”

It’s “Okay, these metrics are not going to look pretty for the first three to six months, and you just need to be comfortable with that.”

Generally, I’ve been really lucky in having exec teams I’ve worked with and have worked for that have been incredibly supportive in that especially when I’m trying to fix something. But it’s also making them understand what’s going on.

Perfect story: I worked for a large company ─ I’m not going to tell you which one. I came in close to v1 and they had their first security vulnerability and it was a doozy. Someone had hard coded a password into the product and I sat down and looked at my build engineers and said, “Okay, how quickly can we turn on a new build with code modification?” and they said, “Three to five days.”

I looked at them rather perplexed and I was like, “Okay, humor me for a second. Anytime we need to push a change, how long does it take?”

“Three to five days.”

I went, “Okay,” and we took the three to five days and went through the system and got to fix and make sure that the customers were back to good and secure.

And then, I burnt it all down because I found out that the three to five days was taking because they were still manually doing builds. They had no CI/CD pipeline whatsoever.

So it was like, “Okay, we start with the basics. We start with the CI/CD platform and go through that entire exercise.” And then, we started looking at code coverage and the code coverage was somewhere around five percent; and it was only for the really gnarly sections of code that the senior lead who had recently left and really was the only guy writing tests wrote test for it.

When I actually pulled my boss, my boss’s boss and said, “Look, I’m just going to give you a standard report. This is where we’re at,” they looked at me and they wanted to start yelling and then realized I’d only been there for a month or a month and a half.

They said, “Okay. How do you plan to fix it?”

I said, “Funny that you should ask that.”

The entire list of things that I had thought about over the previous two weeks.

“These are all the things we need to fix and do that assessment.

“Okay, once you have those, how quickly can you deliver?”

When we did the napkin to mouth application, we went from saying “once every six months to once every quarter,” and then by the following year, we were releasing once a month and just increasing that velocity.

Ledge: Release velocity comes up a lot as a key metric. You mentioned personal productivity. You’re going to measure that a throughput capacity ─

Colin: I look at it in a “Is this guy or girl hitting their commits when they commit to do something and when things get tossed out?” That happens at a team level for the reporting and above. But from looking below and how I have to lead a team day to day, I look at that interview and it’s like, “Okay, who needs a stick? Who needs a carrot? Who is going through a rough time in life? Who desperately needs a vacation?”

That team I just talked about, two-thirds of that team, needed a vacation when I joined. So I had to start putting people on mandatory vacation and cycle them through while we got through it.

So it’s keeping an eye ─ you can still be in the lead and strategize and look at code; but for the depth of understanding of what’s going, you’re going to have to give the strategy part or the leadership part to really spend time with that if you’re trying to understand what’s going on and giving that advice.

It all comes down to time and attention. We do it on a daily basis. How many times are you going to look at Facebook? How many commits are you going to get done in a day? How many meetings are you going to sit through?

Your attention is divided up on a daily basis and it’s just a matter of what you choose to focus your attention on in that time.

In the course of a given role, you can shift among those three but you can only pick two in a given time.

The individual productivity piece really comes down to understanding the person’s work day to day. And you can tell that through commit velocity. You can do that through commit velocity or the iteration or the release.

It’s just being in tune with those people because if you’re leading, they are your people. You’re responsible for getting them through the nasty stuff and helping them enjoy the good stuff.

Ledge: Obviously, we’re talking a lot about big organizations and “Fix it up as the leader” and all these things where there’s a lot more tolerance for “take some time; fix some things.” You probably also had been the solo engineer in startup land. I think that we find a lot of our folks end up there, at least, on some projects.

Maybe they’re on a 50-person team but, sometimes, they’re the de facto etcetera. In that situation, I wonder how you deal with code reviews or checking yourself when you’re literally the only person in the repo.

Colin: As my wife’s close friend would say, it’s all about your mise en place, to pull a culinary technique out there. It’s your organization. It’s your self-discipline to get yourself through that.

For me, it’s always been going back to the basic of test coverage and “How often can I get something in front of somebody else regardless of whether they’re looking to code.

When I was at a company called “Usermind,” I was the first employee and really the first engineer. I worked with the CTO but the CTO was so busy helping set up the organization and helping raise money. He didn’t have time to help build the prototype.

So getting a code review from him was pretty much a non-starter just by him being a very technically capable person.

So I would write tests and I would show versions of the product in front of the two founders as quickly as possible. But there are always deployment jitters especially when you’re working as a solo artist, so to speak.

A perfect example was when it was demo day for the company, and I was sitting back at the home office while the two founders were down in San Jose. They had given me a rough timescales to when the pitch was going to happen or the demo was going to happen. So I made sure everything was ready to rock and roll.

About two minutes into the demo, I started seeing log error messages and things were going wrong. It was like, okay, I can’t just let this fail in front of them. It’s my responsibility. It’s my ownership.

I reboot the entire system in front of them. Fortunately because there was a lack of sessions and it was a pure prototype environment, the demo kept going without a hitch and they never experienced anything except for one small glitch that the CTO caught at the corner of his eye when they were doing a click-through demo.

So you’re always going to have the preparation and the mise en place. You’re going to go through and do all the stuff and you’re going to do it to the best of your ability.

But when the time comes, you need to own it. You own it at the end of the day. Being there to literally kick the server if you need to is, ultimately, your responsibility. There’s not another DevOps guy who is there to help you. There’s not another whole team behind you. It’s just you.

Another example was when I was very early at Simply Measured. I was working with the CTO and like we talked about in pre-interview, I took over a good chunk.

So when we shook that, I was on deployment and I was sitting there cracking out code and cranking out the deployment on repeat with the final countdown playing in perpetuity until we got it done. It was kind of a nice little drive us crazy kind of thing that forced us to get it done.

But you’re there. You’re forcing code out. You’re making sure it works. You’re rounding square pegs and trying to put them in the hole when you have to.

It’s always interesting because I was talking to a former boss of mine not too long ago because he had to get stuck doing IC work for a little while because his guys were so slammed that he had to sit there and do stuff.

And he goes, “I have to apologize if I ever thought that I could shove you into an IC role while being a leader in my organization.”

Because it’s really hard to switch that context. And I was like, “Yes, you really have to be ready for it, don’t you?” and he goes, “Yes, I do.”

You don’t get that kind of delay or time or time to adjust. No, you don’t.

Something to think about for the future when you bring leaders and you make them do a C work.

Ledge: So you’ve had some experience in the crazy engineering market. I don’t know if you’re paying attention. Obviously, we are immersed in the data of the labor market; and it’s tighter than I’ve ever seen along the engineering lines: prices increasing, people’s expectations not decreasing.

And I just wonder ─ last question ─ about your broad thinking around where things are going for remote and freelance and W2. There’s just so much going on now.

Colin: I think companies have to get more flexible and they have to get comfortable with paying outside of their pay bounds because, at the end of the day, I’m a capitalist. And the market is going to bear what the market’s going to bear.

I was having a really interesting conversation with a corporate recruiter about a year and a half ago. And they’re like the H1 visa so prices are going to come down.

I looked at them and it’s like, “You, obviously, have no concept of economics or politics. You just tightened the market which means prices are going to go through the roof.”

At least, through the local demographics here in Seattle, they have. They’ve gone kind of crazy. Anyone who is familiar with the whole Amazon HQ2, that’s proof in the pudding. They can’t afford to shove any more engineers in downtown Seattle because they own all of the real estate in downtown Seattle at this point and they need more places and they need more engineers.

And everyone else is in the same boat at a lesser degree. So when you have a restricted supply and unlimited demand, what happens?

The price goes up.

The only way to fix it is to increase the pool of engineers and technical talent; and that takes time for all we’ve talked about code.org and the journeyman programmer, I’ll put it that way.

Getting them to a technical acumen that may be of interest to an Amazon or a Microsoft or an Apptio or Simply Measured, that’s a very steep ramp and it takes a lot of time.

When I talk to my alma mater, my particular focus is I try and give them as much practical advice as I possibly can about increasing the student body and increasing the variety of skills that those students come out with because, ultimately, that’s what helps differentiate them in the market and gets them a better paying job.

Ledge: Great insights! Colin, it’s good to have you, man.

Colin: You as well.