Software dominates high-traffic commerce with Leon Fayer

Ledge: Leon, great to have you on. Thanks for joining us.

Leon: Hey, thanks for having me, Ledge. Happy to be here.

Ledge: Fantastic. Would you please just give a little background story – yourself, your work? I know you’ve been around for a while, been doing a lot of interesting things, so love to get the audience to get to know you a little bit.

Leon: I always wanted to be in technology ever since I was a kid, shocking enough, so I started rather early. I started pretty much the same time the web started – around early ’90s? I know I’m dating myself here.

Frankly I’ve done a bit of everything and anything, I’ve done a bunch of government contracts earlier on, realized it’s not for me just because of the pace. During the first internet boom, done a bunch of startups. Some were more successful than others, as you can imagine. For many, many years really I’ve been doing consulting for the high traffic systems applications organizations of the world. Talking about Wikipedias, National Geographics, AdSense of the world.

I’ve got some interesting insight in how the sausage is made, so to speak, in some of those organizations, both technological non-technological ones that are powered by technology.

Ledge: Yeah. Off-mike, you and I were talking about just how you’ve moved in the human direction of, high-performance systems need high-performance organizations or it’s just not going to walk.

I kind of think of it like pets. They say owners and pets start to look the same. I think organization dysfunction or underperforming organizations lead to underperforming systems, always. They almost model the way that the humans have behaved, starts to be embedded into the software.

I don’t know how you think about that but, please, dive in a little bit.

Leon: Yeah, you’re right. I think a lot of times the technology is the easiest part of the equation, because there is a multiple. You can use a number of different technologies, especially nowadays when the new database or programming language is getting spun up almost every week – for better or worse, let’s be honest about it. But at the end of the day, the technology’s role is to support the business. If technology just exists for the sake of technology, then it’s not really required.

Approaching technological aspect from a perspective of, how can it support the business drivers? Whether that it’s revenue or efficiency gains, or we can deliver better value for the customer, is the way to go.

Really, whereas even 10 years, maybe 15 years ago, there were technology companies and there were traditional brick-and-mortar back-office IT companies. Nowadays, every company is a technology company. You’re looking at industries that you never thought about as technology.

Capital One is a great example. It’s a huge bank, credit card, and they’ve been positioning themselves as a technology company recently.

Ledge: Yeah, There’s endless cases now where you don’t even think about it, but even manufacturing – IoT, distribution, Everything is being upended. Retail. Forget about it. The retail apocalypse. There’s so much stuff now.

Leon: Look at what Amazon has done to its competitors – brick and mortar competitors.

Ledge: Yeah, and how healthcare is about to just freak out if Amazon starts to come in. Ultimately, whether you’re moving goods or bits, you’re just dominated by software and how those systems work together.

Leon: Frankly, a fantastic example of that would be media companies. Traditional media companies – print magazines, newspapers – they’ve all been seeing steadily declining sales over the past decade, probably. So jumped on the bandwagon, so to speak earlier than others, and been more successful where others have been dragging behind. It is an interesting shift in mentality.

Ledge: How are you advising companies or talking about high-performance organizations? You talked about, many times this is a change management type of paradigm because you’re going into a legacy situation. All the stuff was there, but it’s changing the macro environment so fast that it’s putting enormous pressure on these organizations to think differently and to even staff differently. You have all kinds of people issues there.

What’s the cutting edge right now? What are you thinking about when you talk to audiences about this topic?

Leon: It’s interesting. It’s actually one of the things that I’ve been doing for the past couple of months, is trying to align the organizational goals with what technology is to be. Oftentimes, it’s approached purely as a technology problem; hey, we have a legacy monolith that’s been running for the past decade, it’s using ancient technology, so let’s spend the next six months…

Ledge: “Microservices!”.

Leon: Exactly. “We’re going to go microservices! We’re going to go full stack. We’re going to hire…”

Ledge: Wait, wait. Serverless.

Leon: Yeah, that’s right. We’ve got to serverless as well.

Ledge: AI. Machine learning. Bitcoin. Blockchain.

Leon: Exactly. You need all of those to be cutting edge, and you see many, many companies spend frankly enormous amounts of money into re-architecting what they have, then launching it – or never launching it really – and failing miserably.

Ledge: Not even having the human infrastructure to deal with that. No DevOps.

I had another guest I thought was great, who said, “You know, look, when you’re on the bleeding edge, you’re going to bleed.” That’s absolutely right.

Leon: Frankly when you’re on the bleeding edge, a lot of the immediate concerns that you have, even on your technology operational side, is how do you deal with your SLAs. How do you deal with responses. How do you deal when something does break.

If you look at the enterprise software, if something broke you didn’t have to fix it for months and months if not longer because it would come out with the next patch that would just ship on the CD to somebody.

That mentality has shifted tremendously with the whole continuous delivery model. If you’re running anything web related – whether it’s a SaaS, whether it’s frankly even your blog – any change that you make is available to the world, whether you like it or not. Which means every mistake that you make is going to be available to the world, whether you like it or not.

Ledge: When every customer has a social media or some way to complain about what you’re doing in a public fashion if you don’t fix it right away.

Leon: Absolutely. There’s a number of tangible numbers that you can put on the [00:06:47. It’s both monetary and long-term opportunity-wise.

Thinking from a performance perspective, I want to say, just 200 milliseconds of performance improvement increases conversion of sales by 7%. That’s huge. That’s just because people are impatient. Everybody nowadays will look around, “Ooh, a piece of candy,” and if the pace doesn’t allow it within two seconds they go to their competitors, because thankfully there’s a whole bunch of [00:07:15].

Yeah, so there’s a lot of consideration. Again, all those consideration come from delivery of the end product. Which supports starts to go? Whether to supports sales, whether to support inbound leads. Frankly, whether to support customers who are paying money, who are trying to have a good experience and you want to retain them.

As you’re undergoing that rewrite or re-innovation or whatever you want to call it frankly, technology is great because you will need to support your growing needs. Chances are if you’re undergoing this you’re already bottlenecked somewhere. But realistically, you will need to rethink your work structure, and more important you will need to rethink your whole delivery pipeline.

The whole concept is fantastic in that regard, because it forces you to look at each individual step at your deliver chain and say, hey, can we improve this particular one? Can we make this step more efficient? Is this step a bottleneck? If somebody goes on vacation, are we completely dead in the water for the next two weeks? Not that it could even happen to anybody.

Ledge: Never heard of that.

Leon: But beyond that, again, you’re focusing on delivery. The focus should be, what is the organizational goal for delivery to their customers? If those are not aligned, then you have a problem.

Ledge: Those are not even defined in some cases. I mean, you could walk into any number of companies and they don’t really know what they do.

Leon: You’re absolutely right. Again, going back to those traditional companies that were not built technology first, they still have a lot of that brick and mortar technology where the IT is somebody who sits in a basement with the lights off and just codes.

Ledge: I was that guy, at a media company. I can tell you how that worked out.

Leon: I’ve seen it time and time again. They’re just not used to the fact that technology is the core part of the business now, and the needs to be traded to the first classes.

Ledge: There are companies that directly sell technology; software companies that actually sell that product. Then there’s everybody else who both consumes and sells something that’s completely enabled by a software work chain that goes through every function of the business.

What we might think of as IT is still, in a broader scale maybe is, shipping and logistics and operations and finance. Maybe what ERP systems were trying to do – let’s tie everything together. Data driven marketing. Analytics. All this stuff.

It’s very hard to escape now, and where we had a series there of big stories where big companies would try to people go over their old school business with technology to make the old system sort of more efficient, that really didn’t maybe grab all the value that was there. It just sort of added a little bit of sheen on top of the old pile of operations.

Leon: Interestingly enough, it was a prevalent message with all the organizations who tried to adopt DevOps. I say adopt DevOps, but in their mind they wanted to buy DevOps – which is a cardinal sin in my head.

So many people I’ve seen and have talked when I was consulting. Going into an organization who is interested and who was willing and had all the buy-in from the executives. “We want to change. We want to go through a digital transformation. We want to improve our delivery.”

I would go in and identify the first bottlenecks, immediate bottlenecks which were up there. Like your delivery pipeline, it takes three weeks between the developer hand-off to a consumer. We need to solve certain things. Or, you need to add this role, so eliminate this step in the process, whatever that may be. It’s a process change, effectively. There’s not a technology change. It’s nothing that’s got to be replaced. They don’t have to go to the cloud. Don’t have to throw any. It’s literally a process change they need to do to become more efficient, to improve their productivity, to reduce number of errors.

You present those options and you hear back, “No, no, no, no, no.”

Ledge: No. We just wanted DevOps.

Leon: “We can’t change any of the processes.” You’re looking at them thinking, well, you want to change, you want to improve, yet you’re not willing to change the processes that are the bottlenecks.

Ledge: But, can’t we?

Leon: It’s not the technology but it is your project. Funny, it’s as if you’ve done that once or twice before.

Ledge: Well, I think everybody comes with the best of intentions. I think that’s the danger spot, is that you can read a lot of blog posts and kind of think like, yeah, I ought to do these things. When the rubber hits the road, one of the biggest issues that’s going to happen is wrestling with personnel and skill inventory.

That, you might have the best, nicest people who have been around for a while, but across the organization they don’t have a level of skillset. You might need to invest a year. If you want to develop those people, fine. Or, you’re going to have wrestle with the difficult decisions of replacing humans, which carries a lot more cultural implication and costs.

Leon: Yeah. Frankly that’s how you cascade. It starts with setting cascading goals.

Organizationally, we want to support a growth, for example. Then you move it a little down and say, well, to support the growth what do we need to do on the technology side? We’ll need to modernize the system. Well, in order to modernize the system in a timely fashion – because those are goals you set per quarter or on a semiannual basis, whatever that may be – what do we need to do? Do we need to change our technology? Do we need to change our team? Do we need to change our process to be more effective? Do we need to change our price to reduce the number of defects that are coming in, so we can focus on more new development?

All those questions need to ask in support of that eventual goal that you’re trying to [00:13:20].

Ledge: Do you often staff change separately than the legacy? You could imagine it like, hey, let’s build up the right thing over here in a different section of the company versus having to change each little piece of the monolith.

You’re really organizationalizing and conceptualizing microservices as your org. That’s rife with problems as well.

Leon: It depends. It depends a lot on the size of the team, on the scale of the team. and on the knowledge base. Building something, and you’ve seen it many, many time again, building something with new people who have no tribal knowledge about how the system currently works, and more often than not, very little documentation. They build something to mimic it. Then, if you’re doing a gap analysis and you end up with something that doesn’t work.

Ledge: Well, I use those two dangerous words of feature parity.

Leon: Exactly. On the other hand, you’ve also got to be mindful of your team itself. If you do have a team of strong engineers who are passionate, who are interested in learning, who are interested in evolving to a new process, to new languages, to new technology stack, you want to encourage that. You don’t want to lose those people.

At the end of the day it is a change, and not everybody… Well, nobody likes change, really, but some people are much better at dealing with change than others.

Ledge: You make a good point about the tribal knowledge. I think that’s the piece that very often gets missed in these types of initiatives. It has to do with organizational tribal knowledge and also just the domain expertise that comes from years and years of…

No one set out to do this in a junky, incorrect way. They set out to build the system at the time, knowing what they knew, given the resources they were given. Just tearing it all down or saying, when you look at a legacy code base, “I can’t believe you did that way. That was so dumb.” Well, it was probably for a good reason that you just don’t know anymore.

That’s the challenge. We could sit all around and we could document forever and never do any work, and that would be inconsumable.

We did a masterful job there of cataloguing all the things that are possibly a disaster. Let’s go to some minutes of talking about, hey, I’m up against this type of situation, I know I need to modernize one way or another, it’s time to move forward, digital transformation or not. I’ve got to do stuff on a daily basis that matters.

How do you distill that down into just the best first steps for people to think about? More businesses are in this situation than there are starting anew.

Leon: Right. There’s a couple of steps that you’ve to acknowledge before you even start. First of all, you’ve got to acknowledge that there is already a problem. A lot of people won’t even admit it. It’s a whole admital phase.

You talk to people, especially who’s been in a legacy organization for many years, they are comfortable with where they are. They’re just comfortable. It’s complacency, let’s be honest about it.

The conversation you have is like, “Look, we want to change this, we want to move to this,” and the answer you get from them this, “Well, why? We’re better than we were yesterday or it works now.” There’s much resistance to change.

One of the first thing is to admit there is a problem, because otherwise either you as an expert wouldn’t be there, or there wouldn’t be a call from executives to do something. There is a problem. Whether it’s financial, or something, there is a problem.

What was I going to say? I lost my train of thought.

A second thing is, you’ve got to understand where those problem are. Coming in and saying, “Well, we’re running a legacy and everything is a crap. We’re getting too many defects,” is great, but if you don’t have a tangible number to justify it then you don’t even really know if it’s bad or if it’s worse or if it’s actually better than.

Ledge: Is that a monitoring question? It’s just starting to capture data in any sense?

Leon: Yeah. It is monetary. It is observability. It is generally a question of collecting metrics across the board.

I’m a huge proponent, and I can talk about this for hours, about measuring from business down. So you understand how business is doing, and how the changes that you do within your team, within technology team, within technology process, within technology itself, affect those metrics. Does the revenue go up or down when you make performance improvements? When you release a new sprint, what is the value on the visitors or their perceived performance? All those affect the bottom line.

The bottom line for business is money. It doesn’t matter how you manage it. You can manage it in the page views, or you can manage it the new users, or you can manage it the actual hard core revenue. At the end of the day it’s money.

Understand where the problems lie. For that, value stream mapping I touched on before is important because you understand whether those problems, that you can observe now and measure, are taken from the particular piece of the process. Then you can optimize further. Really you cannot improve what you can’t measure.

Ledge: Is there a quick way for listeners to maybe just dive into value stream mapping?

That’s a whole discipline, but if there’s some interest there to take that holistic view during the design process, how does anybody even get into that?

Leon: There’s a lot of good material on that. Just Google and it will give you a lot of good definitions.

In essence, what it is, you go to the whiteboard and you define your process from inception to delivery. You define those steps and then you measure individual efficiency of each. So, how long does it take from a CEO to call product and say, “Hey, I have this great new idea, Let’s put it to market because we need to get it done,” to when they put together the requirements? Where that formalizes into functional requirements for the developers, for how long it gets developed for. What the development process is actually, because you’re trying to go as a smaller components [00:19:55]. How many times does a ticket come back from QA because QA found bugs in it? That kind of thing.

Then, how long it takes for it to be approved. How long does it take to be… Regression test. How long do the get deployed? All those steps can be measured.

I’ve seen a situation where literally the deployment process was contingent on one guy, and if he was out deployment won’t happen for a week or two.

Ledge: Right. That happens all the time.

Leon: In. There is one guy who sits in the corner who’s responsible for everything, so it’s a hit-by-the-bus factor. That shouldn’t happen.

Even more so, there is certain efficiencies. Over-bloated teams, is one of the things I’ve seen. You have teams that are stand-up that are the size of 15 to 25 people. So your stand-ups take half-an-hour to 45 minutes, then nobody actually gets a chance to do anything and it’s just a routine to put a check box saying we are following agile methodology.

There’s a lot of things that. There are a lot of knobs and buttons that you can adjust in order to optimize your process.

Ledge: It makes me think that you’re striking on such a good summation of why third party consultants are not a waste of time. As a third party, you can walk into that organization and notice and see those things in a way that nobody can when they’re sitting inside the system.

Leon: Absolutely. Absolutely. It’s funny, I think me and you started talking around my talk about biases in tech.

Ledge: Oh, nice. Yeah.

Leon: One of those is, we talked a little bit about comfort and complacency. It’s a whole at times. Even when you have new people come into an organization, so let’s say into existing teams, into mature teams, and they come in with ideas. They look at it, it’s like, “Oh my god, why are we doing it this way? We should change. Why are we still running decommissioned version of Linux kernels? It’s supported anymore.”

People say, “Yeah, we know. We know. It’s on our list to fix,” and after a certain you see a shift where the people who were so eager and excited when they started get into the sameof saying, “Yeah, I know, but there are so many other issues that we need to address so it’ll be just there until…”

Ledge: “There hasn’t been a zero day, so just let it go until a 100 million records get hacked.”

Leon: Exactly. That’s the problem. There is a certain level of comfort that settles in.

Frankly, in a lot of organizations, technology team is not given an opportunity to see the larger picture. I know I’m being harsh on our technology processes, but it spawns way outside of technology. If business units don’t provide you with the data or the guidance of what it is – if you don’t understand the effect that your change has on the business, you don’t know how to improve this. Again, you can’t improve what you can’t measure, and if you don’t get those metrics then you push out code, kind of wash your hands, and say , on to the next sprint.

Ledge: If you don’t feel supported. It’s just, “Nobody listens to me when I talk anyway.” You get this balkanization of IT and the software and product engineering functions. C

Complacency doesn’t just come from comfort, it also comes from discomfort. “I don’t feel listened to.”

If my organization is giving lip service to feedback but not taking it, then you can just go, “Well, it’s easier for me to take my check and just kind of clock out at the end of the day and go play with my kids.”

I think that stuff happens even with good intentions because, organizations, every human you add to it, it just becomes more complex. It’s hard to keep everybody up to date on everything.

Leon: To your point, that is why so often you see outside consultants coming in on every level. Personally, I’ve been brought in by executives saying, “Hey, can you come in and just sit down with our technology team and understand why we’re not delivering as quickly as we want, why we’re having such a high rate of defects? Basically do an assessment of what we’re doing.”

I’ve also had VPs of technology and engineering asking, “Hey, can you come in and look at it with a fresh eye and provide some feedback to our business units?”

Ledge: Executives.

Leon: Saying, “Hey,”… Exactly. “Look, we need to do this in order to accomplish our goals.”

Ledge: We’ve been screaming about this and nobody would give us budget. Or, we do all new features and we don’t work on tech debt.

Leon: It’s funny enough how it goes, it all boils down to communication.

Anecdotally, I’ve seen where the business would bring us in and in a hallway conversation. A CFO asked me if, in my investigations and dealing with the team, I found out why the financial reports weren’t working for the past three weeks. I’ve asked the technology team and the answer was, “Yeah, we heard about it but it’s low on our priority chain.” Like, “Let me rephrase this question. If I told you that your paycheck depends on that report being run by finance, would it be higher in your priority than it is now?,” because that’s the case.

Ledge: Well, and they’re going to hear from product, and they’re going to hear from everybody else. Like everybody wants everything all the time.

Leon: The other way around. As you’re coming in and you come into product or you come in to CEO and say, “Look. You have this great vision. You have these goals, you want to increase sales by 30% by the end of this quarter. That’s great. Your sales team is targeting and that’s fantastic. I can guarantee you that if you increase your traffic by 5% the system will collapse without doing anything about it. In order to achieve your goal, you can’t do that without the technical team actually doing some changes and improving what they do before you do this.”

There’s a direct correlation between the two and it is bi-directional really.

Ledge: Man, we’re going to run out of time here. It might be the summation is, just do all the things at the same time, try to balance. Good luck, this is hard. Start out every day with…

If I could summarize, I think it’s like, build a trustful and communication-heavy organization. Put on your listening hat no matter where you are because it’s important, and the feedback will come through. Put the hard work into spending a couple of days at the whiteboard setting priorities and making sure everybody is on the same page.

Leon: Yeah, well. You’re right. It’s not easy. If it was easy, well, we wouldn’t have these problems.

Especially, it’s more prevalent in organizations that already have the established processes that are very slow to move and change them.

At the end of the day, approach it as a goal driven exercise. You have goals that you’ve got to reach. Your goals have got to support the overall business goals. Approach from that perspective.

Everything you do and everywhere you try to focus your efforts, measure it against that. How will it support my goals and eventually organizational goals? You set up your OKR – I don’t know if everybody is familiar with OKR, which is Objective Key Results measurements. You set those up, and I know this sounds extremely corporate, but effectively what they are is a benchmark for you to gauge your priorities your effort.

Ledge: OKRs I think came from working Google. Is that right? I forget who kicked that off but, yeah.

Leon: I forget when it came on. Effectively, you’ve got to benchmark something. I can’t stress this enough. You also have got to focus on something that’s going to bring the largest impact to the overall goals.

Ledge: Whatever you measure will get managed. Measure the right things.

Leon: Exactly.

Ledge: Well, Leon, good spending time with you. Great insights, and I look forward to the next chapter once you have some more stories to tell.

Leon: Anytime. It was a pleasure.