Shift to AI – Episode 3 – Meet the CISO Guarding America’s Supply Chain: Securing Rail Infrastructure in the Agentic Era

Presented by Cycode, the Agentic Development Security Platform

Roland Cloutier

Well, guys, welcome to Shift to AI. I’m Roland Cloutier, Global Chief Security Officer, formerly of TikTok, ADP, and EMC/Dell, and this is our first live recording of Shift to AI. Cycode is our partner in delivering this message about the importance of not just the shift to AI, but what that work actually looks like. And I’m really excited about today, because for years security leaders have relied on playbooks – and that’s what we’re going to get into. We can’t rely on playbooks anymore. The shift happens every day, every minute. Pipelines may call models, models may call agents, agents will call other agents. So now we have all of these changes we have to deal with as security leaders, and it means durable work has to move up a level. But our discipline in what we do on a daily basis has to stay in place to be successful. So we’ve called this episode “Playbooks Are Dead. Long Live Frameworks.” The point is that procedures are not useless, but systems do not sit still. We’re going to dive deep into this today.

I’m super excited to have Alex here. My guest today is Alex Levy, Chief Security Officer at BNSF Railway. Alex brings the perspective of a security leader operating in critical infrastructure, and that’s why I wanted him for today – because what Alex and his team do is critical beyond just running a business and ensuring the digital defense of an organization. It’s critical to the economy. It’s critical to the people who live in the communities his company’s rail lines run through.

His recent writing on entropy and honesty is what caught my eye. He argues that the old assumption of predictable machines and unpredictable humans no longer holds. And I think that’s a perfect starting point for this question: if AI makes what we do for a living less deterministic, what frameworks still deserve an affirmative place in our programs and our operating models? Today we’re going to talk about AI code and pipeline posture, software supply chain, security engineering – we’re going to go all over the place with Alex today.

Alex, thank you for joining. I really appreciate it.

Alex Levy

You’re very welcome. Pleasure to be here.

Roland Cloutier

I want to start with a question I ask all my guests, because I think it’s telling in how you think and how you operate. Before you have your second cup of coffee, in that first thirty minutes of the day, what are you looking for to get your day going?

Alex Levy

I tend to spend a lot of time, at least in the morning, trying to understand what the current threat landscape looks like. Every single day we wake up and there’s something in the news – especially when we’re talking about AI – some new agentic break, some new hack perpetrated by a model, or some breakout of a control. So I spend the first half hour of my day just catching up on what happened overnight. I do it before bed, too, because that eight-hour overnight period tends to lead to a lot of interesting turns of events.

Roland Cloutier

I want to pick on that a little, because this week you’ve been talking about breakout models, but there have also been some really interesting attacks over the last few days in critical infrastructure defense. How does that change for you? If it’s a Wall Street Journal moment, I’m a public company, I know my board of directors is going to call – but it’s not the same level of “uh oh” for me as it might be for you. Tell me about when you see impactful incidents through malicious code targeting other critical infrastructure. What do you and your team do?

Take the recent example of the water treatment facilities and reservoirs impacted by OT threats. When you’re in critical infrastructure, especially where there’s a physical plant, OT is a huge target, and with minimal security boundaries in traditional OT environments, it’s very concerning.

Alex Levy

Right. So a lot of what we’re spending time on is figuring out how to harden those infrastructure components against new threats. What was old is now new again – tech that people weren’t focusing on, that threat actors weren’t focusing on, has come full circle and is now easily exploited. You’re starting to see controllers and microcode with newfound exploits we all have to protect against. And that’s scary.

Roland Cloutier

We’ll get to that, because I want to get into advancing pipeline defense when AI is used in OT environments too. But first – you wrote that machines used to be the predictable part of security. When did you stop thinking that was true?

Alex Levy

In the last year or so, I’d say the whole notion of standardized configuration management and predetermined deployment pipelines gave you a deterministic output of what your compute environment actually looked like. You could predict how your servers and applications would function within a constrained environment. That’s no longer the case. You’ve always had chaos on the human side of it; now the previously controllable side is no longer controllable either, because when you put controls into the hands of models and agents that are effectively reasoning, they change that entire dynamic. So there’s chaos on both ends of the spectrum now.

Roland Cloutier

A lot of people probably don’t know this about you, but you didn’t come out of the CFO’s office or the audit office. You’re an engineer, by God – you came from cyber engineering, and you’re still an engineer at heart. As the engineer in your heart, leading all the functions across your portfolio of programs, how has this shift to AI changed how you look at your engineering teams and their code, specifically code developed for AI?

Alex Levy

Some of the trends I’ve seen include an overreliance on AI, especially by our developers – and developers everywhere. Every company needs to be cognizant of this. If you ran a simulation where, after your developers had been vibe coding for three or six months, you took it away for a day and saw what happened – you’d create tremendous chaos in your environment, because people who’d been coding for twenty, twenty-five years forget how, simply because they’ve become so dependent on the tools. So if you think less about the tech and more about the human-risk side of things, that’s a big concern people need to be aware of.

Roland Cloutier

Let’s get into the playbook discussion a bit. What’s the difference between a playbook that decays and a framework that compounds?

Alex Levy

I think we’ve all seen decay and drift over time – we all have our fun challenges with configuration management. Frameworks are how you define the rules of the road, but you need to apply them to your specific vertical, to your environment, in a way that makes sense for how you and your teams operate. They’re infinitely malleable. That’s why I try to implement a framework-level discussion in everything we do, rather than drawing hard and fast lines in the sand for how we operate – especially as AI becomes more and more prevalent. The hard lines we drew six months ago no longer apply. You have to be flexible as the world shifts around you.

Roland Cloutier

I don’t know how deep you can get into this, but I agree with you one hundred percent – frameworks have to be malleable to take the situation, the tech, the problem you’re dealing with, and match it to the control necessary to complete the business mission, whatever that may be. But when you’re in critical infrastructure, and I’ve been there – there’s oversight, there’s requirements, there’s provability, there’s liability if a negative-impact event happens. How do you ensure your operating teams, your compliance teams, everyone is on the same page about what’s being applied to that framework? What’s your secret sauce? I know you do it well, but it’s probably one of the hardest things to do as a CISO.

Alex Levy

It is. When you’re dealing with heavy regulatory environments, it’s pretty common that the regulations don’t catch up to the tech for quite a while. You have to have a really good observability program to prove out what you’re doing in the environment, and that your controls are working the way they should – regardless of whether those controls are performed by a human, some automation, or AI. The end result is ultimately what matters, along with the journey to it. When you have internal compliance and audit teams, the goal is to make them part of the discussion when you’re building out something new. That way they understand how things are going to flow, they can ask questions up front, and they get comfortable with the road you’re taking, rather than the traditional, solid boundaries of control we’re all used to.

Roland Cloutier

Things are going to be dynamic. The controls, if done properly, will pivot on the fly as the tech does, and you have to update your documentation and policies to adapt. Otherwise, you’re going to fail every audit. So when AI can write code, change code, review code, or operate inside delivery pipelines autonomously – what does AI code and pipeline posture management need to include, if we’re saying we’re going to rely on it because it probably does the job better and faster and keeps up with everything else we’re monitoring, defending, and controlling?

Alex Levy

There are a couple of different schools of thought here. The current school of thought is that every so many check-ins by a model, you’d have some type of human review just to keep it honest. But I think that’s going to quickly swamp your teams – it’s not sustainable given the volume of code being written by models. Ultimately, I think we’ll end up with a multi-model, or triplicate-model, solution, where your models are checking themselves, using disparate models. Say you’re using public models from OpenAI and Opus from Anthropic – from a security perspective, we’re pitting the models against each other to validate their findings. We’re trying to give self-regulating oversight to everything we’re doing, because we know humans can’t keep up with the volume.

Roland Cloutier

How do you create that checkpoint? I mean, obviously AI is going to do the work to validate the AI – where does human validation come into play, from your point of view?

Alex Levy

It’s in the deployment area. That’s where we see most of the risk – when you start to automate your workflow end to end, from the moment a developer commits code, and then let the model promote that all the way through to production, doing its own self-testing, QA, and in some cases UAT before it goes to production. That’s pretty dangerous, at least with the guardrails we have today. It may not be that way in six months – we’ve seen leaps – but I try to do as much as I can with my teams to validate what’s going into our production infrastructure. AI is helping us develop at a rapid rate, but we don’t want it to become a liability to our stability.

Roland Cloutier

You just made me think about something as you’re putting things into the production environment – where does our production environment start and stop? This whole discussion around supply chain and digital supply chain just got flipped. Is it third-party risk anymore, or is it just risk? Does software supply chain now include model supply chain, prompt supply chain, tool supply chain, identity supply chain, OT supply chain, PLC supply chain? It’s crazy, all the things you have to think about now. How are you looking at this?

Alex Levy

Looking at this going forward, TPRM has expanded. If you think about third-party risk, unless you have complete command and control from birth to death, it’s third-party risk. That entire area has grown. If you’re using models from Anthropic, OpenAI, Google, or whoever, you’re at the mercy of their capabilities and their availability. So if you build that into your workflow and your production lifecycle, you also have to build resiliency around it. We all saw what happened with Fable – it was released, and thirteen hours later it was unreleased. If you’re relying on third-party models, at any point they could disappear. So you need to build for resiliency. We all know vulnerability management is at a crossroads – we can’t patch nearly as fast as the volume coming out.

Roland Cloutier

So the discussion for how we manage these environments going forward is really a discussion in blast-radius mitigation and resiliency. Let’s talk about resiliency a bit, since you brought it up – I’m a converged CSO, I’ve been doing resiliency and trustworthy defense for a long time, but it means something different in freight rail. What does it mean in practical terms, given the criticality and sensitivity of your job? You might have people listening from banks, retail, software companies – very different from what you do. How do you look at it differently now that you’re in critical infrastructure?

Alex Levy

Coming from financial services for years, I looked at it through a very specific lens, and that’s changed dramatically for me since joining the railroad. I didn’t have an appreciation for how much the transportation system is the backbone of everything that happens in the country. Every commodity, at some point, moves by rail – food on the shelves, goods in stores, all of it. For every train we run, we’re taking three hundred trucks off the road. Multiply that by thousands of trains a day, and it adds up quickly. If we were unable to move trains or ship products, it’s not only catastrophic for the consumer – it’s catastrophic for the economy, and subsequently the economies of many other countries, because everything we purchase through overseas purchasing programs gets shipped by rail at some point in the chain.

Roland Cloutier

The downstream residual risk you have to look at to make your daily decisions goes so far beyond rail – and it’s not true that you only operate in the United States, because the risk extends so far beyond it. It’s really impactful. It’s such an interesting space to be in.

Alex Levy

It’s a lot of fun, nerve-wracking, but a lot of fun. We have products coming in every single day from thirty, forty, fifty countries, and they all rely on us to deliver to the end consumer.

Roland Cloutier

It’s funny – we were talking before this – I always wanted to do rail. I always wanted that ability to have such an impact in this mission space, because it’s one of the few jobs where there’s mass impact in every town, every state. It’s fantastic. I want to get into something you touched on earlier – being okay with things changing, being malleable, graceful failure. You’ve said it’s really a question of how contained the blast is and how fast you see it. What telemetry, controls, or evidence processes have become non-negotiable? They just have to be there.

Alex Levy

All security tooling we use is mandated as we roll out – container visibility, edge protection, some of the normal things we’ve all been working on for years. But one area that’s completely non-negotiable, where you can’t really be flexible, is identity. We’ve all been talking about identity as the perimeter for years, and the cloud discussion really brought identity as a perimeter to the forefront – but now identity is the gating factor. When you’re looking at agent and agent-to-agent usage, if you’re permissioning your agents, what can they get to? What kind of damage could they cause if they create some type of maelstrom in your environment? The only real lockdown capability that exists today is identity-based. So we really should be focusing on permissioning our agents and agentic workflows as minimally as possible. We’ve all been talking about least privilege for twenty years – it has to happen now, otherwise the damage can be catastrophic.

Roland Cloutier

I want to jump into culture, because this is such a huge culture change – and you’ve served under great leaders, and you are one. I know you believe culture is critical to setting the mission and the mindset. How do you build a culture where someone can say, “I think I just did something risky with an AI tool – can I get help fast?” How do you set the expectation that it’s okay to fail?

Alex Levy

We have to live in a blame-free society, especially in security – fix the problem, not the blame. You want to spend as much time as you can drilling that into the workforce, not just within your tech organization but everywhere, as AI proliferates through every business unit and every company. People need to feel comfortable speaking up when they believe they’ve made a mistake, even if it turns out to be a false positive. You want to create that culture. We started doing that with spam years ago, when people began reporting it, and we got people pretty good – and appropriately paranoid – about it. It took some time, but we got there. We need to get there with AI as well.

Roland Cloutier

Let’s talk about our people, our teams – because we’re accountable for making sure we have the right resources in place to do the jobs we’re asking them to do, and over the next twenty-four months our organizations are going to change dynamically. I want to ask about the top two skill sets in a couple of different areas. First, because you’re an engineer at heart – security engineers: what are the top two skill sets they need? And on the other end of the spectrum, from a compliance standpoint – what are the top two skill sets someone in a GRC-type program is going to need?

Alex Levy

On the engineering side, I’m looking for people who can do hunt and dynamic defense – people who can really work to generate and identify patterns of bad behavior. In this case, it’s not human bad behavior, it’s model behavior. We need people who can help build and hunt for problems in the environment and build observability – people who work within our pipelines to see different potential problems as they arise.

On the GRC side, I’m looking for people who understand how to work with regulatory entities, audit, and compliance teams to really frame out what the new world looks like from a policy perspective. Policies and standards are going to become very fluid. In large organizations, you’ll find a policy that was last updated in 2019, because not much has changed from an overarching standards perspective in that time – maybe a protocol or cipher strength here and there, but the general policies haven’t adopted much change in seven or eight years. That’s all going to change. Take vulnerability management as an example: how many people have their environment set up so you test patches in dev, then QA, then production, with A/B testing along the way? Maybe that took a week or a week and a half. You don’t have that kind of time anymore.

Roland Cloutier

We need to shift to allow for things that are significantly more dynamic. This is one of the areas I’m most excited about when I think about work that can change, because right now we create these policies, frameworks, or guardrails, and then the dev organization has to go figure it out, and then they need a meeting with an engineer, and then compliance takes weeks to get an answer out to market. We’re going to be able to let our agentic workforce do that on behalf of our internal customers. I want to stick on the workforce question for a minute, because it’s so critically important we get this right. If we have some number of employees today, but we’re going to have ten times, or a hundred times, that many agentic employees in the future – how do you look at the future organization of the organization itself to be able to accomplish that?

Alex Levy

It’s a great question. I think we’re going to end up with more and more skilled engineers, and fewer click-ops and process-based people over time, purely because we need engineers who can keep up – people who can design methods of keeping pace with the sheer volume of output that’s occurring. Human-based controls, in a year or eighteen months, aren’t really going to be a thing anymore. The only way you’re going to sustain the rate at which the models are improving is to have engineers building compliance guardrails and effectively a bubble around your environment.

Roland Cloutier

We’re talking about frameworks, controls, and models across different parts of the program. From a framework perspective, which security frameworks have survived contact with AI, and which have become, as you’ve said, more ceremonial at this point? Give us a hint of what’s lasting.

Alex Levy

I don’t know that there’s necessarily one particular framework – I think there are pieces of different frameworks that make sense in different environments. Take vulnerability management – it’s topical, but look at just the last three months: the shift from patching everything all the time to focusing on non-exploitable vulnerabilities. As you read into the different frameworks, they’re going to adapt too – they’ll have to. We’re going to see them chip away at things that are more agentic-friendly and specific. I can’t say any one of them will permanently last as a whole. You have to look at them as tools in your toolbox – pick this piece from NIST, this piece from ISO 27001, or whatever framework you want, and make it topical to you.

Roland Cloutier

Since you brought up NIST, and I know you have a great respect for and reliance on it – where do NIST’s SSDF or CSF-style practices still help, and where do we need them to change for the AI era?

Alex Levy

I think the change has to be about dynamic defense – that’s really where everybody needs to focus. For those of us who were at Black Hat, we saw a lot of vendors focusing on the new agentic SOC – the ability for environments to pivot automatically when they see threats. I think the frameworks need to adapt to something that looks very dynamic. The expectation is going to be, “this is how you design agent defense,” whatever that means in your world. I spoke with probably ten different people on that topic and got ten different modes of operation. So until we have some consensus, that’s really where the frameworks will get some teeth – we need to drive consensus on what good looks like in an ever-changing world.

Roland Cloutier

I love this concept of agentic SOC – I mean, the reality is it’s going to have to happen, whether we segment the detection and auto-defense capabilities into separate platforms or not. The only people I know who’ve done it one hundred percent are, of course, the model companies – I think Anthropic has done their own fully agentic SOC. How long do you think it’ll be before organizations your size are able to move to even greater than seventy-five percent agentic SOC?

Alex Levy

We’re already greater than seventy-five percent. We’ve been focused on it solidly for about a year now. It really comes down to thinking about the triage aspects of the massive alerting we get. Previously you’d have a SOC of ten to two hundred people, depending on your size, reviewing all the alerts coming through from various observability platforms, pulling data, running it through SOAR – and honestly, it was never truly efficient. Take a look at it: ninety-nine percent of what comes through is generally noise. And if you have that data over two, three, or four years, that’s the output of humans’ work at looking for noise. That’s what you should be training your models on. From there, you let your model say, “this is what’s actually relevant for my environment, this is the type of attacks we see, this is what you should be looking for” – and it can subsequently auto-triage from there.