The Token Dilemma: What CISOs Need to Know About AI Cost and Tokenomics

Shift to AI – Episode 5 – The Token Dilemma: What CISOs Need to Know About AI Cost and Tokenomics

Presented by Cycode, the Agentic Development Security Platform

Guest: Poonacha Kongetira, CEO of Classie AI and former Google and NVIDIA infrastructure leader

Roland Cloutier

Hi everyone, this is Roland Cloutier, Global Chief Security Officer and Digital Business Executive, here with another episode of Shift to AI, presented by Cycode.

AI adoption has been framed as a capability race – move faster, automate more, and put AI into every workflow. But senior leaders are starting to run into different questions. What does this actually cost? And where is AI worth paying for? That is the token dilemma.

Tokens are not just a technical billing unit. They are becoming a new way to understand the economics of AI-enabled work. Input, output, context, caching, reasoning, agent loops, model choice, and deployment models all affect cost. For CISOs, security executives, and risk and privacy executives, the goal is not to become pricing analysts. The goal is to ask better questions. Is this workflow deterministic or judgment-based? Does more context improve the outcome? What is the assumed business outcome? What is the cost per completed task? What happens when an agent loops through the same model call fifty times?

And when the CFO or CIO asks whether the organization should go all-in on AI, the right answer is more nuanced than yes or no. AI everywhere is not a strategy. AI where context changes the outcome is strategy.

So I am super excited today because our guest is like the chief of tokenomics for me. My guest is Poonacha Kongetira, CEO and co-founder of Classie AI. Poonacha has built and led teams across the AI infrastructure and product landscape, including leadership experience at Google and NVIDIA. That background matters for this conversation, because token economics is not only about software pricing – it’s connected to the infrastructure, the inference, model behavior, workload design, product strategy, and how enterprises make cost decisions at scale. Poonacha and I have worked together over the past several months, and what I really value is the way he brings a very precise systems-level perspective to AI without losing the executive leadership lens – or losing me.

So today we are going to make token economics understandable for security leaders, and practical enough to use in a CFO or CIO conversation. Poonacha, thanks for joining me today.

Poonacha Kongetira

Thanks, Roland. Always a pleasure – we always have great conversations, and it’s nice to be able to share this with your wider audience.

Roland Cloutier

All right, Poonacha, let’s jump right in. Walk me through the first thirty minutes of your morning – what are you looking at before you’ve had your second cup of coffee, or your second pot of tea? And how about the last thirty minutes of your day – what does that look like?

Poonacha Kongetira

The last thirty minutes of the day first, I think. It’s amazing how much actually happens in a day, so I like to regroup at the end of it – review what happened during the day, not just in my own organization or with customers, but also what happened in the AI universe, and decide what I’m actually going to do tomorrow.

I’m a very early riser, so when I wake up in the morning, I like to be ready with a few things to work on, because that’s the time when I’m undisturbed and can actually work on creative stuff. So: last thirty minutes, figure out what happened and go to sleep. Then wake up, go over what I decided to work on, and get started. With that one or two hours of alone time, the rest of the day just follows nicely. That’s my routine.

Roland Cloutier

I know you’re an early riser – I’m an East Coaster, you’re a West Coaster, and when I start getting messages from you just before eight o’clock in the morning, I know you’re an early riser, man.

I want to frame this next question around how quickly things are changing for security, risk, and privacy executives. A lot of leaders are asking whether their organizations should go a hundred percent AI. Poonacha, is that even the right question to ask?

Poonacha Kongetira

I think it depends – there’s some merit to it. The way I’ve thought about breaking this down is that AI is intensely experiential. We all had that moment, whether you were building AI infrastructure like I was, or you weren’t, when the ChatGPT moment came out and you could type your question into a sparse little screen and get an amazing answer. It became clear to all of us that we were onto something different.

I think a lot of AI leaders got into that and said everybody has to do AI. My sense is what they meant is that every employee must have this experience of AI – they shouldn’t be blind to it. They all need to know the capability and tooling around AI, what the possibilities are.

But in terms of saying we should take all our workflows and make them AI-first, in enterprises that have been running for decades or longer – there, I’d go carefully. Whenever you change all of your workflows around a new engine like AI, which acts like a black box and exhibits variable behavior, you want to be careful, because whenever you take things offline, bringing them back online is the big problem. I learned this at NVIDIA – lights-on methodology. Keep the lights on.

So: go carefully, decide based on ROI outcomes whether you can take something to production. But definitely make sure your employees are educated and know how to use the new tooling. That’s how I’d split it up.

Roland Cloutier

It’s scary how fast this has moved. This isn’t just a new wave of technology and business concepts – we had the internet, we had cloud, we had mobile, we had decentralization – a lot of things happened, but nothing got adopted within the living, breathing entity that is our businesses this quickly, and changed hearts and minds this fast.

That ‘lights on’ point isn’t my next question, but I do want to touch on this idea of resiliency and who we’re relying on – that could be a whole other episode. I want to get to token economics, because until I started speaking with you, I understood the base concepts of tokens and cost analysis and model trends, but not at the level I needed to. I’d like to turn it over to you – I think you might have some whiteboard material. Take us through it.

Poonacha Kongetira

You can’t be an engineer without something like a whiteboard, right? Even the concept of what a token actually is, is elusive, so let me start there.

The user prompt is what you ask the model. So the question is: how do I explain token costs? The model comes back with, essentially, you’re going to pay for that – because any answer costs money. How does that translate? The model doesn’t read your sentence as-is. It goes into something called a tokenizer, which converts your sentence into chunks called tokens. The word ‘I’ might be a token on its own, but a phrase like ‘explain token costs’ might translate into four, five, or six tokens. That transformation produces an input token stream. Models understand meaning through numbers, and tokens are chunks of vector representations of numbers that carry meaning.

Once the model receives the input tokens, it turns the crank and produces a series of output tokens, which go back through the tokenizer and turn into a sentence. That’s how tokens work. The language of the LLM is tokens. The language of us is English, or whatever language we’re speaking. The tokenizer converts between the two. And because the model understands tokens, the language of economics here is also tokens – it’s easy to count, easy to apply.

The simple version of the cost formula: take the input token cost, multiply by the number of input tokens. Take the output token cost, multiply by the number of output tokens. Why are they priced differently? It’s quite a bit more expensive to generate output tokens.

Roland Cloutier

This was probably the simplest and most detailed explanation of the basic concept of tokens at the same time. Let’s keep rolling.

Poonacha Kongetira

So why are output tokens priced differently than input tokens, and why are large models expensive? Let’s take the input again: ‘how do I explain token costs.’ Ignoring the tokenizer, that goes into one end of the model. The large language model is composed of a number of identical stages, called transformer stages, stacked one on top of the other. The question percolates through all of these stages until you reach the other end, and by the time you get there, you have the first word of the model’s response – in this case, the word ‘your.’

The first pass doesn’t generate the full answer – it generates the first word. That word is then recirculated and appended to your original question, and the model runs again. That generates the second word, ‘gonna.’ That goes back through again. This recursive process is called an autoregressive transformer. In five passes, you generate a five-word response.

So the GPU seconds you spend are substantial – it cost the model provider more to serve that response, and you pay more as a result. That’s why output tokens are more expensive than input tokens: all of the input tokens you send get digested in one pass, while each output token adds to the workload. So if you want to save money, get the model to talk more briefly to you.

Here’s another one: why are smaller models cheaper? Keeping it simple – fewer transformer stages means a smaller model, more transformer stages means a bigger model. Run through a smaller model and you consume fewer GPU seconds, so it’s cheaper for the provider, and cheaper for you. Smaller models are great for small, simple tasks – knowing when to use what is the interesting piece.

Roland Cloutier

I’m curious – if I’m a security executive looking to understand levers, that’s obviously a clear one for deciding what technology to use in what part of my operations. Are there other levers we should be thinking about?

Poonacha Kongetira

Yeah, there’s one more. Why do agents get expensive with large context, and what does large context actually mean? First, it’s worth understanding what context means. Take the word ‘bank’ – it could mean the thing you put money into, or the bank of a river. Knowing which is only possible if you understand the sentence it’s used in, and therefore the context. Models are great at deriving meaning from surrounding context, which is why we want to give models as much context as possible – it’s a huge improvement over older methods like regular expressions, which are imperfect at discerning context.

Now imagine you’re asking this of an agent system rather than just an LLM. It’s hooked up with tools, and it has a system prompt – the instructions you’re giving the agent on how to respond. Say the system prompt says: attach massive documents, connect all possible tools, invoke many skills. In practice, hopefully one doesn’t do all of this – but let’s say you do. That entire piece of text gets passed along with the user’s tokens to be processed by the model, so your input tokens grow by that amount.

When you connect tools, each one is probably an MCP server, which comes with its own description of capabilities, safety precautions, and so on – a lot of text. Skills the model invokes are a lot of text. And then there’s your two-week-old chat history with your favorite assistant, which also tends to get passed into the model along with everything else. So input token costs can balloon with context.

With very large context windows – a hundred thousand, a million tokens – these things accumulate, and we as humans tend not to want to start a new session. So every time you ask a simple question, you might be processing a hundred thousand tokens, which isn’t a great idea.

Roland Cloutier

Is there a different cost across the different types of tokens – system prompt versus context tokens? How should we understand that?

Poonacha Kongetira

Absolutely. We’ve talked about output tokens being more expensive because you run a pass every time you generate a token. Input tokens – system prompt and the rest – translate into real input tokens, but the nice thing is that especially in a conversation or chat history, a lot of these are reusable. That’s a feature called prompt caching.

The computation involved – the giant matrix multiplication – tends to be very expensive, but the pre-computed result of persistent tokens can be reused. So if you have a chat history or repetitive questions using the same context, make sure you invoke prompt caching, and choose a vendor that enables it and passes the savings to you. Cached tokens are about a tenth of the cost of other tokens. And output tokens can be five to ten times the cost of an input token. That’s the hierarchy of token costs.

Roland Cloutier

That makes sense. So context is the variable – this episode’s whole premise is that context is the variable that changes everything. You’ve talked about price – how does context change quality, risk, and latency?

Poonacha Kongetira

Great question. Context, used properly and responsibly, allows a model to make good decisions. The quality of responses typically goes up with more context, because the model has more to reason on – more background on the task, a task-appropriate, context-appropriate response, understanding your style.

But with larger context windows – accumulated context from chat history and so on – the cost of computation goes up, and so does what you pay. Latency goes up too, because more context means more math and more memory to fetch.

Risk is the interesting one. You’d think risk would go down, but it depends on how you use it. If you connect everything up carelessly – documents that are contradictory, and so on – and tell the model to go figure it out, it may get confused by contradictory elements, or be forced to compress or drop information when there’s too much context, and give you lower-quality answers. So it comes back to the human: keep your desk clean, feed the model responsibly, don’t be sloppy.

Roland Cloutier

If there’s one thing you could leave executives with from this session, what would it be?

Poonacha Kongetira

Tokenomics can be fairly complicated – we’ve touched on the basics, which get you about 75% of the way there. There are more advanced things you can do: deciding whether to use smaller or bigger models, knowing when to use what, which is a big cost saving. There’s quantization, model architecture, and self-hosted models – beyond a certain amount of spend, it might make economic sense to self-host.

But the one thing everyone needs now is accurate, granular spend attribution. That’s missing in most infrastructure today – you get a big bill at the end of the month and don’t know who used it, which application, what was efficient or not. Spend attribution at a granular level is the missing piece, and everyone needs to figure out how to get it.

It also helps you identify sloppy usage or spend. You might run a POC, set an ROI target, spend the money, and it doesn’t deliver – but when you look at the spend attribution, you can find out whether the money was used efficiently or on things unrelated to the POC. Without that, you can draw the wrong conclusions. Spend attribution is the main thing.

Roland Cloutier

That’s a great point. I want to get into decision-making, because we’re often asked whether a workflow belongs in deterministic automation or in an independent AI model. How should leaders decide?

Poonacha Kongetira

Great question. First, it’s hazardous to assume you can move everything to AI – you still need compute, business continuity, and so on. Setting that aside, the key is understanding the problem surface you’re deciding on.

If your problem surface is easily specifiable – take a security example: if someone violated a business policy under these conditions, do this, this, and this – that’s an easy construct to put into a deterministic workflow, because you understand the problem surface well. Use traditional software: it’s cheaper, less variable, and you know what you’re going to get.

On the other hand, deciding what constitutes a business policy violation in the first place – that’s fuzzy. Who’s doing it, where, what was their intent, what does the policy actually say – figuring that out in real time is hard.

Roland Cloutier

Insider threat is a perfect example. We have insider threat teams who want to use this new capability, but understanding the policy, the intent, the documents being used, the source, the destination, the HR zone or org someone’s in – it’s all critical information that just isn’t deterministic.

Poonacha Kongetira

That fuzzy space is almost where you generate the signal that something happened. The way we’ve partitioned our architecture, that lives in a model. As soon as you have the signal, what you do with it lives in deterministic code that everybody understands easily. That’s an example of how you’d partition it.

Roland Cloutier

Give me an example of a security or engineering workflow where using AI is just expensive theater – not necessary.

Poonacha Kongetira

Let me preface this by saying I’m massively impressed by how we can use AI for almost anything – as a POC. The difference between that and production is scale. Can you take something from three people playing with it and telling you how great it is, to three hundred thousand people in a large organization, or your entire customer base, sending all their traffic through it? Because that’s what security is.

That POC might have been wrapped around a cloud-deployed, trillion-parameter model that provided beautiful responses – but can you actually scale that to a work surface of three hundred thousand people using agents? Most likely no. That’s the difference between theater and not.

Early on – maybe 2024 – I think that was okay, everybody was learning. But today, there’s no excuse for building a POC without also asking: how are you going to scale this to production? What’s it going to cost? Do I have the compute supply to scale to production, even if I pay for it? And do I have the governance models and supervision structures to govern this thing in production, so that as an organization I don’t incur liability, and I can still treat my customers the same way even though I’ve changed the engine under the hood?

Roland Cloutier

Great questions to ask. I’m running a POC right now that’s going to do something I haven’t been able to do in my entire career – looking at logs from half a million employees across eighty-two countries, different languages, different models. How is this really going to work?

Poonacha Kongetira

That’s one of the things that got beaten into our heads at Google – everything you build eventually has to be delivered to six billion humans on the planet. It doesn’t have to be six billion people, but it’s got to be more than six.

Roland Cloutier

I think of the same thing – no different from social media companies. At TikTok, we had to deliver defense, operations, trust and safety across billions of users and hundreds of languages, in the context of jurisdictional and cultural differences, and still hit ninety-eight percent efficacy and efficiency. It’s not easy.

That takes me to the next area, because you mentioned FTEs and employees. As a leader, my organization is going to change, and every CISO listening to this is thinking the same thing: I’m going from a hundred practitioners, two hundred practitioners, a thousand practitioners – that’s my N – and now I’ll have N times ten, times a thousand agentic things doing work for us, alongside us, with us. My managers will be managing humans and agents. So what changes economically when a workflow becomes agentic – when it’s calling models, tools, retrieval systems, and other agents repeatedly before it finishes? What are the hidden things we need to be aware of?

Poonacha Kongetira

A few things. First, on cost: agents are known to get into loops. Being able to have real-time understanding of what these things are costing on a transaction level is something we’ve built instrumentation and tooling to manage. As it’s happening, you count the dollars and cents, count the number of loops, and have a way to break out of it or shut it down. These things are basic, but the cost can’t just show up as the bill at the end of the month – because context accumulates and becomes expensive, and agents themselves can get into a reasoning loop.

A reasoning loop is essentially a model saying, to answer your question, I’ve got to think about this – and it generates output tokens you don’t see, which it uses to decide on a way forward, which can go around in loops, all invisible to you. So infrastructure to detect this happening in real time, and to be aware of it so you can act quickly, is critical from a cost standpoint. There’s a governance angle too – probably a different episode – but just knowing in real time what’s going on, whether these things are stuck, whether they’re generating useless tokens you’re going to have to pay for, matters.

Roland Cloutier

This transparency and visibility into cost is becoming more important to us than it’s ever been. I also want to come back to model selection – should we use small models, frontier models, fine-tuned models, or deterministic logic? Is there a base framework for making that decision?

Poonacha Kongetira

Some of it is acquired knowledge. There are model brokers that will discern your prompts and route them accordingly – at a bare minimum, that’s worth looking at, though accuracy depends on your use case.

A critical piece of development for every organization preparing for this is figuring out how to build evaluations – infrastructure exists, ours included, to help with that, but the ability to evaluate responses is critical. If you have a way to rank human customer service agents, similar metrics can be deployed to rank AI customer service agents.

In terms of technique: inference means you give the model your input, it runs, you pay the token cost, you get what you get. If you want to improve tooling for your specific use case, there are methods under post-training – restructuring your prompts, fine-tuning a model on a restricted set of tokens, giving it new instructions. If your prompts and instructions become voluminous, that gets more expensive; fine-tuning to do something more basic, better, is worth the one-time cost because it pays off afterward.

But really, the place we all need to go is building infrastructure for continual learning, so models improve with your usage patterns. Selecting providers who think about continuous learning is valuable. And building your own benchmarks is great too – that stays with your organization, and you can use it to evaluate vendors and build optionality.

Roland Cloutier

I’ve got about fifteen more questions on that, but let’s keep moving – we talked a bit about FinOps already. What does cost governance for AI look like if we’re not trying to slow things down?

Poonacha Kongetira

Thank God tokenmaxxing is done. Everybody found out it actually costs a lot of money – people initially maxed out on tokens, then realized it was a bad idea, which is great. Now we have per-employee budgets, which keep cost overruns under control – the FinOps team isn’t surprised anymore. But even that is still a reaction, probably a better one.

I’d go back to the earlier question: are employees educated about how to use AI? At the employee level, are they being efficacious about how they use it? That relates to FinOps – maybe more DevOps, technically – but if you’re not looking at cost, you’re missing a key piece. Efficacious usage also touches on whether people are using it for personal or company business – it’s all company expense tokens either way – and whether they’re using context efficiently.

The other piece touches on something we discussed earlier: how do you decide how to allocate token budget – real dollars – to a business unit, in pursuit of a business function? What do you bring on first? I’d start with a view toward ROI and the metrics you’ll use to judge return – most enterprises already have those metrics, because they’re running those business functions. What they might not have is spend attribution: without knowing where the spend is going, the denominator is fuzzy, and that can lead to the wrong conclusions. Start with clear success metrics, then get a tight understanding of where the spend is going, whether it lines up with those metrics, and whether you’re getting ROI. Those are the two things I’d advocate: one at the employee level, one at the business-unit level.

Roland Cloutier

That’s really important – it’s its own whole area of user education and accountability. Since you brought up the CFO: if I’m sitting down with the CFO and CIO, budgets are coming up, everyone’s building their plans – what should security executives bring to that conversation that only we can bring?

Poonacha Kongetira

There are a lot of players in deciding where AI budget goes, but it’s hard to say anyone has a better view of the overall shape of how AI is being used in an organization than someone with security operations responsibility – because they know, or should know, what’s running on every surface. Security is a whole-enterprise-surface function. A lot of other people know their own silo well and are focused on delivering their piece of it, but the overall view across the organization – the CISO tends to know the most about that, because their job is also to detect what’s going on everywhere.

If they can relate employee time usage and token usage to the surface of applications they have visibility into, that gives them a holistic picture that almost nobody else in the room has. That’s a genuinely valuable, normalizing thing to bring to the discussion.

Roland Cloutier

I like that – a lot of our tools can provide that kind of visibility, and at the end of the day it’s just data. It doesn’t lie, and it informs the whole discussion. I saved the hardest question for last: if a CISO wants to get smarter about AI economics next week, what are the first three numbers they should ask their team for?

Poonacha Kongetira

Beyond the basic education we covered – what token costs are and so on, which they should just go figure out themselves – I’d say: first, for their organization’s surface, just know what’s running, who’s using it, and what the usage pattern looks like, personal versus business. That’s a must-have first question, because it speaks to the value they bring in knowing what’s going on across the enterprise.

Second, from a financial point of view: what is this costing – whether that’s employee time or tokens? You can slice that a few different ways, but it’s all company expense.

And third: how can you slice and dice that spend so it’s not one opaque lump nobody knows what to do with, but something you can actually reason about and make decisions on as a leadership team. Those are the three things I’d advocate.

Roland Cloutier

Three excellent things, and three things we can act on this week – a lot of organizations are already building instrumentation for the coming fiscal year to do exactly this.

So there you have it, straight from Poonacha. The token dilemma isn’t really about whether AI is affordable – it’s about whether leaders can tell what AI is paying for. Tokens are becoming a unit of business operations. They measure how much context we send, how much reasoning we ask for, how much output we generate, and how many times an automated workflow calls a model before the work is done.

The leaders who win won’t simply tell their teams to use more or less AI – they’ll understand the shape of the work. Known, repeatable, rules-based work should often stay deterministic, as Poonacha said. Ambiguous, context-rich, judgment-heavy work is where AI earns its cost – not cost control that kills innovation. The shift is learning to govern AI as an economic system: model choice, context discipline, routing, caching, budgets, showback, and the data assets that help the business make sense of costs per completed task.

Poonacha, this is exactly the conversation security leaders need right now – enough technical grounding to understand the economics, and enough executive framing to make better business decisions. I can’t thank you enough for joining us, and I look forward to the other three podcasts we’ve already dreamed up on other topics. Thank you very much.

Poonacha Kongetira

Always a pleasure, Roland. Thank you so much for having me on.