There’s a conversation I have with basically everyone now. It goes: “I have this problem. Can AI solve it?”
It’s like asking whether a brilliant new hire can solve a complicated business problem on their first day. Perhaps. They may be the smartest person you’ve ever brought in. But nobody has told them what your business considers urgent, what you’ve already tried, or what you want on your desk Monday morning. Talent is one component.
I can’t be smug about this, because in our exploration, my team and I both made this same mistake.
We pointed AI at the problem first
We wanted help with network health across our sites. We had plenty of telemetry: jitter, latency, packet loss, and bandwidth tests that each device runs at scattered times around 3 AM. So we did the obvious thing and pointed Copilot Studio at the raw data.
We found out quickly that it couldn’t make the correlations we were hoping for. The model’s ability wasn’t the problem. It had no way to know what we meant by “warning” or “critical.”
Is a 100 Mbps decline over 30 days a warning, or is it critical? Is packet loss spiking to 3% for a day a warning, or is it critical?
The telemetry doesn’t answer those questions, and neither does a model. It wasn’t a mind reader about what we wanted to take action on. No new hire would be either.
Those questions were ours to answer
That was the uncomfortable part. Every one of those questions is a business decision. What is worth acting on today? What can wait? What do we want to hear about on a Monday morning? Until we answered them, there was nothing for the AI to be right about.
AI can’t apply judgment that an organization hasn’t made explicit. Someone has to decide it, and then the data has to be shaped so a model can see it.
We had been treating the AI as the magic. The magic was the upfront work: deciding the goal, then engineering the data to serve it.
That was the revelation for me. Good AI engineering and good data engineering are essentially lockstep. In a system like ours, you can’t do the AI part well without the data part, because the AI can only work with what the data work hands it.
It changed the way I look at artificial intelligence, and how I think enterprises should approach it if they want to be efficient: goal first, data second, AI last.
The work that made it work
So we went back and did it properly:
- Pulled the raw data from our firewall vendor’s API.
- Wrote PowerShell scripts to collect bandwidth metrics.
- Imported our circuit contracts, so the AI knows what each site is supposed to be getting.
- Imported all of it cleanly into one place.
- Built SQL views that pivot the data exactly the way we wanted it, so the AI had the context it needed.
It was a massive amount of work, and none of it was AI work.
Today the agent hands us a report every Monday morning on our five most problematic sites. It might show a site getting 50 up and 50 down with clean latency, which implies a switching issue. It might tell us a site is in critical health because it has exceeded 25% packet loss. It’s the analysis we wanted, on a schedule, with no person doing it by hand.
A fair question is why we need AI at all once that work is done. A simple rule can tell you packet loss passed 25%. The harder calls need context and reasoning. We prompt the model as an infrastructure business analyst with core knowledge of what that role entails, and because we import the circuit contracts, it knows what each site is paying for. When a site on a 1 gig up, 1 gig down circuit tests at 100 down, it reads that as a likely switching issue, a layer 2 problem where something in the cable run or the data link layer is wrong. Reading the metrics against each other and against the contract is the analysis we wanted off a person’s plate, and it’s the job the model does well once the data is shaped for it.
The 5% people see
Demos actually make me fearful now. Building the AI layer was about 5% of the work. It’s also all that people see, so they gravitate to that 5%. I can’t blame anyone for doing that, because it’s the flashiest part.
But it quietly undermines all the hard data engineering it took to get to that point. People walk away assuming the prompt, or the AI, is the system. A lot of them still believe you can throw raw data at a model, with no agreed schema and no work on how the data is pivoted, and expect good results. We believed something close to that ourselves until we tried it.
Models will keep getting better at working through messy data on their own, and some of the shaping we did by hand may get easier. What no model can work out is what your business considers worth acting on. That part stays with you.
The temptation is to see AI as the flashy thing that will be the system that solves your problems. What it really does is make you analyze your business’s data, and work out how reasoning can be injected at a specific point in a workflow to assist.
An AI system is a pipeline with one unpredictable step in it. That is what ours turned out to be.
Define what you need first
What I see most often is people trying to force AI at a problem without defining what they need. There is a big difference between these two requests:
- Tool first: “Integrate HubSpot into Copilot.”
- Outcome first: “Our sales team doesn’t know which leads are worth the most. We want a daily analysis of leads on these three metrics that change every day, so the team starts each morning with its highest-value target.”
The first tells you what to connect. The second tells you what to build: which data you need, how often it refreshes, how it has to be shaped, and who acts on the result. Every component you implement is influenced by the outcome, and the data engineering most of all.
What I ask now
So when someone asks me whether AI can solve their problem, my answer is still perhaps, the same answer I’d give about the new hire. But I no longer treat it as a question about AI. I ask what solved looks like. What do you need to know, how often, what will you do when you know it, and how will you check that it’s right?
Those questions are harder than they sound. We couldn’t answer them for our own network until a model failed in front of us and forced the issue. That failed attempt wasn’t wasted, because it showed us exactly what we hadn’t defined. The waste would have been to keep adjusting the prompt and hoping.
Once you can answer them, most of the path is laid out. The goal tells you what data you need. The data work decides what the AI can be trusted to say. And the AI, when it finally arrives, does what it’s good at: turning well-shaped data into analysis nobody has to do by hand.
AI is not the system. It’s the last part you add, after you’ve decided what you need.
Leave a Reply