AI agents can help reduce production failures in network change management at specific points in the existing workflow.
2
A network knowledge graph built from live network data gives agents the context they need for impact assessment, test planning, and execution.
3
A production workflow can connect ServiceNow, multiple specialized agents, GitHub change files, and a digital twin for network testing.
Summary
Ola Mabadeje describes a Cisco Outshift project for reducing failures in network change management. The system connects natural-language ITSM tickets with specialized agents, a network knowledge graph, and a digital twin that runs tests against proposed changes. The graph ingests data from controllers, devices, telemetry, configuration systems, and Splunk, then maps it into an OpenConfig-based schema. Agents use different graph layers depending on the task, such as reading raw configuration for drift checks or combining configuration, control-plane, and data-plane information for reachability tests. A demo follows a firewall-rule change from a ServiceNow ticket through impact assessment, test-plan creation, GitHub configuration retrieval, and test execution. Mabadeje also explains why the query agent was fine-tuned with schema information and example AQL queries. He is careful to say the system is still an MVP and that evaluation must connect agent and graph metrics to customer outcomes.
Change management failures are a customer problem where agents can help at specific workflow steps
The project began with a customer asking how AI could reduce failures in production during change management. Mabadeje says the team examined the workflow before deciding whether the problem needed AI or could be handled with rule-based automation. They identified specific points where agents could add value, including impact assessment, testing, and reasoning about possible network failures. The product process moved from customer questions to prototypes, testing, an MVP, and eventual production use if the product reached product-market fit.
The system connects an ITSM language interface, specialized agents, and a network digital twin
The proposed system has three parts. Network operations teams and other systems interact with it through natural language, including an ITSM tool such as ServiceNow. A multi-agent application assigns work to agents for impact assessment, testing, and reasoning about possible failures. The third part is a digital twin of the production network. Mabadeje describes that twin as a knowledge graph combined with tools that can execute network tests.
The ingestion pipeline turns varied network data into an agent-readable graph
Network environments contain devices from different vendors, including firewalls, switches, and routers. Controllers, device agents, configuration systems, and other sources expose data in formats such as YANG and JSON, through streaming telemetry or configuration files. The team built an ingestion service that performs ETL and converts this information into one OpenConfig-based schema. The product requirements included multimodel data, fast access to node information, one schema framework, future vector indexing for semantic search, easier integration, and support for multiple vendors.
Graph layers let agents retrieve the amount of network context needed for each test
The graph is organized into layers that expose different kinds of network information. A configuration-drift test can go directly to raw configuration files for comparison. A reachability test may need raw configuration together with control-plane and data-plane information. This structure lets an agent choose the relevant layer instead of traversing the entire graph for every request. The graph therefore supplies context in a form that matches the decision or tool call the agent needs to make.
An open agent framework is intended to make agents easier to connect across systems
Mabadeje presents a broader effort involving Outshift by Cisco, LangChain, Galileo, and other supporters. The proposed framework covers agent identity, schemas for skills and capabilities, an agent directory, composition at semantic and synthetic layers, and process observability. He points to agency.org, a GitHub repository, documentation, and sample applications. The group is also integrating protocols such as MCP and A2A because the goal is to avoid bespoke agent integrations that require rebuilding agents whenever another agent is added.
Fine-tuning the graph query agent reduced token use and response time
The application currently has five agents. An assistant agent plans and coordinates work, while other agents use ReAct reasoning loops. The query agent interacts directly with the knowledge graph. The team first tried using RAG for graph queries, but it did not work well enough. They fine-tuned the query agent with schema information and example queries. Before that change, AQL queries moved through all graph layers during reasoning loops, consuming many tokens and taking a long time. Fine-tuning produced a drastic reduction in token consumption and the time needed to return results.
The firewall-change demo carries a ticket through planning, approval context, and testing
The demo starts with an engineer submitting a ServiceNow ticket for a firewall-rule change needed to add a new server. Agents summarize the request, create an impact assessment, and attach it to the ticket. They then create a test plan with test cases and expected results, which is also attached for the approval board. The proposed firewall configuration comes from a GitHub pull request. The execution agent uses that change candidate and a recent network snapshot from the knowledge graph to run tests in the digital twin, then attaches a report showing passed and failed tests and recommendations for failed cases.
Evaluation must connect graph and agent measurements to the customer's use case
Mabadeje says evaluation is needed to understand whether the system delivers customer value. The team is measuring the agents, the knowledge graph, and the digital twin, while focusing on quantitative metrics. For the graph, they are looking at extrinsic metrics rather than only intrinsic measures because the results need to map back to the customer's use case. He describes the project as an MVP and says the team is still learning, while identifying the knowledge graph and an open framework for building agents as the two main building blocks for scaling the system.
"So the goal was for us to create this ingestion pipeline that can represent the network in such a way that agents can take the right actions in a meaningful way and predictive way."04:14
Who should watch
You work on network change management and need proposed changes to be assessed against current network state before production deployment.
Your team is connecting ITSM workflows to network automation and needs an architecture with specialized agents and graph-based context.
You are evaluating agentic network operations systems and want to see how a product team measures a still-developing MVP against customer outcomes.
