



Network Management Research Group                                 Y. Cui
Internet-Draft                                                    Y. Wei
Intended status: Informational                                    K. Chi
Expires: 5 January 2026                                           X. Xie
                                                     Tsinghua University
                                                             4 July 2025


Framework and Automation Levels for AI-Assisted Network Protocol Testing
                      draft-cui-nmrg-auto-test-00

Abstract

   This document presents an AI-assisted framework for automating the
   testing of network protocol implementations.  The proposed framework
   encompasses essential components such as protocol comprehension, test
   case generation, automated script and configuration synthesis, and
   iterative refinement through feedback mechanisms.  In addition, the
   document defines a multi-level model of test automation maturity,
   ranging from fully manual procedures (Level 0) to fully autonomous
   and adaptive systems (Level 5), providing a structured approach to
   evaluating and advancing automation capabilities.  Leveraging recent
   advancements in artificial intelligence, particularly large language
   models (LLMs), the framework illustrates how AI technologies can be
   applied to enhance the efficiency, scalability, and consistency of
   protocol testing.  This document serves both as a reference
   architecture and as a roadmap to guide the evolution of protocol
   testing practices in light of emerging AI capabilities.

About This Document

   This note is to be removed before publishing as an RFC.

   Status information for this document may be found at
   https://datatracker.ietf.org/doc/draft-cui-nmrg-auto-test/.

   Discussion of this document takes place on the NMRG Research Group
   mailing list (mailto:nmrg@irtf.org), which is archived at
   https://datatracker.ietf.org/rg/nmrg/.  Subscribe at
   https://www.ietf.org/mailman/listinfo/nmrg/.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.






Cui, et al.              Expires 5 January 2026                 [Page 1]

Internet-Draft                   FALANPT                       July 2025


   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 5 January 2026.

Copyright Notice

   Copyright (c) 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Definition and Acronyms . . . . . . . . . . . . . . . . . . .   3
   3.  Network Protocol Testing Scenarios  . . . . . . . . . . . . .   4
   4.  Key Elements of Network Protocol Testing  . . . . . . . . . .   4
   5.  Automated Network Protocol Test Framework . . . . . . . . . .   5
     5.1.  Protocol Understanding  . . . . . . . . . . . . . . . . .   6
     5.2.  Test Case Generation  . . . . . . . . . . . . . . . . . .   6
     5.3.  Test Script and DUT Configuration Generation  . . . . . .   7
     5.4.  Test Case Execution . . . . . . . . . . . . . . . . . . .   7
     5.5.  Report Analysis, Feedback and Refinement  . . . . . . . .   7
   6.  Automation Maturity Levels in Network Protocol Testing  . . .   8
     6.1.  L0: Manual Testing  . . . . . . . . . . . . . . . . . . .  10
     6.2.  L1: Tool-Assisted Testing . . . . . . . . . . . . . . . .  10
     6.3.  L2: Partial Automation  . . . . . . . . . . . . . . . . .  10
     6.4.  L3: Conditional Automation  . . . . . . . . . . . . . . .  11
     6.5.  L4: High Automation . . . . . . . . . . . . . . . . . . .  11
     6.6.  L5: Full Automation . . . . . . . . . . . . . . . . . . .  11
   7.  An Example of LLM-based Automated Network Protocol Test
           Framework (From L2 to L3) . . . . . . . . . . . . . . . .  12
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .  13
   9.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  13
   Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . .  13
   Contributors  . . . . . . . . . . . . . . . . . . . . . . . . . .  14



Cui, et al.              Expires 5 January 2026                 [Page 2]

Internet-Draft                   FALANPT                       July 2025


   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  14

1.  Introduction

   As protocol specifications evolve rapidly, traditional testing
   methods that rely heavily on manual effort or static models struggle
   to keep pace.  Testing involves validating that a device's behavior
   complies with the protocol's defined semantics, often documented in
   RFCs.  In recent years, emerging application domains such as the
   industrial internet, low-altitude economy and satellite internet have
   further accelerated the proliferation of proprietary or rapidly
   changing protocols, making comprehensive and timely testing even more
   challenging.

   This document proposes an automated network protocol testing
   framework that reduces manual effort, enhances test quality, and
   adapts to new specifications efficiently.  The framework consists of
   four key modules: protocol understanding, test case generation, test
   script conversion, and feedback-based refinement.  It emphasizes
   modular design, reuse of existing knowledge, and AI-assisted
   processes to facilitate accurate and scalable testing.

   In addition to the proposed framework, this document also defines a
   six-level classification system (Levels 0 to 5) to characterize the
   evolution of automation maturity in network protocol testing.  These
   levels serve as a technology roadmap, helping researchers evaluate
   the current state of their systems and set future goals.  Each level
   captures increasing capabilities in protocol understanding,
   orchestration, analysis, and human independence.

2.  Definition and Acronyms

   DUT: Device Under Test

   Tester: A network device for protocol conformance and performance
   testing.  It can generate specific network traffic or emulate
   particular network devices to facilitate the execution of test cases.

   LLM: Large Language Model

   FSM: Finite State Machine

   API: Application Programming Interface

   CLI: Command Line Interface

   Test Case: A specification of conditions and inputs to evaluate a
   protocol behavior.



Cui, et al.              Expires 5 January 2026                 [Page 3]

Internet-Draft                   FALANPT                       July 2025


   Test Script: An executable program or sequence that carries out a
   test case on a device.

3.  Network Protocol Testing Scenarios

   Network protocol testing is required in many scenarios.  This
   document outlines two common phases where protocol testing plays a
   critical role:

   1.  Device Development Phase: During the development of network
       equipment, vendors must ensure that their devices conform to
       protocol specifications.  This requires the construction of a
       large number of test cases.  Testing during this phase may
       involve both protocol testers and the DUT, or it may be performed
       solely through interconnection among DUTs.

   2.  Procurement Evaluation Phase: In the context of equipment
       acquisition by network operators or enterprises, candidate
       equipment suppliers need to demonstrate compliance with specified
       requirements.  In this phase, third-party organizations typically
       perform the testing to ensure neutrality.  This type of testing
       is usually conducted as black-box testing, requiring the use of
       protocol testers interconnected with the DUT.  The test cases are
       executed while observing whether the DUT behaves in accordance
       with expected protocol specifications.

4.  Key Elements of Network Protocol Testing

   Network protocol testing is a complex and comprehensive process that
   typically involves multiple parties and various necessary components.
   The following entities are generally involved in protocol testing:

   1.  DUT: The DUT can be a physical network device (such as switches,
       routers, firewalls, etc.) or a virtual network device (such as
       FRRouting (FRR) routers and others).

   2.  Protocol Tester: A protocol tester is a specialized network
       device that usually implements a standard and comprehensive
       protocol stack.  It can generate test traffic, collect and
       analyze incoming traffic, and produce test results.  Protocol
       testers can typically be controlled via scripts, allowing
       automated interaction with the DUT to carry out protocol tests.









Cui, et al.              Expires 5 January 2026                 [Page 4]

Internet-Draft                   FALANPT                       July 2025


   3.  Test Cases: Protocol test cases may cover various categories,
       including protocol conformance tests, functional tests, and
       performance tests, etc.  Each test case typically includes
       essential elements such as test topology, step-by-step
       procedures, and expected results.  A well-defined test case also
       includes detailed configuration parameters.

   4.  DUT Configuration: Before executing a test case, the DUT must be
       initialized with specific configurations according to the test
       case requirements (setup).  Throughout the test, the DUT
       configuration may undergo multiple modifications as dictated by
       the test scenario.  Upon test completion, appropriate
       configurations are usually applied to restore the DUT to its
       initial state (teardown).

   5.  Tester Configuration and Execution Scripts: In test scenarios
       involving protocol testers, the tester often plays the active
       role by generating test traffic and orchestrating the test
       process.  This requires the preparation of both tester-specific
       configurations and execution scripts.  Tester scripts are
       typically designed in coordination with the DUT configurations to
       ensure proper interaction during the test.

5.  Automated Network Protocol Test Framework

   A typical network protocol test automation framework is as follows.

     +---------+                                        +----------+
     |   RFC   |   +----------+   +-----------------+   | Test Env.|
     |Documents|-->|          |-->|  Tester Script  |-->| +------+ |
     +---------+   |Test Case |   |   Generation    |   | |Tester| |
                   |          |   | --------------- |   | +-^----+ |
     +---------+   |Generation|   |DUT Configuration|   |   |  |   |
     |  Human  |-->|          |-->|   Generation    |-->| +----v-+ |
     | Intent  |   +----------+   +-----------------+   | | DUT  | |
     +---------+         ^                 ^            | +------+ |
                         |                 |            +----------+
                   +-----------------------------+           |  Test
                   |   Feedback and Refinement   |<----------+ Report
                   +-----------------------------+











Cui, et al.              Expires 5 January 2026                 [Page 5]

Internet-Draft                   FALANPT                       July 2025


5.1.  Protocol Understanding

   Protocol understanding forms the foundation for automated test case
   generation.  Since protocol specifications are typically written in
   natural language, it is necessary to model the core functionalities
   of the protocol and extract a machine-readable representation.  This
   process involves identifying key behavioral semantics and operational
   logic from the specification text.

   In addition to high-level functional modeling, structured data
   extraction of protocol details, such as packet field definitions,
   state machines, and message sequences, is also an essential component
   of protocol understanding.  These structured representations serve as
   a blueprint for downstream tasks, enabling accurate and comprehensive
   test case synthesis based on the intended protocol behavior.

5.2.  Test Case Generation

   Once a machine-readable protocol specification is available, the next
   step is to identify test points based on human intent and extend them
   into concrete test cases.  Test points are typically derived from
   constraints described in the protocol specification, such as the
   correctness of packet processing logic or the validity of protocol
   state transitions.  Each test case elaborates on a specific test
   point and includes detailed test procedures and expected outcomes.
   It may also include a representative set of test parameters (e.g.,
   frame lengths) to ensure coverage of edge conditions.  Conformance
   test cases are generally categorized into positive and negative
   types.  Positive test cases verify that the protocol implementation
   correctly handles valid inputs, while negative test cases examine how
   the system responds to malformed or unexpected inputs.

   The quality of generated test cases is typically evaluated along two
   primary dimensions: correctness and coverage.  Correctness assesses
   whether a test case accurately reflects the intended semantics of the
   protocol.  Coverage evaluates whether the test suite exercises all
   protocol definitions and constraints, including performance-related
   behaviors and robustness requirements.  However, as test cases are
   often represented using a mix of natural language, topology diagrams,
   and configuration snippets, their inherent ambiguity makes systematic
   quality evaluation difficult.  Effective metrics for test case
   quality assessment are still lacking, which remains an open research
   challenge.








Cui, et al.              Expires 5 January 2026                 [Page 6]

Internet-Draft                   FALANPT                       July 2025


5.3.  Test Script and DUT Configuration Generation

   Test cases are often translated into executable scripts using
   available API documentation and runtime environments.  This process
   requires mapping natural language described test steps to specific
   function calls and configurations of test equipment and DUTs.

   Since tester scripts and DUT configuration files are typically used
   together, they must be generated in a coordinated manner rather than
   in isolation.  The generated configurations must ensure mutual
   interoperability within the test topology and align with the step-by-
   step actions defined in the test case.  This includes setting
   compatible protocol parameters, interface bindings, and execution
   triggers to facilitate correct protocol interactions and achieve the
   intended test objectives.

   Before deploying the test scripts and corresponding configurations,
   it is essential to validate both their syntactic and semantic
   correctness.  Although the protocol testing environment is isolated
   from production networks and thus inherently more tolerant to
   failure, invalid scripts or misconfigured devices can still render
   test executions ineffective or misleading.  Therefore, a verification
   step is necessary to ensure that the generated artifacts conform to
   the expected syntax of the execution environment and accurately
   implement the intended test logic as defined by the test case
   specification.

5.4.  Test Case Execution

   The execution of test cases involves the automated deployment of
   configurations to the DUT as well as the automated execution of test
   scripts on the tester.  This process is typically carried out in
   batches and requires a test case management system to coordinate the
   workflow.  Additionally, intermediate configuration updates during
   the execution phase may be necessary and should be handled
   accordingly.

5.5.  Report Analysis, Feedback and Refinement

   Test reports represent the most critical output of a network protocol
   testing workflow.  They typically indicate whether each test case has
   passed or failed and, in the event of failure, include detailed error
   information specifying which expected behaviors were not satisfied.
   These reports serve as an essential reference for device improvement,
   standard compliance assessment, or procurement decision-making.






Cui, et al.              Expires 5 January 2026                 [Page 7]

Internet-Draft                   FALANPT                       July 2025


   However, due to the potential inaccuracies in test case descriptions,
   generated scripts or device configurations, a test failure does not
   always indicate a protocol implementation defect.  Therefore, failed
   test cases require further inspection using execution logs,
   diagnostic outputs, and relevant runtime context.  This motivates the
   integration of a feedback and refinement mechanism into the
   framework.  The feedback loop analyzes runtime behaviors to detect
   discrepancies that are difficult to identify through static
   inspection alone.  This iterative refinement process is necessary to
   improve the reliability of the automated testing system.

6.  Automation Maturity Levels in Network Protocol Testing

   To describe the varying degrees of automation adopted in protocol
   testing practices, we define a six-level maturity model.  These
   levels reflect technical progress from fully manual testing to self-
   optimizing, autonomous systems.  The classification is intended as a
   reference model, not as a fixed pipeline structure.

































Cui, et al.              Expires 5 January 2026                 [Page 8]

Internet-Draft                   FALANPT                       July 2025


   +=====+================+==================+===========+============+
   |Level| RFC            | Test Asset       |Result     |Human       |
   |     | Interpretation | Generation &     |Analysis & |Involvement |
   |     |                | Execution        |Feedback   |            |
   +=====+================+==================+===========+============+
   |L0   | Manual reading | Fully manual     |Manual     |Full-time   |
   |     |                | scripting and    |observation|intervention|
   |     |                | CLI-based        |and logging|            |
   |     |                | execution        |           |            |
   +-----+----------------+------------------+-----------+------------+
   |L1   | Human-guided   | Script templates |Manual     |High (per   |
   |     | parsing tools  | with tool-       |review with|test run)   |
   |     |                | assisted         |basic tools|            |
   |     |                | execution        |           |            |
   +-----+----------------+------------------+-----------+------------+
   |L2   | Template-based | Basic autogen of |Rule-based |Moderate    |
   |     | extraction     | config & scripts |validation |(Manual     |
   |     |                | for standard     |with human |correction  |
   |     |                | cases            |triage     |and tuning) |
   +-----+----------------+------------------+-----------+------------+
   |L3   | Rule-based     | Parameterized    |ML-assisted|Supervisory |
   |     | semantic       | generation and   |anomaly    |confirmation|
   |     | parsing        | batch            |detection  |            |
   |     |                | orchestration    |           |            |
   +-----+----------------+------------------+-----------+------------+
   |L4   | Structured     | Objective-driven |Correlated |Minimal     |
   |     | model          | synthesis with   |failure    |(strategic  |
   |     | interpretation | end-to-end       |analysis   |input)      |
   |     |                | automation       |and report |            |
   |     |                |                  |generation |            |
   +-----+----------------+------------------+-----------+------------+
   |L5   | Adaptive       | Self-adaptive    |Predictive |None        |
   |     | protocol       | generation and   |diagnostics|(optional   |
   |     | modeling       | self-optimizing  |and        |audit)      |
   |     |                | execution        |remediation|            |
   |     |                |                  |proposals  |            |
   +-----+----------------+------------------+-----------+------------+

     Table 1: Automation Maturity Matrix for Network Protocol Testing

   As shown in Table 1, the automation progression can be characterized
   along four dimensions: specification interpretation, test
   orchestration, result analysis, and human oversight.  Each level
   reflects an increasing degree of system autonomy and decreasing human
   involvement.






Cui, et al.              Expires 5 January 2026                 [Page 9]

Internet-Draft                   FALANPT                       July 2025


6.1.  L0: Manual Testing

   Description: All testing tasks are performed manually by test
   engineers.

   Key Characteristics:

   *  Protocol understanding, test case design, topology setup,
      scripting, execution, and result analysis all rely on manual work.

   *  Tools are only used for basic assistance (e.g., packet capture via
      Wireshark).

6.2.  L1: Tool-Assisted Testing

   Description: Tools are used to assist in some testing steps, but the
   core logic is still human-driven.

   Key Characteristics:

   *  Automation includes test script execution and automated result
      comparison.

   *  Manual effort is still required for test case design, topology
      setup, and exception analysis.

6.3.  L2: Partial Automation

   Description: Basic test case generation and execution are automated,
   but critical decisions still require human input.

   Key Characteristics:

   *  Automation includes:

      -  A framework that generates basic test cases (e.g., covering
         mandatory fields and FSMs defined in RFCs) and corresponding
         tester scripts and DUT configurations.

      -  Topology generation for a single test case.

   *  Manual effort includes:

      -  Designing complex or edge case scenarios.

      -  Root cause analysis when tests fail.





Cui, et al.              Expires 5 January 2026                [Page 10]

Internet-Draft                   FALANPT                       July 2025


6.4.  L3: Conditional Automation

   Description: The system can autonomously complete the test loop, but
   relies on human-defined rules and constraints.

   Key Characteristics:

   *  Automation includes:

      -  Complex test cases generation based on semantic understanding
         of RFCs (e.g., core function modeling from RFCs).

      -  Minimal common topology synthesis for a set of test cases.

      -  Automated result analysis with anomaly detection.

   *  Manual effort includes:

      -  Reviewing the test plan and confirming whether flagged
         anomalies represent real protocol violations.

6.5.  L4: High Automation

   Description: Full automation of the testing pipeline, with minimal
   human involvement limited to high-level adjustments.

   Key Characteristics:

   *  Automation includes:

      -  End-to-end automation from RFC parsing to test report
         generation.

      -  Automated result analysis with root cause analysis.

      -  Automated recovery from environment issues.

   *  Manual effort includes:

      -  Defining high-level test objectives, with the system
         decomposing tasks accordingly.

6.6.  L5: Full Automation

   Description: Adaptive testing, where the system independently
   determines testing strategies and continuously optimizes coverage.

   Key Characteristics:



Cui, et al.              Expires 5 January 2026                [Page 11]

Internet-Draft                   FALANPT                       July 2025


   *  Automation includes:

      -  Learning protocol implementation specifics (e.g., proprietary
         extensions) and generating targeted test cases.

      -  Leveraging historical data to predict potential defects.

      -  Iterative self-optimization to improve efficiency.

   *  Manual effort: None.  The system autonomously outputs a final
      compliance report along with remediation suggestions.

7.  An Example of LLM-based Automated Network Protocol Test Framework
    (From L2 to L3)

   The emergence of LLMs has significantly advanced the degree of
   automation achievable in network protocol testing.  Within the
   proposed framework, LLMs serve as central agents in multiple stages
   of the pipeline, enabling a transition from Level 2 (Partial
   Automation) to Level 3 (Conditional Automation).  By leveraging LLMs
   with domain-specific prompting strategies, the system is able to
   extract, synthesize, and refine testing artifacts with minimal human
   intervention.

   At the protocol understanding stage, LLMs can ingest RFC documents
   and identify structured components such as protocol field
   definitions, message formats, and finite state machines.  These
   outputs, often difficult to parse using traditional rule-based
   methods due to inconsistencies in document structure, are extracted
   by prompting the LLM with carefully designed templates.  The
   resulting structured data serves as the semantic backbone for
   subsequent test case generation.

   Based on the extracted protocol semantics, LLMs are also capable of
   generating test scenarios that aim to exercise key protocol
   behaviors.  These scenarios are proposed in natural language and then
   instantiated into formal test cases through alignment with predefined
   schemas.  The use of LLMs allows for dynamic adaptation, enabling the
   test generation process to generalize across different protocol
   families while maintaining context awareness.  Although the coverage
   achieved is not exhaustive, the LLM is often effective at identifying
   common corner cases and expected failure modes, especially when
   guided by selected examples or constraint-based instructions.

   For the script generation phase, LLMs assist in mapping abstract test
   cases to executable configurations/scripts for both the DUT/tester.
   Given appropriate configuration/API documentation and environment
   context, the model synthesizes CLI commands, configuration snippets,



Cui, et al.              Expires 5 January 2026                [Page 12]

Internet-Draft                   FALANPT                       July 2025


   and tester control scripts (e.g., Python-based test harnesses or TCL
   scripts for commercial testbeds).  This process is enhanced by
   retrieval-augmented generation, where relevant code examples or past
   case patterns are integrated into the model's prompt to increase the
   reliability and correctness of the output.

   Finally, during the feedback and refinement loop, LLMs analyze
   execution logs and output discrepancies to identify failure points.
   While current models may not fully replace human diagnosis, they can
   summarize error types, hypothesize root causes, and suggest concrete
   revisions to test cases or script logic.  These suggestions are then
   validated by experts before being applied, ensuring both efficiency
   and correctness.

   Despite these capabilities, it is important to note that LLMs are
   fundamentally probabilistic and lack strong guarantees of determinism
   or correctness.  Therefore, while the LLM-enabled system can execute
   a full test pipeline with reduced human effort, human oversight
   remains essential, particularly for verifying generated artifacts and
   handling ambiguous or novel scenarios.  Nonetheless, the integration
   of LLMs marks a clear advance beyond traditional script-driven
   automation tools, and provides a practical path for elevating
   protocol testing practices toward Level 3 maturity.

8.  Security Considerations

   1.  Execution of Unverified Generated Code: Automatically generated
       test scripts or configurations (e.g., CLI commands, tester
       control scripts) may include incorrect or harmful instructions
       that misconfigure devices or disrupt test environments.
       Mitigation: All generated artifacts should undergo validation,
       including syntax checking, semantic verification against protocol
       constraints, and dry-run execution in sandboxed environments.

   2.  AI-Assisted Component Risks: LLMs may produce incorrect or
       insecure outputs due to their probabilistic nature or prompt
       manipulation.  Mitigation: Apply input sanitization, prompt
       hardening, and human-in-the-loop validation for critical
       operations.

9.  IANA Considerations

   This document has no IANA actions.

Acknowledgments

   This work is supported by the National Key R&D Program of China.




Cui, et al.              Expires 5 January 2026                [Page 13]

Internet-Draft                   FALANPT                       July 2025


Contributors

   Zhen Li
   Beijing Xinertel Technology Co., Ltd.
   Email: lizhen_fz@xinertel.com

   Zhanyou Li
   Beijing Xinertel Technology Co., Ltd.
   Email: lizy@xinertel.com

Authors' Addresses

   Yong Cui
   Tsinghua University
   Email: cuiyong@tsinghua.edu.cn


   Yunze Wei
   Tsinghua University
   Email: wyz23@mails.tsinghua.edu.cn


   Kaiwen Chi
   Tsinghua University
   Email: ckw24@mails.tsinghua.edu.cn


   Xiaohui Xie
   Tsinghua University
   Email: xiexiaohui@tsinghua.edu.cn





















Cui, et al.              Expires 5 January 2026                [Page 14]
