Nikhil Vyakaranam Technical Marketing Engineer at Cisco Systems   All Articles by Nikhil Vyakaranam

How to contribute your article to Netmanias.com !           List of Contributors

 

Dilip Krishna S Technical Marketing Engineer at Cisco Systems   All Articles by Dilip Krishna S

How to contribute your article to Netmanias.com !           List of Contributors

 

 

The purpose of interoperability testing is to prove that end-to-end functionality between (at least) two communicating systems is as required by the standard(s) on which those systems are based.

-ETSI

With many different kinds of network elements, a Service Provider has one of the most complicated network. No one vendor has expertise on all network elements. Most service providers opt and prefer a multi vendor contract and as a result service provider network ends up with  products from many different vendors. It is extremely important for all the various network elements to understand and communicate with each other. A small incompatibility, like change in the protocol version, could potentially bring down an entire operator network. Network interoperability is indispensable in order to achieve end-to-end connectivity. 

Despite its criticality, there are very few test tools dedicated to such tests. Some tools that have capability of interoperability testing lack essential features. As such, Interoperability testing is predominantly a manual process which costs the  vendors and service providers considerable amount of time and resources.

This white paper will present a new product - an Easy, Quick and Reliable Cloud based solution - that can automate and simplify the entire Interoperability testing process leading to significant savings in operating expenditures of the vendors and service providers. The paper focuses primarily on the mobility market, however, the product architecture can be very easily extended to many other domains and markets.

The Interoperability Market

With more than 900 operators across the globe, telecom is one of the highly competitive market. GSMA lists more than 350 associate members who contribute to the mobile ecosystem in more than one way. The service provider network encompasses products from many different vendors.

The number of IoT (Internet of Things) devices are expected to rise exponentially in the coming years. With around 600 million devices now, the number is expected to reach 5 billion in the next 4 to 5 years. There are already more than 450 IoT platforms and are set to rise higher as many more new protocols and technologies come into the IoT realm. Evolution to 5G, which has been designed to cater to the needs of the Internet of Things, will catapult the growth of IoT devices. With numerous IoT devices and platforms, there would be an increase in incompatibility among them and would trigger a substantial need for interoperability tests.

Many more interfaces and network functions will be added to the telecom network as 5G gains momentum. The 5G network is quite different from LTE and earlier releases. A lot of significant changes have been made to the network elements in order to decrease network latency and improve and simplify the overall network architecture. The advent of 5G will in all likelihood increase service providers requirement for a comprehensive interoperability tests with all 5G vendors.

Groups like the 5G-TF (5G Technical Forum) lead by Verizon and 5G-SIG (5G Special Interest Group) lead by Korea Telecom have attempted to develop essential 5G specifications much before 5G specification could be standardized. These early specifications will most likely be quite different from the 3GPP 5G specifications. The equipments deployed in the service provider network would possibly be a mixture of 3GPP 5G compliant and non-compliant entities. This would further encourage the service providers to have a thorough interoperability test before any product is introduced in their network.

The Interoperability Process

The Interoperability Test procedure is predominantly a manual process. The process involves at least two active vendors and a few supporting vendors. 

The IOT process is usually initiated by the vendor - called the Governing Vendor -  who is responsible for the software upgrade at the operator’s network. An important requirement of the upgrade is to make sure that the new software doesn’t break any existing workflow. This is generally proven by the governing vendor by executing Interoperability Tests with all the interfacing products. The governing vendor, after consultations with the service provider and participating vendors, comes up with a list of test cases to be executed , the testing schedule, the configurations and IP plan of the participating products, and the names of test engineers who would be executing the tests. All of the above information is captured in a document called the Joint Test Plan (JTP). For a product that interfaces with many other vendor products,  a separate JTP is created for each interface.  Creation of JTP may take from a few days (if new JTP is similar to the previous JTP) to a few weeks (if it is a completely new JTP). 

Interoperability Testing commences once the JTP is agreed upon by all vendors and the service provider. All vendors use the JTP to build their network and configure their respective products. The products interface with each other securely through VPN tunnels. It generally takes a week or two to build a new setup. Many vendors avoid creation and re-creation of new setups by having a dedicated IOT lab with permanent VPN connections to participating vendor’s lab. 

After establishing an end to end connectivity, the approved test cases captured in the JTP are executed. Depending on the complexity of the interface, number of test cases to be executed and the experience and agility of the engineers, the time taken to complete the execution of test cases can vary from a few days to a few months. The test execution status and the actions to be taken for the failed tests, if any, are captured in the Joint Test Report (JTR) document.

Fig1: Interoperability Process

Challenges faced by Vendors & Service Providers

The Interoperability test procedure in telecom domain is long and cumbersome. Being a manual process, the current interoperability procedure introduces many human errors and delays. 

Vendors:

For the vendors, interoperability tests post a significant burden on their Operating Expenditures. The tests require a dedicated setup and engineers. Additional simulators or tools may be required to run a few of the test scenarios. A vendor may also decide to use simulators/tools to reduce the dependencies on supporting vendors. Additional cost may be involved for the procurement of the simulators/tools. 

Engineers:

For the engineers, the main challenge lies in the execution of the test cases. Not all test cases are simple and straightforward to execute. An experienced engineer may know many tricks of the trade which can lead to faster completion of test cases, but a newcomer has to almost completely rely on notes and knowledge of the experienced colleague. A number of Interoperability test sessions have been delayed due to inexperienced engineers. 

Same set of test cases may have to executed for legacy interfaces or even for some new interfaces. Boredom can set in very quickly for an engineer who has been working on the same stuff for a long time, leading to a disgruntled employee. 

Service Provider:

For the Service Provider, the biggest concern is significant loss of time. An operator cannot upgrade a software unless all the Interoperability tests are executed with favourable results. Delays due to manual process are not uncommon and add significant burden to the operators Go-To-Market strategies. 

Due to unavailability of a standard set of tools among the vendors, the vendors may not be able to test many of cases requested by the service provider. 

Dissimilar tools across vendors also lead to the capture of traces and logs in many different formats making it difficult for the service provider to verify the execution of test cases. 

Some service providers develop a few network elements (like HSS, Credit based network elements etc.) in-house. Service provider will have to employ additional resources to carry out interoperability tests for their in-house products, leading to additional operating costs. 

What to look for in an Interoperability Test Tool ?

The Interoperability test tool has the responsibility of managing the entire interoperability process - starting from creation of JTP till the approval of JTR. A few of the important features an interoperability test tool should have are as follows

Version Management

The interoperability process starts from the creation of JTP by the governing vendor after extensive discussions with the participating vendors and the service provider. Multiple versions of JTP, incorporating various comments and changes as suggested by participants, are generated in the process. The same process is followed for the creation of JTR (Joint Test Report) after the tests are executed. The interoperability test tool should have the capability of version management, to create and store all version of the documents. 

VPN Connectivity

The vendor products typically reside securely in the vendor labs. Communication among the vendor products happens though VPN. An testing tool, therefore, should have the capability to connect to both vendor networks through VPN. To test with a tool without a VPN connectivity feature, all vendor products would have to be co-located. This would very easily lead to severe logistics and legal issues.

Tracer

The test tool should have the capability to trace and log the messages being exchanged by the various products. This would help the service providers to have a single format for all logs and traces.

Push Configurations:

Configurations of the network or the product may have to be changed before , after or during the execution of the tests. The test tool should have the capability to automatically change the configuration as and when required without manual intervention.

Simulators:

The Interoperability test tool should have the capability to simulate all interfaces that are not being currently tested, to provide a complete end to end connectivity. For e.g. to test S11 interface between Vendor A having MME product and Vendor B having SGW product, the test tool will have to simulate the surrounding eNodeB, HSS and PGW network elements.

Issues with Current tools

Many tools exist in the market that can test telecom products. Though they are good at testing individual products, they are not well equipped to do an interoperability test. Existing testing tools are primarily focused on the simulation and testing of the product and very little attention has gone into automating the entire end to end interoperability process. The tools available in the market today may have a few of the above mentioned features, but all features are extremely necessary to have a comprehensive Interoperability test tool.

Solution: Interoperability As A Service

Keeping the above benefits in mind, we propose Interoperability as a Service which is a cloud based service that automates and simplifies the entire Interoperability Process. The solution has the following features

Features

Dashboard: An intuitive web based user interface that allows the user to securely login to the service.

• It would present a list of interfaces and test cases associated to each interface to assist in the JTP creation. 
• Dashboard would show test execution status and various other analytics associated with the tests during the execution of tests. 
• Dashboard would provide simple user interface to make various product and VPN related configurations

VPN Service: A VPN service would be run as part of Interoperability service. Vendors can securely connect the product in their lab to the interoperability service via VPN. The interoperability service would then securely forward packets received from originating network element to the destination element. 

Version Control: All documents would be stored under a version management software like SVN. The JTP and JTR especially would undergo many rounds of revisions and updates and all versions would be easily accessible.

API Service: Interoperability service would provide API service to connect to a 3rd Party test case providers like the Network Vendors Interoperability Test Forum (NVIOT). This would enable 3rd party test case providers to add/update test cases which would be automatically reflected in the user’s dashboard.

Configuration Update: The service has the capability to automatically update configurations on vendor products for corresponding test cases using advanced protocols like netconf.

Simulators: Interoperability service will have the capability to simulate all supporting network elements that are not directly participating in the Interoperability tests. 

Tracer: All messages and relevant traces will be logged in a standard format. 

Report Generation: The service can generate relevant reports at all stages of the Interoperability process. Starting from generation of JTP to daily reports on execution status to a JTR on completion of all tests would be done automatically by the service.

Notifications: At every stage all stakeholders would be notified of the relevant activities. 

Analytics: The interoperability service can provide useful statistics to the vendors and operators on the test case executions. It can help them to assess and improve the overall interoperability process.

Benefits

Easy selection of test cases: The intuitive and interactive user interface will enable the service providers and vendors to quickly select all relevant test cases from the list. Any updates to the test case selection would be version controlled. On approval by all stakeholders, a JTP report would be sent automatically to all. The cumbersome 2 week process would be reduced exponentially.

Quick and Hassle free execution of tests: By eliminating the need of manual intervention, the interoperability tests can run continuously which ultimately would lead to much faster execution and completion of test cases. This provides a major benefit to the service providers as the test execution time would be dramatically reduced from weeks and months to just a few days if not few hours. A complete test report with all relevant logs and traces would be automatically sent to all participating vendors and service provider.

Pay as you test: Interoperability As A Test is a complete automated test service. Vendors and Service Providers no longer need dedicated human resources to perform these tests. They can simply pay for the service as and when required.

Increase in test execution quality: Automating the entire interoperability process conveniently removes all human errors leading to a more superior validation and  verification of tests and better quality of reports.

Consistent logging and tracing formats: Interoperability service will trace in the same format for all tests in all interfaces, making it easy for the service providers to find a needle in the haystack. 

Automatic updation of tests: Interoperability service provides an API service for 3rd party test case providers to add or update test cases. All the changes will be instantly visible in the dashboard and will be available for selection for the JTP. As the standard evolves the test case list in the service would also evolve with the same pace.  

Use Cases in non-telecom domains

Though the focus of this white paper has been on telecom domain, Interoperability As A service can very easily be used in many other non-telecom domains.

Internet of Things: With 5G around the corner, the number of devices supporting IoT is expected to increase exponentially. Increase in devices would eventually lead to increase in IoT platforms as well. Interoperability service can be used to validate the IoT devices with the platforms. 

IoT Platform Certification: With increase in the number of IoT devices and platforms it will become difficult to verify which device would work on which platform. Interoperability service can provide a certification to the device on all platforms in which it is interoperable.

Collaboration tools: Collaboration tools like webex etc. have to interwork with many different kind of accessories. Interoperability service can be used to test and verify interoperability for the collaboration tools.

Other domains: The service can potentially be used to verify interoperability in many other domains like the healthcare, finance and aeronautics. 

Conclusion

Interoperability As A Service is the need of the market. Significant amount of time, resources and money is spent to carry out interoperability related activities. It is vital that such activities are automated to improve the quality of tests and generation of reports. Many of these tests are repetitive and mundane.By eliminating the need for human intervention, the time and resource of the skilled engineers can be put to better use. 5G and Internet Of Things will bring about many more interoperability test requirements and a service like this can potentially save millions of dollars in operating costs for the vendors and service providers. With many wide use cases across various telecom and non-telecom domains, interoperability as a service can play a critical role in providing better and faster growth across various industries.

For more articles by Nikhil Vyakaranam and Dilip Krishna S on Technically Speaking, please see: https://cloudifynetwork.com