IP Office Technical Tip (Region: Global)

195: Using Packet Capture Software to verify IP Network VoIP QoS Operation

19th October 2007 - Full PDF Text Version

Converged networks can experience changes over time from the initial deployment. New users, servers, applications and topology changes are common. As the network changes it is possible that Voice over IP (VoIP) quality may become affected. In the time preceding a thorough followup traffic analysis and network assessment the below procedures can be employed to provide a preliminary investigation of the network’s impact on VoIP traffic and conversation quality. The techniques can be utilized with many commercial and freeware IP packet capture software. Those that offer additional VoIP monitoring and analysis abilities are better-suited to the task. This demonstration utilizes packet capture software called "Wireshark".

NOTE 1: The Avaya IP Office can be inter-connected via various network equipment and provider services. Be advised that Avaya requires mandatory network assessments whenever using any networking technologies to inter-connect IP Office hardware and/or software. This is to ensure the network’s readiness for delivering customer expected quality VoIP. These procedures are not a substitute for proper network planning or the requirement for network assessments before deploying converged networks. These procedures are not a substitute for regular network monitoring and analysis or follow-up network assessments for existing networks.

NOTE 2: Avaya provides the following as information only and in no way implies or otherwise assumes responsibility for customer network troubleshooting or issue isolation. The customer’s network, its maintenance and proper operation is the sole responsibility of the customer and the organization charged with its maintenance and upkeep.

Network Requirements

The following is an outline of Avaya’s network recommendations to ensure a network is capable of supporting customer expected VoIP quality. Full details with a break down of the meanings of each parameter are in the document "Avaya IP Voice Quality Network Requirements" available at http://support.avaya.com.

Network delay: Network Delay is the length of time it takes a packet to traverse the network in one direction.

• 80ms (milliseconds) delay or less can (but may not) yield TOLL quality.

• 80ms to 180ms delays can give business communication quality. This is better than cell-phone quality.

• Delays exceeding 180ms may be acceptable depending on customer expectations, analog trunks used, Codec type, etc.

• If greater, echo and or delay may be experienced.

Network jitter: Jitter is a measure of variance in the time it takes for communications to traverse from the sender to the receiver. Jitter is a measure of the variability of delay.

• Toll quality jitter is less than 20ms or less than one/half the packet payload. If greater, echo may be experienced.

Network packet loss: Network packet loss occurs when packets are sent, but not received at the final destination due to some network problem.

• 1% or less can yield toll quality depending on other factors.

• 3% or less Business Communications quality. Slightly better than cellphone quality.

• More than 3% may be acceptable for voice but may interfere with signaling.

• If greater, the customer may experience missed words, cutoff conversation, and/or call setup delay.

The recommendations above will be used as the guideline later in this document when comparing and measuring the VoIP packet traces.

IP Office Configuration Checks

Before beginning the test calls and packet captures, review the IP Office for correct configuration and operation related to network connectivity.

1. Check that the VoIP Line definitions within each IP Office are correctly configured

2. Check that the IP Routes in each IP Office are correctly configured

3. Check the IP Office System Gatekeeper tab for correct DiffServ Control Point (DSCP) settings

Review the IP Office VoIP Line Programming

Review the VoIP Line configurations, Incoming / Outgoing Group ID, number of channels, are set correctly.
 

Review the IP Office VoIP Line Programming (continued)

Review the IP Office IP Route programming

Ensure the IP Office has the correct default route configured to the remote site or individual IP routes to multiple remote IP Office locations. Check with the network administrator that the configured routes are directed to the local LAN’s default router IP address, in the example below - 192.168.42.254.

Review the IP Office DSCP settings

The below screen shot displays the IP Office default DSCP settings. Users can adjust these settings but then assume responsibility the network equipment is configured to recognize and prioritize any changes to these settings.

Preliminary Network Checks

1. If VLAN’s are deployed, check the network switch VLAN settings and trunk interface configuration are set per the vendor’s recommendations.

2. Check the QoS settings of the router configuration for prioritizing IP Office VoIP packets (Hex = b8, ASCII = 46), and not setting DSCP.

Equipment Required

1. Managed network switch with ‘port mirroring’ capability or 10/100 Hub (not a switch).

2. Laptop with IP packet capture software (if possible have two laptop’s available - one for use at the central and one at the remote site)

The Test Points Diagram

This diagram represents the test network employed to simulate network connectivity between IP Offices using standard networking equipment seen in the field. Use this diagram as reference for the "Test Points" when capturing traffic in the customer’s network.

Test Calls

If possible, prepare to setup and run packet captures at both the central and remote locations for simultaneous packet capture of the test calls. Testing and capturing in this way will make the results easier to review and compare. If this is not possible initiate the packet capture tests at the central site at ‘Test Point 1’ and then relocate the Laptop and repeat the tests at the remote site using ‘Test Point 2’. Start at the central site IP Office location and begin the preparation for the packet capture.

It is recommended that the test calls and captures are gathered during normal business or operating hours. This will capture typical data network traffic and usage during calling times. It may assist in identifying any traffic or traffic patterns that may be interfering with IP Office operation.

1. Identify the customer switch port that the IP Office LAN port is connected to.

2. Assign an IP address to the test Laptop network interface within the local networks IP range.

3. If supported (a ‘managed’ switch) enable port mirroring on the network switch between the IP Office port and any open/available port where the Laptop, with the packet capture software, will be connected to. If necessary adjust for any VLAN’s in use.

4. If ‘port mirroring’ is not supported (unmanaged, plug and play switch) insert a 10/100 Hub between the IP Office LAN port and the customer switch port it is connected to.

5. If installing the 10/100 Hub in step 4, verify that the connections between the IP Office and the network switch are set for the original connection speed (10 or 100) and duplex (full or half).

6. After either of the above connections is installed, initiate a test call from the central to the remote IP Office and verify correct operation and conversation is heard at both locations.

7. If possible minimize user calls between the two IP Offices during the testing period, this will make identification and analysis of the test calls simpler in the next section.

8. Open the packet capture software. Start a new packet capture.

9. Initiate a call between the central and remote IP Offices, speak for several seconds.

10. Stop the test call.

11. Stop the packet capture.

12. Display the packet capture of the test call. "File > Save As" to prevent accidentally clearing the capture.

Analyzing the Captured Test Call and Traffic

When the packet capture is stopped the software should display a window similar to below:

1. Note the display settings in the capture window, in particular the "Time" and "Delta" columns. These can provide a quick visual read of the differences in individual packet arrival times of the call / packet flow. Changing the display to show these parameters will vary depending on the software used.

2. In the test network the central site "IP Office" is assigned IP address of 192.168.42.1. The remote site IP Office is assigned IP address 192.168.58.1. You must know the IP addresses of the IP Offices and associated applications in the customer network under test.

Across the top file bar select "Statistics > Conversation List > IP4", this will automatically search and identify all IP address endpoints and their conversations in the test trace. In switched LANs you should see only IP addresses that need to connect to the IP Office - other IP Offices, IP Hardphones or Softphones, Phone Manager Pro, VMPro, etc. Investigate IP addresses that appear unknown or unrelated to IP Office equipment or software operation. In this example there were two (2) conversations active during the packet capture.

NOTE 3: It is important to record, investigate, and identify all IP flows in the test call capture. Mis-configured Servers, Computers or network equipment may be directing or re-directing packets to the IP Office. These flows could interfere with normal IP Office operation and affect VoIP quality.

Example:
In the display screen capture above, IP addresses 192.168.42.1 and 192.168.42.250 are identified and connected via UDP port 50794 and 1901. Checking the website http://support.Avaya.com and searching IP Office ports (FIND: "IP Office Ports") we see that UDP port 50794 is related to the Sysmon (Monitor) application. The Manager PC (192.168.42.250) was running a Sysmon trace at the time of the test call capture.

3. After reviewing and identifying all IP addresses in the capture you can enable Display Filtering, displaying only the IP addresses of the IP Offices involved in the Test Calls. Across the top File bar find and select "Analyze" > "Display Filters". Use the example below to create a new filter. The example here uses 192.168.42.1 and 192.168.58.1. Configure with the two IP addresses of the IP Offices used in the customer network.

When completed "Save" the filter and "Apply".

With the new Display Filter applied the display should appear similar to the screen capture below. Note the "Filter" window bar displaying the details of the recently created and applied filter. The display should now show only those packets between the two IP Offices of the test call. (The "Clear" button to the right of the Filter window is used to remove the filter from the packet capture).

4. Across the top File bar locate and select "Statistics" > "VoIP calls". Within seconds a new pop-up window is displayed (see below). You can see in the display the call was initiated from IP Office 192.168.58.1.

Select and highlight the test call and select "Graph". The window below is the result of the "Graph" function. This is a trace of the H.323 call setup including the H.225 and H.245 components. The "Graph Analysis" window shows an IP Office H.323 call setup. Note the "Time" column - this shows a call setup with minimum delay or re-transmissions due to network errors.

5. Close the two windows opened above, leave only the original display open (as seen on page 8). Leave the two IP address Filters in place. Now select "Statistics" > "RTP" > "Show All Streams". The window below "RTP Streams" should be displayed.

Acceptable

6. This window displays several useful items of information related to details of the RTP packet call flow:
   Source / Destination IP adresses,
   Source / Destination ports,
   Payload (Codec in use),
   Number of packets in the stream,
   Lost packets (if any),
   Max Delta (maximum time between two packets arrival),
   Max Jitter,
   Mean Jitter.

Compare the display below with that above - this shows an unacceptable delay in the RTP stream in one direction (Max Delta = 341.31 milliseconds).

Unacceptable

NOTE 4: The RTP conversation streams are independent of each other; one from 192.168.42.1 to 192.168.58.1 and one from 192.168.58.1 to 192.168.42.1. This is standard of H.323 RTP streams. One explanation of how conversation quality can differ between caller and called party (one reports an issue, the other does not). This can result from several possibilities such as; mis-configuration of ingress or egress QoS settings, mis-configured port speed and duplex settings, additional hops in source / destination routes and mis-configured Firewall settings (one-way VoIP).

Compare the overall RTP stream results to the Avaya requirements for VoIP traffic and network assessments below (from the document "Avaya IP Voice Quality Network Requirements" on page 2).

7. Select an RTP packet from the capture for further detailed analysis. In the Packet Details pane expand the ‘Internet Protocol’ field. Note the DSCP is set to a hex value of 0xb8 which is "Expedited Forwarding". This is the industry standard priority marking for all RTP media streams. Check that both transmit and receive RTP streams have this DSCP marking (see screen capture below).

NOTE 5: If there is no DSCP marking in the receive packets the most likely cause is the network equipment ‘stripping’ or modifying the packets during transit. Without the DSCP setting and correct network equipment configuration to prioritize the RTP streams VoIP quality across the network cannot be assured.

Correct

The following screens are an example captured from a network. It demonstrates that the network is modifying and removing the RTP stream DSCP settings somewhere in the network between IP Office locations.

Correct
Note the transmit RTP stream from the local IP Office shows the DSCP correctly set in packet #110.

Incorrect
Note the receive RTP stream from the remote IP Office shows the DSCP setting in packet #111 has been removed.

In addition, identify and check several of the IP Office H.323 call setup packets (both send and receive). By default the IP Office DOES NOT apply DSCP settings to the call setup packets. In the example below the customer has applied a DSCP setting to the H.323 call setup (DSCP = 0xA0). This is configured in the "System > Gatekeeper" tab of the IP Office Manager.

The user has the option to change the default DSCP for IP Office H.323 call setup if desired. If this is done then the user or network provider assumes responsibility to correctly configure the network equipment to identify and prioritize packets with the user configured DSCP setting. If the call setup DSCP is changed it is recommended this change apply to all IP Offices in the SCN network.

Continue testing using two new test points "Test Point 3" and "Test Point 4" seen in the Test Network Diagram. As previously recommended, if possible prepare to setup and run packet captures at both the central and remote locations for simultaneous packet capture of the test calls. If this is not possible initiate the capture at the central site at ‘Test Point 3’ and then relocate the Laptop and repeat the tests at the remote site using ‘Test Point 4’.

Refer to the "Test Calls" procedures for the new test points as described on page 5 beginning with item 1.

Analyze the new packet captures from Test Points 3 and 4 using the same analysis techniques described for Test Points 1 and 2.

Using the four (4) test points and the packet captures from each compare and identify the most likely location of errors affecting the VoIP calls. The Table below provides a quick reference worksheet for comparing and isolating possible issues from the four test calls. After a review of the packet captures the tester should have valid data to identify the areas requiring additional investigation to correct VoIP quality issues.

Test Result Matrix

Additional Items to Check - possible issues
Items of interest to investigate and verify in the event of "out-of-specifications" indications in the above tests:

1. Speed and Duplex settings on all interfaces.

2. VLAN switch settings and assignments - if applicable.

3. QoS configuration of the Routers.

4. Correct VLAN to QoS mapping in network equipment.

5. IP Office DSCP settings - all sites.

6. Customer, Provider / Carrier network prioritizing - not setting - DSCP.

7. VPN / Firewall settings - if applicable - blocking IP Office operational ports.

8. Any Switch / Router "Access Lists" blocking IP Office operational ports.

9. Network circuit interfaces (CSU/DSU) for transmit or receive errors.

10. Mis-configured, incorrect IP routes.

11. Switch / Router interface statistics - high number of collisions, high bandwidth usage, high errors or resets.

12. Non-IP Office traffic or applications attempting connection to the IP Office IP address.

13. Contact the network circuit/s provider - request review and the above information from their network.

14. Perform a Network Assessment and review the results.