[[Testing]]



Add New Page: You are not allowed to add pages Select section/namespace. New page title.
 

Testing

Mesh Routing Tech Evaluations

Below are possible tests to run on a deployed Commotion network, and possible field deployment needs to accommodate full gamut of testing.

Quality of Service Testing Metrics (QoS)

  • Throughput
  • Packet Loss - Expected Transmission Count
  • Delay - Expected Transmission time
  • Jitter
  • Mobile Node - Energy Consumption
  • Protocol Overhead
  • Memory Consumption
  • CPU Load

Test Suites

  • NOTE:
  • * nodes are mesh-enabled devices
  • * client devices are non-mesh enabled devices that are connected to a node. As our primary use case is wireless client communication, all client devices are to be connected to nodes wirelessly. If static application servers are a supported use case in international deployments all tests should be repeated with one client device attached over an ethernet port.
  • QoS per-user Traffic Load Testing Metrics (vs. single node)*

Quantity of Traffic priority (low, medium, high, and overload load sizes)

This setup requires one node and four client/server(s). The clients will all be attached to the single node. The clients will be split into teams. Each team will send a stream of data at different bandwidths. (low vs. high, etc.) in every possible combination of (low, medium, high, and overload load sizes) and retaining the performance metrics received. This will be done using iperf. These tests will be run twice. Once in UDP mode and once in TCP mode to account for protocols impact on traffic prioritization.

Quantity of Traffic priority (varied packet sizes)

This setup requires one node and four client/server(s). All clients need to be attached to the node. The clients will be split into teams. Each team will send a stream of data with differing packet sizes. Each team will retain the performance metrics received. Tool for testing to be decided. These tests will be run twice. Once in UDP mode and once in TCP mode to account for protocols impact on traffic prioritization.

* Traffic/Service priority (data, voice, video overload and QoS)

This setup requires one node and four client/server(s). All clients need to be attached to the node. The clients will be split into teams. Each team will send a stream of data that mimics a target data type. Each team will retain the performance metrics received. Tool for testing to be decided. These tests will be rerun to match each data type vs. each other data type above.

  • Back-haul congestion vs. client traffic

This setup requires three nodes and four client/server(s). The nodes will need to be arranged in a line with two clients attached to the central node and one client attached to the far nodes. The clients need to be in range of only one node to ensure that no hand-off takes place.

  • * high client vs. low back-haul

In this case the central team of clients will be transferring a high amount of data between them with the end teams passing a low bandwith transfer.

  • * high back-haul vs. low client

In this case the central team of clients will be transferring a low amount of data between them with the end teams passing a high bandwith transfer.

  • * high client vs. high back-haul

In this case the central team of clients will be transferring a high amount of data between them with the end teams passing a high bandwith transfer.

  • * low client vs. low back-haul

In this case the central team of clients will be transferring a low amount of data between them with the end teams passing a low bandwith transfer.

  • QoS Per-hop Testing Metrics (vs. n hops)*
  • QoS metrics on 1, 2, …, n hops

This test will require a string of nodes and two clients. The clients will repeat a bandwidth test (UDP and TCP) on the same node, one node away, two nodes away, up to the largest set of nodes possible away. The nodes will need to be placed far enough away form each other that there is no possibility of hand-off during testing(as it will skew the accuracy of testing.)

  • QoS vs Traffic Load per hop
  • * emulate a variety of traffic loads and packet sizes per hop at various points in test packets path.

*QoS Directional Testing Metrics* (controlled upstream, downstream, & bidirectional directions)

These tests are to be done on a single node. One client will be attached to the node via ethernet, and one attached to the node via wireless. The tests will consist of client generated traffic between the devices that simulates up and downloads of various sizes and speeds. The devices will have their times synced before the test starts and track when they receive and send packets. This will be used to clock the speed of traffic between the data being uploaded by the router to the eth device and sent downstream to the wlan device.

  • hi-up, low remaining
  • hi-down, low remaining
  • hi, hi, hi
  • hi, hi, low
  • hi, low, hi
  • low, hi, hi

*Routing Testing metrics* (via simulated via Fail-over conditions )

These tests require coordinated removal of nodes while monitoring traffic.

  • Redundancy

Redundancy cannot be tested at this time as we do not have a redundant traffic or storage system in place.

  • Flow Redirection

This test will require three nodes and two clients. The nodes will have full connectivity to each other but at distances (or interference) on one node to ensure that there is a prioritized path between the other two. One client will have connectivity to only one node, while the other has connectivity to the remaining two. This test will start by having a client A - client B data transfer over the best path. After collecting base data the “best path” will be degraded forcing hand-off to the now better path. We will measure the total time a client takes to switch once the route is degraded, the impact upon the data transfer, and the time to switch from receipt of OLSRd routing messages on the client device itself.

  • Path Loss

This test will mimic the flow redirection test except that it will use four nodes and two clients The nodes will be arranged so that there is a node at each end that can only communicate to two nodes in the middle of the path. This test will keep the client connected to the end node and degrade performance of the central nodes. this will test how a node handles flow redirection on a further scale. This test should also be repeated with six nodes that cause the three hop path to be degraded.

  • Routing loops (by time and loop iterations)

TODO: Look to battle-mesh to see what process they use to examine this

  • QoS Security Setting metrics*
  • Test QoS against various combinations of security setting to test bottlenecks
  • ServalD

Test thruput of data using all combinations of ServalD running. Authenticated traffic, authEncrypted traffic, encrypted traffic, no serval security installed etc. This will be done to see the impact of servalD on performance. This will be done with two client nodes (running ServalD) over a two hop mesh, a single AP, multi-hop p2p transfer, and broadcast over multiple hops.

  • OLSRd-mdns

Test OLSRd-mdns' impact on traffic thru-put on tcp traffic and udp traffic.

QoS Mobility Conditions (multiple velocities [walking -> train])

  • motion of client node w/ static mesh nodes

This test requires a client and three static nodes that only slightly overlap. The client will need to send data to another static (non-moving) client who is attached to the initial node. The mobile client will move out of range of the inital node and into the range of the second node, and then the third. These tests will be done at walking speed, bike speed, car speed, and -if legal, allowed, and recorded - shot out of a cannon speed.

  • motion of mesh node w. static mesh node (bus with node on it)

This test requires a vehicle with a node attached, a client in that vehicle, a static client, and two static nodes. The first client will attach to the mobile node and transfer data to the static client. The mobile node will then move out of range of the static client to force it to mesh with the first, and then second static node. This test will be repeated in both directions, as well as at different speeds of movement.

* motion of multiple client nodes in relation to each other

This is the “RUN AWAY”, and “CHARGE” test of Commotion. To start two clients will start at either end of a three node network. The clients will start a data transfer and then move towards each other screaming CHARGE! at the top of their lungs. Once they reach each other they will wait until the data transfer stabilizes to create a new base data at which point they will turn and move away from each other, back to either end shouting RUN AWAY!. Finally the clients will meet at one end of the three node deployment and move to the other end in tandem, preferably skipping and holding hands. These tests will be done at various speeds from walking to riding bikes (possibly with jousting poles), to driving.

* mobility and congestion on static mesh nodes (moving from sparse to heavy mesh traffic area while VOIP [on train through city[)

This test will require repeating all of the above tests with the added fun of having a four node mesh with one node being barraged with heavy loads of traffic. This will cause the clients to move through different levels of traffic on nodes.

QoS Interference Testing Metrics

  • non-commotion nodes on same channel

This test will require measuring the QOS of two clients communicating over a single, and then three node mesh when there are various other non-commotion wireless devices using the same commotion channel. This test will start at a baseline of no other wireless devices on the same channel and then quickly add more and more devices to interfere. We will measure just how many devices begin to cause significant degradation to the commotion network. On the three node mesh we will test the devices at an end point and at the central node.

  • multiple node interference

This test will be as above except with commotion nodes on the same channel. This test is vital to determining the amount of commotion devices on a channel that can co-exist before QOS suffers. This will be used later in our work on dynamic configuration and gain control… which the person writing this document is going to be developing, so really, please make sure we do this one.

  • multiple node inter-path interference (a→b, c→d in same range)

This test is as above except that it is not just measuring interference. This test will have two, four, and six pairs of clients that are transferring data to each other over different nodes that overlap. We will have to set static routes for this traffic in order to ensure that we are not measuring one nodes degradation with increased traffic, but instead the impact of a node processing and throwing away packets that are mesh, but not marked for that node.

  • link-capacity (non-commotion traffic on 2.4ghz)

This test is as above except that it is non commotion traffic. to do this we will set up the non-commotion (non olsrd) routers and have clients transfer data to or through them creating non-olsrd traffic and measuring the difference in delay of the node processing and tossing non-commotion traffic.

Testbed Requirements based on test suite defined above

Node Testing Setup needs

  • loops
  • overloading a single node with traffic
  • * back-haul and client traffic
  • * mobile can be used for client traffic?
  • WWW connection bottlenecks
  • multiple paths (back-haul data)
  • single-path to x node (for path-loss testing and bottlenecks)
  • overlapping ranges of multiple paths (a→b, c→d)
  • non-overlapping/near range-limit connections (for mobile node testing with congestion)
  • mobile nodes battery use logging for checking draw under different loads

External field/tech research needed

  • Active non-commotion active access points & the channels they are using (war-driving fun times!)
  • levels of 2.4ghz band interference in test area

Test Methodology

Preferred Test Environment

Testing the UI and other User Interface elements may be performed on a single Commotion node and wifi client as convenient.

However, testing mesh-level operations generally requires a minimum mesh of *3 Commotion nodes (1 gateway, 2 repeaters OR 2 gateways, 1 repeater)* for completeness of test coverage, or 2 nodes for bare minimum (1 gateway, 1 repeater). These nodes may be running “Commotion-OpenWRT”:/projects/commotion-openwrt , “Commotion-Linux”:/projects/commotion-linux , “Commotion-Android”:/projects/commotion-android , and/or whichever platform for Commotion is being tested.

Furthermore, and distinct from the Commotion nodes, you want ideally *2 clients*, or bare minimum 1 client, e.g. a Windows laptop or a wifi-capable tablet, to test connecting to the APs od each Commotion node under test.

Virtual nodes under VirtualBox

Besides testing Commotion on the target architecture directly, it is expecting some rounds of testing will occur on virtual machines (VMs), eg. those created by VirtualBox, for the sake of reducing development time. In particular, testing the UI and general User Experience elements should be doable on singleton VMs. In addition, mesh-level testing can occur on a network between multiple VMs under the same VirtualBox host.

Virtual-Box explains more about creating VMs running Commotion as guest VMs under VirtualBox.

Virtual nodes under VMware

Alternately, Vmware-Player explains more about running Commotion under VMware Player.

Neighborhood and Community Testbed Status

OTI operates or is involved in three different testing environments:

  1. In office testbed

A bleeding edge in-office testbed used for testing the latest images and reported bugs in the commotion software.

  1. Labratory Testbed

A stable, 8 node wireless testbed utilizing RF shielded cages and an RF routing matrix switch. The RF links between wireless nodes have variable and programmable attenuators and a many-to-many matrix router for configuring specific link qualities between each node. This is the first test environment for extensive functional and stress testing, since it is highly controllable.

  1. Neighborhood Testbed

This consists of an 18 node rooftop network in the Cass Corridor neighborhood of Detroit, MI.

References

development_resources/testing/testing_procedures_and_methodologies.txt · Last modified: 2014/02/25 20:40 (external edit)
 
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported