Data Stream Processing for Packet-Level Analytics

Abstract

One of the most challenging tasks for network operators is implementing accurate per-packet monitoring, looking for signs of performance degradation, security threats, and so on. Upon critical event detection, corrective actions must be taken to keep the network running smoothly. Implementing this mechanism requires the analysis of packet streams in a real-time (or close to) fashion. In a softwarized network context, Stream Processing Systems (SPSs) can be adopted for this purpose. Recent solutions based on traditional SPSs, such as Storm and Flink, can support the definition of general complex queries, but they show poor performance at scale. To handle input data rates in the order of gigabits per seconds, programmable switch platforms are typically used, although they offer limited expressiveness. With the proposed approach, we intend to offer high performance and expressive power in a unified framework by solely relying on SPSs for multicores. Captured packets are translated into a proper tuple format, and network monitoring queries are applied to tuple streams. Packet analysis tasks are expressed as streaming pipelines, running on general-purpose programmable network devices, and a second stage of elaboration can process aggregated statistics from different devices. Experiments carried out with an example monitoring application show that the system is able to handle realistic traffic at a 10 Gb/s speed. The same application scales almost up to 20 Gb/s speed thanks to the simple optimizations of the underlying framework. Hence, the approach proves to be viable and calls for the investigation of more extensive optimizations to support more complex elaborations and higher data rates.

Keywords: data stream processing; multicore programming; packet-level analysis; software defined networking.

Listing 1

Business logic of the source operator.

Listing 2

Structure of a sniffer object.

Listing 3

Business logic of the flowid operator.

Listing 4

Business logic of the stats operator.

Listing 5

Business logic of the sink operator.

Listing 6

Construction of the application graph.

Figure 1

High-level view of the framework architecture.

Figure 2

Stream processing application. Stream elements are represented in grey. The application receives a stream in input and performs some computation on each received item, producing a stream of results in output. Operators can be stateless or stateful, in case they maintain a state containing information on the data processed so far.

Figure 3

Data-flow graph representation of a stream processing application. The processing pipeline is defined by the operators (nodes) in the graph. Their business logic is executed on every stream item traversing the corresponding path in the graph. The application logic applied to each flowing element results from the execution of successive transformations in the graph.

Figure 4

Processing pipeline of the example application. At the capturing level, three software packet-capturing tools are currently supported: pcap, TPACKET, and Netmap.

Figure 5

Speed test evaluation.

Figure 6

Evaluation on real-traffic replayed at 10 Gb/s from a pcap file. Application is run with source and sink operators only.

Figure 7

Evaluation on real-traffic replayed at 10 Gb/s from a pcap file. Application is run with all four operators in the processing pipeline.

Figure 8

Evaluation on real-traffic replayed at 20 Gb/s from a pcap file. Application is run with source and sink operators only.

Figure 9

Evaluation on real-traffic replayed at 20 Gb/s from a pcap file. Application is run with all four operators in the processing pipeline.