Transcript D-10 (Degioanni) To the Terabyte and Beyond

To the Terabyte and Beyond! Leveraging Pilot
and Wireshark to Analyze Truly Massive Packet
Traces
June 17, 2010
Loris Degioanni
CTO | CACE Technologies
SHARKFEST ‘10
Stanford University
June 14-17, 2010
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010
Packet Aquisition
Live network traffic
Content Indexing Thread
Captures packets off the live
interface and creates an index that
speeds up “index-friendly” Views.
Capture thread
Moves packets from the live interface
into the memory buffer.
Pilot Live Views Threads
Perform live network analysis,
including trending and alerting.
Memory
buffer
View
SQL DB
Index
Storage
Pilot Indexed Views Threads
Return summary information about
terabytes of traffic in a matter of
seconds.
View
SQL DB
Write-to-disk thread
Pilot Offline Views Threads
Moves packets from the
memory buffer into the packet
storage system. Create time
indices for fast packet retrieval.
Perform retrospective analysis on
stored packets.
View
SQL DB
RAID level 0
Packet Storage
System
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010
Capture Card
• Dedicated card is essential
–
–
–
–
No network stack overhead
Minimizes copies
Optimizes locality
Filtering capability in the card normally not really useful
•
•
•
Unless in some unusual conditions, the application wants to see everything
PCI bus is the only resource that card filtering optimizes
Any tap nowadays can do basic filtering
– Small packets is the worst condition
• CACE Turbocap
– Hybrid between home-built and off the shelf
– No unnecessary features (who needs filtering?)
– Affordable price
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010
CPU
• Bottlenecks
– CPU clock (expensive)
– Number of CPUS (cheap)
• Multi-threading hard to leverage when capturing and processing network
packets
– Network monitoring is intrinsically sequential
• Locking is evil
– Doing things more than once is better than locking
• At 10Gbps, cache coherency is a big deal
•
Small packets is the worst condition
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010
Disk
• Bottlenecks
–
–
–
–
Single disk write speed
Number of spindles
Raid Controller
Big packets is the worst condition
• Solid State? Not a good idea yet
– Single disk performance is not really the bottleneck
– Cost is an important factor when you build a system with tens of disks
– Reliability not as proven as the old magnetic disks
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010
Disk write speed based on position
11000
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
0
20000
40000
60000
80000
100000
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010
120000
140000
I can capture a lot of packets. Now what?
• Read of packets must be non-disruptive!
• Even if I stop the capture process, since I was writing at full
speed, reading the data is going to take around the same time
of writing it
– Read needs to be localized
– I need high level visibility to reach the point I need
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010
Indexing
Standalone card vs. kit
A network card nowadays is not enough
to build a functional packet
capture system.
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010
Indexing
• While capturing, on a Shark Appliance capture job
• On a trace file, after the fact
• Summary of the network traffic
– Volume, talkers and protocol information
– Coordinated with the packet store
– “Netflow on steroids”
• Designed to be extremely efficient in terms of disk usage
• Coordinated with the packet store
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010
Indexing
Index file
Index entry
Time intervals
Time index
File Positions
pcap file
Packet
SHARKFEST ‘10 | Stanford University | June 14 –17, 2010