Transcript Slide 1

Tranalyzer
Feel the packets, be the packets
Stefan
Burschka
What we do:
Network Troubleshooting, Security:
l
l
l
l
TRANALYZER(T2/3): High Speed and Volume Traffic Analyzer
TRAVIZ: Graphical Toolset for Tranalyzer
Complete Tool Sets for Traffic Mining (TM), Forensics
Artificial Intelligence
Research: TM & Visualisation
Brain support 4 multi-dim datasets
Encrypted Traffic Mining
Operational Picture
Malware and covert channel detection
Nifty stuff
2
“It's the network – go fix it!”
l
3
The Network is slow, The Network is insecure;
NO, it's not Microsoft, shut up, It wasn't me ...
Manager (MBA)
Production (poor Techie)
Finance (MBA)
Always right, DoR
License to Powerpoint
Knows, Always warned,
Always his fault: FUBAR
License to get fired
Knows basic calculus
License to Excel
We didn't find the problem in 4 months, can you do the job
in 2 weeks? (We supply 20TB data)
Troubleshooting, Security
Traffic Mining: Change your perspective
5
What is wrong here?
6
See the disaster now?
Now you have context!
7
Traffic Mining(TM):
Hidden Knowledge: Listen | See, Understand, Invariants, Model
Application in
–
–
–
–
8
Troubleshooting, Security (Classification, Encrypted TM )
Netzwerk usage (VoiP, P2P traffic shaping, application/user profiling)
Profiling & Marketing (usage performance- & market- index)
Law enforcement and Legal Interception (Indication/Evidence)
Basic Need: Versatile Flow Compression
A
B
Definition: (6-Tuple)
Vlan(s), srcIP, srcpPort, dstIP, dstPort, L4Protocol
Or why not a bit more context and meaning ?
srcWho, dstWho
srcNetwork, dstNetwork
Bad, Good
Internal / External
9
Closed source loud Tools
Netflow (Sometimes not so loud, comes with routers)
Pro: Good hands-on tool, flow statistics, header parameters, standard
Cons: Not all statistics we need, no developer support
GigaStor (Horrible loud and exceptional expensive HW)
Pro: heuristic expert system, Graphics, reports, whatever is in the DB
Cons: What we needed is not in the DB, no developer support
DPI (Elacoya, Sandvine,..) (Terrible loud and expensive HW)
Pro: good protocol resolution, nice reports
Cons: Its a DPI not a verstile flow engine with developer support
10
Open source silent SW
Wireshark, T-Shark (packet, flow statistics)
Pro: Hands-on tool, protocol db, GUI, command line, filtering
Cons: Limited flow statistics and file size, post processing difficult
Silk (flow based)
Cons: Not even close to Netflow, 5 tuple, esoteric config
Netmate
Pro: Flow, packet based, nice features,
Cons: Config , handling, 5 tuple, that is, ... University
NTOP(ng)
Pro: Monitoring, flow statistics, config, GUI, Graphics
Cons: not really flow based as we need it, protocol encapsulation?
IDS (SNORT, BRO)
Pro: Alarming, regex, flexible
Cons: Alarming, no Flows, BRO: memory leaks, university stuff
11
Need an Allrounder, script friendly
between Wireshark, Netflow and
2006: Somebody has
to develop me !!
Tranalyzer2(T2), C99, (Geek/Dev/Prof)
High Volume Traffic Preprocessing and Troubleshooting
Open Source
Speed and Memory optimized by *.h“, config and ./autogen.sh -n
Command line based, full pcap, eth and dag cards
Post processing : HEX, ‘text \t’; Bash, AWK, Perl, … friendly
C Plugin based, Linux, Mac, (Windoof)
Subnet labeling (Who, Where, What)
BPF
Hands-on: Anomaly and security related flags
Researchers: Full Statistical and Packet Signal Analysis support
Interfaces: Matlab, GnuPlot, SPSS, Excel, oocacl, soon Netflow tools
The “-s” option: The command line AWK, Perl friendly packet mode
GUI: Traviz (http://sourceforge.net/projects/traviz)
Easy to use but,
You have to know your shit
•
•
13
T3, C99, (Geek/Normalo NonDev/Prof)
High Speed and Volume Troubleshooting, Security, Monitoring
Complete new Concept and Design
Full IPv4/6, more protocols as T2
Basic Features from T2 + new nifty Plugins
Full Subnet labeling and flexible flow aggregation
Multi Threading and Interface: High performance
GUI Support via professional Tool Set: Unlimited flows and files
ipSOM: AI Tool Set to answer ANY question
Core functions into DSP and FPGA in future for the 40Gig+
More non geek/dev user friendly but,
You still have to know your shit
•
14
Report T2
•
/tranalyzer -r ~/wurst/data/weichwurst.dmp -w ~/wurst/results/hartwurst
================================================================================
Tranalyzer 0.5.8 (Anteater), beta. PID: 6123
•================================================================================
Active plugins:
00: protocolStatistics, version 0.5.8
01: basicFlowOutput, version 0.5.8
02: macRecorder, version 0.5.0
03: portBasedClassifier, version 0.5.8
04: basicLayer4CalcStatistics, version 0.5.6
05: tcpFlags, version 0.5.8
06: tcpStates, version 0.5.6
07: icmpDecode, version 0.5.8
08: connectionCounter, version 0.5.5
09: descriptiveStatistics, version 0.5.6
10: nFirstPacketsStats, version 0.5.8
11: packetSizeInterArrivalTimeHisto, version 0.5.8
12: standardFileSink, version 0.5.0
13: textFileSink, version 0.5.8
-->
-->
-->
-->
-->
-->
-->
-->
-->
-->
-->
-->
-->
-->
_protocols.txt,
_flow.txt / bin
ports.txt
subnet.txt
_flow.txt / bin
_flow.txt / bin,
portmap.txt
_flow.txt / bin
_flow.txt / bin
_flow.txt / bin
_flow.txt / bin, _icmpStats.txt
_flow.txt / bin
_flow.txt / bin
_flow.txt / bin
_flow.txt / bin
creates text output _flow.txt
creates binary output _flow.bin
Start processing file: /home/wurst//data/weichwurst.dmp
BPF: (null)
Dump start: 1351794649.186547 sec : Wed 01 Nov 2012 18:30:49.186547
Shutting down Tranalyzer 0.5.8...
Dump stop: 1351837376.118852 sec : Thu 02 Nov 2012 06:22:42.118852
Total dump duration: 42712.932305 sec
Number of processed packets: 6497970
Number of processed traffic bytes: 1749617780
Number of ARP packets: 1603
Number of RARP packets: 5
Number of IPv4 fragmented packets: 299
Number of IPv6 packets: 0
Number of IPv4 flows: 3395325
Average snapped Bandwidth: 327.634 KBit/s
Average full IP Bandwidth: 326.386 Kbit/s
Warning: IPv4 Fragmentation header packet missing
•
15
T2 Protocol File
Total packets captured: 42278
L4 Protocol # Packets
Relative Frequency[%] Protocol description
1
21
0.049671 Internet Control Message Protocol
2
6
0.014192 Internet Group Management Protocol
6
41698 98.628128 Transmission Control Protocol
17
250
0.591324 User Datagram Protocol
103 28
0.066228 Protocol Independent Multicast
Total TCP packets: 41698
Port # Packets
Relative Frequency[%]
80
41519 99.570723 World Wide Web HTTP
445 8
0.019186 Win2k+ Server Message Block
5557 147
0.352535
Total UDP packets: 250
Port # Packets
Relative Frequency[%]
53
2
137 50
138 21
1900 18
1908 2
1985 156
0.800000 Domain Name Server
20.000000 NETBIOS, [trojan] Msinit
8.400000 NETBIOS Datagram Service
7.200000 SSDP
0.800000 Dawn
62.400000 Hot Standby Router Protocol
•
16
T2 ICMP Stats File
Total # of ICMP messages:
22258
ICMP / Total traffic percentage[%]: 0.343
Echo reply / request ratio: 0.892
Type Code # of Messages Relative Frequency [%]
ICMP_ECHOREQUEST
111 0.499
ICMP_ECHOREPLY 99
0.445
ICMP_SOURCE_QUENCH
15
0.067
ICMP_TRACEROUTE 0
0.000
ICMP_DEST_UNREACH
ICMP_NET_UNREACH
60
0.270
ICMP_DEST_UNREACH
ICMP_HOST_UNREACH
15674 70.420
ICMP_DEST_UNREACH
ICMP_PROT_UNREACH
0
0.000
ICMP_DEST_UNREACH
ICMP_PORT_UNREACH
3100 13.928
ICMP_DEST_UNREACH
ICMP_FRAG_NEEDED
0
0.000
ICMP_DEST_UNREACH
ICMP_SR_FAILED 0
0.000
ICMP_DEST_UNREACH
ICMP_NET_UNKNOWN
0
0.000
ICMP_DEST_UNREACH
ICMP_HOST_UNKNOWN
0
0.000
ICMP_DEST_UNREACH
ICMP_HOST_ISOLATED
0
0.000
ICMP_DEST_UNREACH
ICMP_NET_ANO 8
0.036
ICMP_DEST_UNREACH
ICMP_HOST_ANO 600 2.696
ICMP_DEST_UNREACH
ICMP_NET_UNR_TOS
0
0.000
ICMP_DEST_UNREACH
ICMP_HOST_UNR_TOS
0
0.000
ICMP_DEST_UNREACH
ICMP_PKT_FILTERED
776 3.486
ICMP_DEST_UNREACH
ICMP_PREC_VIOLATION 0
0.000
ICMP_DEST_UNREACH
ICMP_PREC_CUTOFF
0
0.000
ICMP_REDIRECT ICMP_REDIR_NET 1125 5.054
ICMP_REDIRECT ICMP_REDIR_HOST 589 2.646
ICMP_REDIRECT ICMP_REDIR_NETTOS
0
0.000
ICMP_REDIRECT ICMP_REDIR_HOSTTOS
0
0.000
ICMP_TIME_EXCEEDED
ICMP_EXC_TTL 95
0.427
ICMP_TIME_EXCEEDED
ICMP_EXC_FRAGTIME
0
0.000
ICMP_TRACEROUTE 0
0.000
•
17
T2 Flow Header File: Hands-On
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
.....
8:NR Minimum layer3 packet size
8:NR Maximum layer3 packet size
19:NR Average packet load ratio
19:NR Send packets per second
19:NR Send bytes per second
19:NR Packet stream asymmetry
19:NR Byte stream asymmetry
8:NR IP Minimum delta IP ID
8:NR IP Maximum delta IP ID
7:NR IP Minimum TTL
7:NR IP Maximum TTL
7:NR IP TTL Change count
13:NR IP Type of Service
14:NR IP aggregated flags
8:NR IP options count
13,15:NR
IP aggregated options
•
18
T2 Flow Header View: Hands-On
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
8:NR TCP packet seq count
10:NR TCP sent seq diff bytes
8:NR TCP sequence number fault count
8:NR TCP packet ack count
10:NR TCP flawless ack received bytes
8:NR TCP ack number fault count
8:NR TCP initial window size
19:NR TCP average window size
8:NR TCP minimum window size
8:NR TCP maximum window size
8:NR TCP window size change down count
8:NR TCP window size change up count
8:NR TCP window size direction change count
13:NR TCP aggregated protocol flags (cwr, ecn, urgent, ack, push, reset, syn, fin)
14:NR TCP aggregated header anomaly flags
8:NR TCP options Packet count
8:NR TCP options count
Yes I know, I should do something
15:NR TCP aggregated options
special for the TimeStamp option
8:NR TCP Maximum Segment Length
7:NR TCP Window Scale
19:NR TCP Trip Time Syn, Syn-Ack | Syn-Ack, Ack
19:NR TCP Round Trip Time Syn, Syn-Ack, Ack | TCP Ack-Ack RTT
19:NR TCP Ack Trip Min
19:NR TCP Ack Trip Max
19:NR TCP Ack Trip Average
13:NR TCP aggregated protocol state flags
15,14:NR
ICMP Aggregated type & code bit field
19:NR ICMP Echo reply/request success ratio
9:NR Number of connections from source IP to different hosts
9:NR Number of connections from destination IP to different hosts
•
19
T2 Flow Header View: TM geeks
68
19:NR Minimum packet length
69
19:NR Maximum packet length
70
19:NR Mean packet length
71
19:NR Lower quartile of packet lengths
72
19:NR Median of packet lengths
73
19:NR Upper quartile of packet lengths
74
19:NR Inter quartile distance of packet lengths
75
19:NR Mode of packet lengths
76
19:NR Range of packet lengths
77
19:NR Standard deviation of packet lengths
78
19:NR Robust standard deviation of packet lengths
All you never wanted
79
19:NR Skewness of packet lengths
80
19:NR Excess of packet lengths
know about statistics
81
19:NR Minimum inter arrival time
flow
82
19:NR Maximum inter arrival time
83
19:NR Mean inter arrival time
84
19:NR Lower quartile of inter arrival times
85
19:NR Median inter arrival times
86
19:NR Upper quartile of inter arrival times
87
19:NR Inter quartile distance of inter arrival times
88
19:NR Mode of inter arrival times
L2/3/4/7
89
19:NR Range of inter arrival times
configurable
90
19:NR Standard deviation of inter arrival times
91
19:NR Robust standard deviation of inter arrival times
Packet
92
19:NR Skewness of inter arrival times
Statistics
93
19:NR Excess of inter arrival times
94
8,25:R L2L3/L4/Payload( s. PACKETLENGTH in packetCapture.h) length and inter-arrival
times for the N first packets
95
8,9,9,9,9:R Packetsize Inter Arrival Time histogram bins
to
in a
•
20
HOW TO find the
needle in the flow stack?
Have a break have a
HEX & ¦ scripting!
T2 Text Flow File: Basic plugins
A
1196278772.439355 1196279184.642073 412.202718 0x9B42
192.168.1.10 0x00000001 2119 68.3.4.5
0x800806034 80
00:0f:1f:cf:7c:45_00:00:0c:07:ac:0a_6387
http 6387 8272
5437587
0
4
15.494803
1.125660
-0.128590
0.999829
1
87
128
128
0x00 0x42 0x0000
116
6231 4116 5437724
2253 63754 64831.988281
65535 3342 2904 5713 0x18 0xF900
0x0000
0x03
0x00000000 0x0000 -1.0 1
1
1
...
B
22
6
464
464
62501
1196278772.409312 1196279184.642073 412.232761 0x9B43
22
192.168.1.10 0x00000001
80
68.3.4.5
0x80080634 2119 6
00:d0:00:64:d0:00_00:0f:1f:cf:7c:45_8272
http 8272 6387
5437587
464
0
1380 20.066333
13190.574633
0.128590
0.999829
1
3
63
63
0x00 0x42
0x0000
8146 5440245
109
116
464
8104 5840
5840.000000 65535 0
0
0
0
0x18 0x1B00
0x0000
0x03 0x00000000 0x0000 -1.0 1
1
1
...
•22
T2 Binary Coding Status:
2^0
2^1
2^2
2^3
2^4
2^5
2^6
2^7
2^8
2^9
2^10
2^11
2^12
2^13
2^14
2^15
0x0001
0x0002
0x0004
0x0008
0x0010
0x0020
0x0040
0x0080
0x0100
0x0200
0x0400
0x0800
0x1000
0x2000
0x4000
0x8000
Flow Warning Flag: If A flow: Invert Flow, NOT client flow
Dump/flow: L3 Snaplength too short
Dump/flow: L2 header length too short
Dump/flow: L3 header length too short
Dump: Warning: IP Fragmentation Detected
Flow: ERROR: Severe Fragmentation Error
Flow: ERROR: Fragmentation Header Sequence Error
Flow ERROR: Fragmentation Pending at end of flow
Flow/Dump: Warning: VLAN(s) detected
Flow/Dump: Warning: MPLS unicast detected
Flow/Dump: Warning: MPLS multicast detected
Flow/Dump: Warning: L2TP detected
Flow/Dump: Warning: PPP detected
Flow/Dump: 0/1: IPv4/IPv6 detected
Flow/Dump: Warning: Land Attack detected
Flow/Dump: Warning: Time Jump
So what is:
0x9B43
•
23
T2 Flow Binary Coding: ipFlags
2^0
2^1
2^2
2^3
2^4
2^5
2^6
2^7
2^8
2^9
2^10
2^11
2^12
2^13
2^14
2^15
0x0001
0x0002
0x0004
0x0008
0x0010
0x0020
0x0040
0x0080
0x0100
0x0200
0x0400
0x0800
0x1000
0x2000
0x4000
0x8000
IP Options present, s. IP Options Type Bit field
IPID out of order
IPID rollover
Fragmentation: Below expected RFC minimum fragment size: 576
Fragmentation: Fragments out of range (Possible tear drop attack)
Fragmentation: MF Flag
Fragmentation: DF Flag
Fragmentation: x Reserved flag bit from IP Header
Fragmentation: Unexpected position of fragment (distance)
Fragmentation: Unexpected sequence of fragment
L3 Checksum Error
L4 Checksum Error
SnapLength Warning: IP Packet truncated, L4 Checksums invalid
Packet Interdistance == 0
Packet Interdistance < 0
Internal State Bit for Interdistance assessment
So what is:
0x1C21
•
24
T2 Flow Binary Coding: tcpFlags
2^0
2^1
2^2
2^3
2^4
2^5
2^6
2^7
2^0
2^1
2^2
2^3
2^4
2^4
2^5
2^6
2^8
2^9
2^10
2^11
2^12
2^13
2^14
2^15
0x01
0x02
0x04
0x08
0x10
0x20
0x40
0x80
FIN
SYN
RST
PSH
ACK
URG
ECE
CWR
0x0001
0x0002
0x0004
0x0008
0x0010
0x0020
0x0040
0x0080
0x0100
0x0200
0x0400
0x0800
0x1000
0x2000
0x4000
0x8000
No more data, finish connection
Synchronize sequence numbers
Reset connection
Push data
Acknowledgement field value valid
Urgent pointer valid
ECN-Echo
Congestion Window Reduced flag is set
Fin-Ack Flag
Syn-Ack Flag
Rst-Ack Flag
Syn-Fin Flag, Scan or malicious packet
Syn-Fin-Rst Flag, potential malicious scan packet or malicious channel
Fin-Rst Flag, abnormal flow termination
Null Flag, potential NULL scan packet, or malicious channel
XMas Flag, potential Xmas scan packet, or malicious channel
Due to packet loss, Sequence Number Retry, retransmit
Sequence Number out of order
Sequence mess in flow order due to pcap pkt loss
Warning: L4 Option field corrupt or not acquired
Syn retransmission
Ack number out of order
Ack Packet loss, probably on the sniffing interface
Internal state: TCP Window Size Machine
So what is:
0x1B
0xC403
•
25
T2 Flow Binary Coding: icmpFlags
Aggregated ICMP Type & Code bit Field
So what is: 0x00000100_0x0001
•
26
T2 Packet Signal: Encrypted VoIP Mining
PacketLength_Packet-Interdistance; …
1023_0.000000;758_0.030043;1380_0.110201;80_0.00000;369_0.000010;230_0.02002
9;1380_0.070101;80_0.000000;50_0.060086;1380_0.070101;80_0.090130; …
Packet Length
time
27
Post processing scripts:
/tranalyzer/trunk/scripts
T2 Statistical Application / User profiling
Packet length-Interdistance Statistics: Fingerprint
PktLen_Packet-IAT_cnt_cntPktLen_cntIAT; …
0_0_2322_6271_2396;0_2_82_6271_90;0_4_114_6271_114;0_6_138_6271_140;0_8_162_6271_164;0_10_157_6271_160;0_12_220
_6271_224;0_14_217_6271_222;0_16_325_6271_325;0_18_373_6271_376;0_20_493_6271_498;0_22_340_6271_343;0_24_238_62
71_238;0_26_283_6271_284;0_28_143_6271_143;0_30_114_6271_114;0_32_139_6271_140;0_34_175_6271_176;0_36_72_6271_7
3;0_38_25_6271_25;0_40_20_6271_20;0_41_12_6271_13;0_42_8_6271_8;0_43_6_6271_6;0_44_6_6271_6;0_45_4_6271_4;0_46_5
_6271_5;0_47_9_6271_10;0_48_9_6271_9;0_49_6_6271_6;0_50_4_6271_4;0_51_4_6271_4;0_52_5_6271_5;0_53_3_6271_3;0_54_
9_6271_9;0_55_7_6271_8;0_56_1_6271_1;0_57_4_6271_4;0_58_1_6271_1;0_59_3_6271_3;0_60_4_6271_4;0_61_4_6271_4;0_62_
2_6271_2;0_63_1_6271_1;0_64_1_6271_1;0_65_1_6271_1;4_0_74_116_2396;4_2_8_116_90;4_6_2_116_140;4_8_2_116_164;4_10
_3_116_160;4_12_4_116_224;4_14_5_116_222;4_18_3_116_376;4_20_5_116_498;4_22_3_116_343;4_26_1_116_284;4_32_1_116_
140;4_34_1_116_176;4_36_1_116_73;4_41_1_116_13;4_47_1_116_10;4_55_1_116_8 …..
•
Post processing scripts:
/tranalyzer/trunk/scripts
Skype: Vulnerable against
TM Attack
•
28
Some T3 Plugins
L7 Protocols: Mail, HTTP, etc
Routing: OSPF
DNS / DHCP
Full PCRE Regex
Signal Processing
Artificial Intelligence (RNN, Bayes, ESOM), nifty entropy shit
Connection Matrix, Centrality
IP Statistics: Host
Database
•
29
So what?
Some Examples
The one way TCP Flow problem
Symptom: on and off access problems
TCP flows established, unidirectional
T2 proofed: Reverse connection exists, not through firewall
Not communicated online mis-configuration of firewall
Trampel
OSPF
FFT of some Packet Signals
•
Packet Length
•
time
•
32
Traffic Mining:
Encrypted Content Guessing
SSH Command Guessing
IP Tunnel Content Profiling
Pitch based Classification
Encrypted Voip Guessing: CCC 2011
33
TM Your OWN: Packet Length Signal
See the features?
Codec training
Burschka (Fischkopp) Linux
Dominic (Student) Windows
SN
Ping min l =3
34
Connection plugin: Social Behaviour
67
60
Frecuency
Frequency
3000,00
40
2000,00
16
20
1000,00
10
5
1
1
0
0,00
0
5
10
15
# Conne ctions
0
3
6
9
12
15
18
21
# Conne ctions
•
35
24
27
What is the Unknown?
•
36
HOW TO find Bad Guys?
Day: 0.7% of all users 42% bandwidth, WTF?
P2P Traffic
P2P Traffic
B
a
rss
h
o
w
M
e
a
n
s
Average Users
???
Percentil User
Normal Traffic
37
HOW TO find Bad Guys?
Night: Same guys @ night 3am, ...
P2P Traffic
Average Users
Machines of
WAREZ guys
38
Layer3/4/whatever Visualization
Graphviz --> Operational Picture in Bootcamp
_flow.txt
Your AWK script
Graphviz: dotty
•
39
Layer3/4 Visualization
Graphviz --> simple forensic Picture
Network Classification
Centrality
Connection Matrix
PCA
Largest Eigenvector Plot / t
•
41
Network / Host Classification
Centrality
ipSOM Operational Picture:
13 Dim statistical T2 Flow parameters
Now conceivable by human brain
Bot Scanner
DNS Zone Transfer
43
Questions / Comments
RFM and try me
Join the development force
Who wants Bootcamp?
http://sourceforge.net/projects/tranalyzer/
http://tranalyzer.com
http://sourceforge.net/projects/traviz
44
Google: Dataming for Hackers
[email protected]